{"id":723,"date":"2025-02-05T00:01:07","date_gmt":"2025-02-04T13:01:07","guid":{"rendered":"https:\/\/www.samontab.com\/web\/?p=723"},"modified":"2025-02-05T03:04:41","modified_gmt":"2025-02-04T16:04:41","slug":"from-cpu-to-npu-the-secret-to-15x-faster-ai-on-intels-latest-chips","status":"publish","type":"post","link":"https:\/\/www.samontab.com\/web\/2025\/02\/from-cpu-to-npu-the-secret-to-15x-faster-ai-on-intels-latest-chips\/","title":{"rendered":"From CPU to NPU: The Secret to ~15x Faster AI on Intel’s Latest Chips"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

Intel’s newest chips now come with a Neural Processing Unit (NPU)<\/strong>, built to handle AI and machine learning tasks more efficiently than a regular CPU. Instead of struggling with AI workloads on the CPU, the NPU is designed to run them faster and with less power. This is great because you can free up the CPU to do other general tasks, but I wanted to know how much better the NPU can run a model, compared to the CPU. Based on my test, it’s roughly a 15x performance boost<\/strong>, which is great.<\/p>\n\n\n\n

If you’re looking to buy an edge device with an NPU, I can recommend the Khadas Mind 2 Mini PC<\/a>, as it’s really small and packs a lot of power, plus it has a small battery that serves as a UPS, so you can just move it around from one USB power supply to another without it losing power. It’s quite nice. OK, now let’s see how I got to that number from the title.<\/p>\n\n\n\n

In real-time computer vision, throughput<\/strong> and latency<\/strong> are two fundamental performance metrics that impact the efficiency and responsiveness of a system. Throughput<\/strong> refers to the number of frames processed per second (FPS), determining how much data the system can handle over time. This is basically what you’re referring to when you ask “how long it takes to process this video”. Latency<\/strong>, on the other hand, is the time it takes to process a single frame from input to output, affecting how quickly the system responds to new data. Low latency is crucial for real-time applications like augmented reality and autonomous driving. When you play around with a system and it feels “laggy”, it’s because it has high latency. You want to keep your latency low and your throughput high.<\/p>\n\n\n\n

I’m going to assume you already installed OpenVINO on your system, and that you have an Intel chip with an NPU in it. You can quickly check if those two things are true by running this command:<\/p>\n\n\n\n

import openvino as ov\n\ncore = ov.Core()\ncore.available_devices<\/code><\/pre>\n\n\n\n
You should see something like [‘CPU’, ‘GPU’, ‘NPU’]<\/strong> as the reply to that. Those are the available devices in OpenVINO. If you don’t see your device, make sure you installed the drivers correctly and troubleshoot it before continuing.<\/p>\n\n\n\n
Next, we need a model. I\u2019ll be using ResNet-50<\/strong>, one of the most well-known Convolutional Neural Network architectures, introduced in Microsoft’s 2015 paper “Deep Residual Learning for Image Recognition”<\/a><\/em> It was trained on ImageNet-1K<\/strong> at a resolution of 224×224<\/strong>, meaning you can feed it an image of that size, and it will predict the probabilities for 1,000 different object categories<\/strong>. You can get the object names here<\/a>.<\/p>\n\n\n\n
Luckily, ResNet-50, optimised for OpenVINO, is available for download here<\/a>. Just grab those two files, resnet50_fp16.xml<\/strong>, and resnet50_fp16.bin<\/strong> and place them in your working folder. If you want to try with another model, you can also do that. Make sure to run the OpenVINO optimiser on your model to get the best performance. I’m also going to use OpenCV for image loading and resizing, so let’s install it first, and make sure numpy is there as well:<\/p>\n\n\n\n
pip install opencv-python numpy<\/code><\/pre>\n\n\n\n
Now let’s classify an image with this model. Write this into a file and save it as classify.py<\/strong>:<\/p>\n\n\n\n
import openvino as ov\nimport numpy as np\nimport cv2\n\ndef classify_image():\n    # Step 1: Load OpenVINO model\n    core = ov.Core()\n    model = core.read_model(\"resnet50_fp16.xml\")\n    compiled_model = core.compile_model(model, \"CPU\")  # Use \"NPU\" if available\n\n    # Step 2: Get input tensor details\n    input_layer = compiled_model.input(0)\n    input_shape = input_layer.shape  # Should be (1, 3, 224, 224)\n\n    # Step 3: Load and preprocess image\n    image = cv2.imread(\"input.jpg\")\n    image = cv2.resize(image, (224, 224))  # Resize to match model input\n    image = image[:, :, ::-1]  # Convert BGR to RGB (OpenCV loads as BGR)\n    image = image.astype(np.float32) \/ 255.0  # Normalise to [0,1]\n    image = np.transpose(image, (2, 0, 1))  # HWC to CHW\n    image = np.expand_dims(image, axis=0)  # Add batch dimension\n\n    # Step 4: Run the inference\n    output = compiled_model(image)[compiled_model.output(0)]\n\n    # Step 5: Process the results\n    top_class = np.argmax(output)  # Get class index\n\n    # Load ImageNet labels (remember to download the file)\n    imagenet_labels = np.array([line.strip() for line in open(\"imagenet_classes.txt\").readlines()])\n\n    # Display result\n    print(f\"Predicted Class: {imagenet_labels[top_class]}\")\n\nif __name__ == \"__main__\":\n    classify_image()\n<\/code><\/pre>\n\n\n\n
Make sure that in the same folder you have these files: classify.py<\/strong>, imagenet_classes.txt<\/strong>, resnet50_fp16.xml<\/strong>, and resnet50_fp16.bin<\/strong>. Now add any image in there and rename it to input.jpg<\/strong>. Now simply call the script:<\/p>\n\n\n\n
python classify.py<\/code><\/pre>\n\n\n\n
You should get the correct predicted class, like this:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Now that we know that the model actually works correctly with OpenVINO, we can benchmark it with a convenient tool that comes with it. It’s called benchmark_app<\/strong>, and it allows you to quickly check the performance of your devices with different models. You can call it like this:<\/p>\n\n\n\n
\n
benchmark_app -m MODEL<\/strong> -d DEVICE<\/strong> -hint HINT<\/strong><\/p>\n<\/blockquote>\n\n\n\n
For this benchmark, I ran these four commands:<\/p>\n\n\n\n
benchmark_app -m \"resnet50_fp16.xml\" -d CPU -hint latency\nbenchmark_app -m \"resnet50_fp16.xml\" -d CPU -hint throughput\nbenchmark_app -m \"resnet50_fp16.xml\" -d NPU -hint latency\nbenchmark_app -m \"resnet50_fp16.xml\" -d NPU -hint throughput<\/code><\/pre>\n\n\n\n
These are the results:<\/p>\n\n\n\n
Device<\/td>Hint<\/td>Median Latency (ms)<\/td>Average Latency (ms)<\/td>Min Latency (ms)<\/td>Max Latency (ms)<\/td>Throughput (FPS)<\/td><\/tr>
CPU<\/td>Latency<\/td>25.31<\/td>24.73<\/td>18.38<\/td>47.16<\/td>40.32<\/td><\/tr>
CPU<\/td>Throughput<\/td>47.38<\/td>63.65<\/td>38.52<\/td>135.35<\/td>62.69<\/td><\/tr>
NPU<\/td>Latency<\/td>1.68<\/td>1.7<\/td>1.52<\/td>8.3<\/td>569.71<\/td><\/tr>
NPU<\/td>Throughput<\/td>9.15<\/td>8.4<\/td>3.21<\/td>86.6<\/td>936.05<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Key Takeaways:<\/h2>\n\n\n\n
NPU (Latency mode) achieves 1.70ms<\/strong> average latency compared to 24.73ms<\/strong> on CPU (~15x improvement<\/strong>)
NPU (Throughput mode) reaches 936.05 FPS<\/strong>, which is ~15x higher<\/strong> than CPU throughput mode (62.69 FPS<\/strong>)<\/p>\n\n\n\n
This confirms that Intel’s NPU significantly outperforms the CPU<\/strong> in both latency and throughput, with a roughly 15x performance boost for this particular model.<\/p>\n","protected":false},"excerpt":{"rendered":"
Intel’s newest chips now come with a Neural Processing Unit (NPU), built to handle AI and machine learning tasks more efficiently than a regular CPU. Instead of struggling with AI workloads on the CPU, the NPU is designed to run them faster and with less power. This is great because you can free up the […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[97,29,21,24],"tags":[110,109,25,92,33],"class_list":["post-723","post","type-post","status-publish","format-standard","hentry","category-ai","category-computer-vision","category-open-source","category-openvino","tag-intel","tag-npu","tag-openvino","tag-performance","tag-python"],"_links":{"self":[{"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/posts\/723","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/comments?post=723"}],"version-history":[{"count":0,"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/posts\/723\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/media?parent=723"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/categories?post=723"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/tags?post=723"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}