Unlike GPUs, CPUs aren't designed for massive parallelism. Because of this, batching inference won't necessarily give you a speed boost here. In fact, it can actually slow the process down.
Instead, I'd recommend exploring CPU-specific AI optimizations. For instance, leveraging AVX512_BF16 instructions could reduce the inference time by 2x or 3x compared to the results in the article. OpenVINO supports this really well on Intel CPUs, and converting an ONNX model to OpenVINO is straightforward.
+1 for OpenVINO, we utilise it for our model. It's quite amazing the inference speed you can get from CPUs that most people would assume are running on a GPU.
OpenVINO only uses ONNX to parse the model, not to execute it. It runs computations through its own highly optimized inference engine specifically designed for Intel hardware. It doesn't rely on the ONNX engine at all, and it will even automatically convert eligible model weights to BF16 for you
Instead, I'd recommend exploring CPU-specific AI optimizations. For instance, leveraging AVX512_BF16 instructions could reduce the inference time by 2x or 3x compared to the results in the article. OpenVINO supports this really well on Intel CPUs, and converting an ONNX model to OpenVINO is straightforward.
OpenVINO only uses ONNX to parse the model, not to execute it. It runs computations through its own highly optimized inference engine specifically designed for Intel hardware. It doesn't rely on the ONNX engine at all, and it will even automatically convert eligible model weights to BF16 for you
You can technically do Q4 quantization for larger embedding models but I am not sure if that plays nice with ONNX.
what we really need it something like auto-round for ONNX