Authors: Adeel Ahmad, Ahmad Tameem Kamal, Nouman Amir

Introduction

RISC-V is making real headway in the AI hardware space, but its software ecosystem hasn’t quite kept up with the hardware’s potential. This blog covers the work 10xEngineers did to start closing that gap.

In collaboration with the RISC-V Software Ecosystem (RISE), 10xEngineers worked on optimizing end-to-end object detection inference on RISC-V, using YOLOv8n as our target model and IREE as the compiler. The work had two main fronts: optimizing the IREE’s MLIR-based compilation pipeline for vision models at FP32, FP16, and INT8 precisions, and building optimized pre- and post-processing pipelines in C++ for the object detection workflow. Everything was benchmarked on the Banana Pi BPI-F3, an 8-core RISC-V vector CPU board with a vector length of 256 bits.

10xEngineers identified and fixed several gaps in IREE’s compilation pipeline and vectorized key pre- and post-processing operations, resulting in substantial latency improvements, particularly at INT8 precision, with accuracy validated against the COCO dataset. All the code developed during this project is open-source. To access the code, visit: 10x-Engineers/rise-yolo-iree · GitHub

Model MLIR Export

IREE accepts MLIR as its input format, so the first step was getting the YOLOv8n model into MLIR. This was done in two steps:

First, the YOLOv8n model was exported to ONNX format using the Ultralytics APIs. This was done for all three precisions: FP32, FP16, and quantized INT8.
Then, IREE’s iree-import-onnx tool was used to convert the ONNX model into MLIR.

Once in MLIR, the model is ready to be compiled by the IREE compiler and executed using the IREE runtime.

Figure 1: YOLOv8n MLIR export and compilation using IREE

IREE Optimizations

The Img2Col Pipeline

YOLOv8n was compiled for RISC-V using IREE’s ConvertConv2DToImg2ColPass. To understand the optimizations, it helps to first understand what this pass does.

The ConvertConv2DToImg2ColPass decomposes a convolution into two ops: a linalg.generic op that performs the Img2Col layout transformation, which rearranges the image data into a format that allows convolution to be applied as a matrix multiplication, and a linalg.matmul op that performs the actual computation. The matmul is then converted to linalg.mmt4d by IREE’s data-tiling framework, and lowered to specialized mmt4d micro-kernels that are optimized for cache utilization and SIMD execution on RISC-V.

Enabling the Img2Col Pipeline for INT8

The Img2Col pipeline worked for FP32 and FP16 precisions, but was not enabled for INT8. The reason is that quantized convolutions in MLIR are represented as linalg.conv_2d_nchw_fchw_q, a separate op that carries additional zero-point and scale parameters for quantization. The ConvertConv2DToImg2ColPass only handled the non-quantized linalg.conv_2d_nchw_fchw op and had no support for the quantized variant.

To fix this, we implemented a rewrite pattern in the QuantizedConvToConvPass that folds linalg.conv_2d_nchw_fchw_q into a standard linalg.conv_2d_nchw_fchw, making it compatible with the existing Img2Col pipeline without requiring any changes to the pipeline itself.

However, enabling the pipeline for INT8 also required a new micro-kernel. The existing mmt4d micro-kernels did not support i8xi8→i32 types for RISC-V, so we implemented one. Together, these two changes unlocked the full Img2Col pipeline for INT8 and were the primary driver behind the large INT8 speed-ups observed.

Vectorizing the Img2Col Layout Transformation

The linalg.generic op doing the Img2Col layout transformation was not being vectorized. The issue lies in the indexing maps of this linalg.generic op. It used non-projected permutation maps, which the standard Linalg vectorizer cannot handle, causing it to fall back to scalar loops that are significantly slower.

We resolved this by introducing a new lowering path that goes through iree_vector_ext.transfer_gather to vector.gather. The vector.gather op eventually maps to strided vector loads at the RVV level. This allowed the compiler to generate proper RVV vector instructions instead of scalar loop.

Layout Transformation Generic:

%4 = linalg.generic {indexing_maps =
[affine_map<(d0, d1, d2, d3, d4, d5) -> (d0, d1, d2, d3)>,
affine_map<(d0, d1, d2, d3, d4, d5) -> (d0, d1, d2, d3, d4, d5)>],
iterator_types = [“parallel”, “parallel”, “parallel”, “parallel”, “parallel”, “parallel”]}
ins(%0 : tensor<1x3x642x642xf32>) outs(%3 : tensor<1x3x3x320x320xf32>) {
^bb0(%in: f32, %out: f32):
linalg.yield %in : f32
} -> tensor<1x3x3x320x320xf32>

After Generic Vectorization:

%2 = iree.vector_ext.transfer_gather %arg0[%c0, %c0, %c0],
%1 : vector<16x16x8> -> %cst 0 [indexing_maps = [#map, #map1]] :
tensor<1x1x31xf32>, vector<1x1x1x1x16xf32>

Lowering to vector dialect op:

%3 = vector.gather %arg0[%c0, %c0, %c0] [%2], %cst 0, %cst :
tensor<1x1x31xf32>, vector<1x1x1x1x16xindex>, vector<1x1x1x1x16xi1>,
vector<1x1x1x1x16xf32> into vector<1x1x1x1x16xf32>

Improving MaxPool Vectorization for NCHW Layouts

MaxPool vectorization in NCHW layout was not handled efficiently by IREE’s vectorizer. The existing approach converted the NCHW layout to NHWC before vectorizing, which introduced redundant vector.transpose operations, adding unnecessary overhead.

We improved this by directly vectorizing the NCHW layout, eliminating the layout conversion step entirely. This removed the transpose ops, resulting in higher throughput for the MaxPool operations.

Elementwise Broadcast Optimization

YOLOv8n contains convolution bias broadcast operations where the convolution bias is broadcast and consumed as the accumulator/output operand of the convolution. Since the bias is a constant tensor, the bias broadcast should ideally be hoisted out of the inference loop into an initialization phase by IREE’s HoistIntoGlobalsPass, so it is computed once rather than on every inference call.

However, HoistIntoGlobalsPass has a default size limit on constants that can be hoisted, controlled by the maxSizeIncreaseThreshold variable, which is set to 1MB by default. As the bias broadcast op result exceeds the 1MB default limit, HoistIntoGlobalsPass skips it, leaving the bias broadcast computation inside the inference loop. This can be fixed by increasing the threshold using the –iree-opt-const-expr-max-size-increase-threshold compiler flag. Using –iree-opt-const-expr-max-size-increase-threshold=7MB causes elementwise broadcast to get hoisted into globals, moving them out of the inference loop.

SiLU Fusion in IREE’s CPU Codegen

IREE fuses the SiLU activation directly into the matmul dispatch, the matmul part being lowered to mmt4d ukernel with a 7,32,1 (M,N,K) tile for f32, so SiLU operates on a vector<7x32xf32> matching the mmt4d’s per-panel output. The critical step is how math.exp gets lowered, handled by a dedicated codegen pass called iree-codegen-math-transform that approximates the exponent function, as shown in Figure 2 below:

Figure 2: Exponent approximation as defined by the iree-codegen-math-transform pass

This fusion comes with a tradeoff: computing n, r, 2^n, and e^r across the 256-bit VLEN tile needs 28 live registers, pushing past the 32-register limit and forcing register spills, as confirmed by the generated assembly. Shrinking the mmt4d’s ukernel tile to 4x32x1 doesn’t help, since it sacrifices tile reuse and runs slower (691.0ms vs 598.0ms on 8 threads), while skipping fusion altogether is slowest of all (741.7ms), since it requires materializing an intermediate buffer and increases memory traffic. So, based on these experiments, we concluded that it is better to leave SILU codegen as it is.

Pack Op Vectorization

Another optimization area was the lowering of the pack operations introduced by IREE’s data-tiling pipeline. Ideally, these pack operations should remain as linalg.pack long enough to be vectorized; however, the behavior was different for some dispatches. IREE’s CombineLayoutTransformationPass combines layout and indexing transformation operations at the boundaries of a dispatch, and in these cases, it rewrote the pack layout transformation into iree_linalg_ext.map_scatter (renamed map_store in newer IREE versions). This was problematic because IREE’s compilation pipeline didn’t support map_scatter/map_store vectorization, so the pack operation fell back to scalar code, leading to poor performance.

Pack Op Before CombineLayoutTransformation Pass:

%pack = linalg.pack %collapsed outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [32, 1] into %5 : tensor<27x102400xf32> -> tensor<3200x27x32x1xf32>

After CombineLayoutTransformation Pass:

%9 = iree_linalg_ext.map_store %8 into %arg2 {
^bb0(%arg3: index, %arg4: index, %arg5: index, %arg6: index, %arg7: index):
%10 = affine.apply affine_map<(d0, d1) -> (d0 + d1)>(%arg6, %arg0)
%11 = affine.apply affine_map<(d0, d1) -> (d0 + d1)>(%arg7, %arg1)
%12 = affine.linearize_index disjoint [%arg3, %arg4, %arg5, %10, %11] by (3, 3, 3, 320, 320) : index
%13:2 = affine.delinearize_index %12 into (27, 102400) : index, index
%14 = affine.linearize_index disjoint [%13#0, %13#1] by (27, 102400) : index
%15:4 = affine.delinearize_index %14 into (27, 1, 3200, 32) : index, index, index, index
iree_linalg_ext.yield %15#2, %15#0, %15#3, %15#1, %true : index, index, index, index, i1
} : tensor<3x3x3x32x32xf32> into tensor<3200x27x32x1xf32> -> tensor<3200x27x32x1xf32>

We initially explored fixing this by enabling vectorized pack ukernels for FP32, FP16, and INT8, for RISC-V. However, this path did not solve the issue, because by the time the CPULowerToUKernelsPass runs in the IREE compilation pipeline, the fused linalg.pack has already been decomposed into map_store by the CombineLayoutTransformationPass, leaving no pack ops to match, so our ukernels were never selected (tracked in iree-org/iree#23920).

The fix came from an upstream patch (merged in iree-org/iree#24485) that sinks the reshape ops across the packing ops, fusing the linalg.pack into the dispatch bindings wherever possible, so it stays on the standard vectorized codegen path rather than degrading into map_scatter/map_store. Because this restored vectorization at the source of the problem and avoided maintaining separate pack ukernels for every precision and tile shape, we favored the upstream vectorization-based approach and dropped the pack ukernel route.

Implementation and Optimization of Pre- and Post-Processing Pipelines

The pre- and post-processing pipelines for YOLOv8n were implemented in C++ using OpenCV’s C++ API.

Pre-Processing

The pre-processing pipeline consists of the following operations: image loading, letterbox resizing to fit the model’s expected input dimensions, normalization of pixel values to the [0, 1] range, and finally a layout conversion from HWC to NCHW, which is the format expected by the model.

Two optimizations were applied to this pipeline. First, selected OpenCV operations were replaced with more optimized manual implementations to reduce overhead. Second, the HWC to NCHW layout conversion was vectorized using RVV instructions, since it accounted for a significant portion of pre-processing runtime.

Post-Processing

The post-processing pipeline takes the raw model output and applies confidence thresholding to filter out low-confidence detections, performs Non-Maximum Suppression (NMS) to eliminate duplicate bounding boxes, and finally draws the resulting bounding boxes over the image.

The main bottleneck in post-processing was confidence filtering, which was vectorized using RVV instructions. Additionally, for the FP16 model, input and output casting operations were also vectorized using RVV to reduce conversion overhead.

Figure 3: YOLOv8n end-to-end runtime flow

Results

All benchmarking was performed on the Banana Pi BPI-F3, an 8-core RISC-V vector CPU board with a vector length of 256 bits.

Model Inference Latency

The most significant gains were observed at INT8 precision. Without LLVM auto-vectorization, single-threaded inference improved from 36023ms to 2082ms, a 17.3x speedup, and multi-threaded inference improved from 5428ms to 421ms, a 12.9x speedup. With LLVM auto-vectorization enabled, the gains were comparatively lower since the baseline was already partially optimized, with single-threaded and multi-threaded speedups of 5.32x and 4.7x, respectively.

At FP16 and FP32 precisions, without LLVM auto-vectorization, FP16 achieved a 1.20x single-threaded and 1.04x multi-threaded speedup, while FP32 achieved a 1.11x single-threaded and 1.04x multi-threaded speedup. With LLVM auto-vectorization enabled, the relative speedups were smaller since the baseline already benefits from auto-vectorization, with FP16 achieving 1.05x single-threaded and 1.06x multi-threaded inference speedup, and FP32 achieving 1.04x single-threaded and 1.01x multi-threaded inference speedup.

Type	Upstream IREE Inference Time (ms) (1Thread)	RISE IREE Inference Time (ms) (1 Thread)	Speed-up	Upstream IREE Inference Time (ms) (8 Threads)	RISE IREE Inference Time (ms) (8 Threads)	Speed-up
INT8	36023	2082	17.30x	5428	421	12.89x
FP16	2224	1860	1.20x	405	389	1.04x
FP32	2604	2346	1.11x	605	582	1.04x

Table 1: Pre- and post-optimization execution times for the YOLOv8n detection model, compiled using IREE at INT8, FP16, and FP32 precisions, for single- and multi- threaded runs. LLVM auto-vectorization was not enabled when benchmarking. Benchmarking was performed on Banana Pi BPI-F3 featuring an 8-core RISC-V vector CPU with VLEN of 256 bits.

Type	Upstream IREE Inference Time (ms) (1 Thread)	RISE IREE Inference Time (ms) (1 Thread)	Speed-up	Upstream IREE Inference Time (ms) (8 Threads)	RISE IREE Inference Time (ms) (8 Threads)	Speed-up
INT8	9601	1805	5.32x	1777	378	4.7x
FP16	1758	1673	1.05x	344	325	1.06x
FP32	2204	2119	1.04x	560	552	1.01x

Table 2: Pre- and post-optimization execution times for the YOLOv8n detection model, compiled using IREE, with LLVM auto-vectorization enabled, at INT8, FP16, and FP32 precisions, for single- and multi-threaded runs on Banana Pi BPI-F3.

Pre- and Post-Processing Latency

Phase	Pre-Optimization (ms)	Post-Optimization (ms)	Speed-up
Pre-Processing	45.1	27.7	1.63x
Input Casting and other minor operations	45.7	4.9	9.3x
Output Casting and other minor operations	19.9	3.5	5.7x
Post-Processing	30.18	18.97	1.59x

Table 3: Pre- and post-optimization latencies for pre-processing, post-processing, casting, and other minor operations on Banana Pi BPI-F3. The input and output casting operations are only needed for the model at FP16 precision.

Accuracy

For accuracy validation, mean Average Precision (mAP) was computed across IoU thresholds from 0.50 to 0.95 using the COCO validation dataset. At FP32, we achieved an mAP of 37.3, which matches the value publicly reported by Ultralytics, confirming that the IREE compiled model is numerically correct. At FP16 and INT8, mAP values of 37.29 and 35.79 were observed, respectively, both within an acceptable range.

Precision	mAP val(50-95) (Post optimizations)
FP32	37.30
FP16	37.29
INT8	35.79

Table 4: mean Average Precision (mAP) across IoU thresholds from 0.50 to 0.95, using the COCO validation dataset. Accuracy Validation was performed on Banana Pi BPI-F3.

Conclusion and Future Work

This work optimizes end-to-end object detection model inference on RISC-V. It optimizes IREE compilation for vision models, achieving ~5.32x latency improvement for YOLOv8n at INT8 precision, along with 4% and 6% gains at FP32 and FP16, respectively. Accuracy validation confirms mAP parity with reported benchmarks. These contributions strengthen RISC-V’s AI software ecosystem and bring it closer to X86 and ARM performance levels.

Looking ahead, the scope of IREE support for RISC-V should expand beyond vision models to cover the latest Gen-AI models. This includes novel architecture families like LFM2.5, as well as multimodal models like Qwen3.5 and Gemma4, which represent the direction the field is moving in. Enabling these models on RISC-V vector hardware via IREE would be a meaningful step toward making RISC-V a viable platform for modern AI workloads.

Additionally, establishing an auto-tuning workflow for RISC-V would significantly accelerate this effort. Currently, tile sizes for expensive ops like matmul and convolution are tuned manually, which is time-consuming. An automated workflow that explores the tile size search space and identifies optimal configurations for a given hardware target would make it much faster to bring up and optimize new models on RISC-V.

Acknowledgements

We would like to thank the RISE team, and Hong-Rong Hsu, in particular, for his technical insights and support throughout the project.

Optimizing IREE Compilation and End-to-End Object Detection Pipeline for RISC-V