Reproducing the Model Zoo benchmarks

After posting my question, I found the solution provide by ​@Steven Hunsche in another post:

This got me through my opencl issue on my AMD Ryzen AI MAX+ 395 PC.
This effectively doubled the throughput achieved (from 350 FPS to 716 FPS).

Still a ways from the 1756 FPS benchmark for ResNet-50 on M.2 Metis.

Here is where I am currently at:

(venv) abbeefai@AlbertaBeefAI:/media/abbeefai/TheExpanse/shared_with_docker/voyager-sdk$ ./inference.py resnet50-imagenet data/coco --enable-opencl --no-display --show-stats
========================================================================                                    
Element                                         Time(𝜇s)   Effective FPS
========================================================================
axinplace-addstreamid0                                10        94,033.6
axtransform-colorconvert-cl0                         323         3,092.8
inference-task0:libtransform_centrecropextra_0
                                                       0     1,357,590.2
inference-task0:libtransform_resize_cl_0              81        12,201.0
inference-task0:libtransform_padding_0                39        25,571.2
inference-task0:inference                          1,415           706.5
inference-task0:Inference latency                 22,083             n/a
inference-task0:libtransform_paddingdequantize_0
                                                       4       238,789.7
inference-task0:libdecode_classification_0             6       160,551.4
inference-task0:Postprocessing latency               614             n/a
inference-task0:Total latency                     25,152             n/a
========================================================================
End-to-end average measurement                                     716.8
========================================================================
Core Temp  : 37.0°C
CPU %      : 5.0%
End-to-end : 716.8fps
Latency    : 19.6ms (min:7.0 max:27.4 σ:1.9 x̄:19.7)ms

Ah, nice work on ramping it up anyway, ​@AlbertaBeef !

In terms of the differences with the benchmarks, a few things that come to mind could be the host system (I think the benchmarks were run on an Intel i9. Not sure about performance difference, but it is an architectural difference, I guess). 

And I think the benchmark numbers are measured against multiple input streams running in parallel to spread the load across all four cores. This might have some interesting directions to test in terms of multiple inputs: https://github.com/axelera-ai-hub/voyager-sdk/blob/release/v1.5/docs/reference/inference.md

Let me know how it goes!