Hi ​@sara!

Hmm, yeah I think that number (2 seconds) is end-to-end, so that include everything from network speed to buffering, to decoding on the host and inference/post processing time.

I wonder if we can do some tests to try and figure out exactly where the bottleneck is? Like, maybe (temporarily) replacing the RTSP feed with a local video file? That might give us a clue as to whether the latency is on the feed, or in processing.

(venv) ~/voyager-sdk$ ./inference.py yolov8s-coco-onnx media/traffic1_480p.mp4 --show-stream-timing --show-stats
=========================================================================                                                                                                                                          
Element                                          Time(𝜇s)   Effective FPS
=========================================================================
qtdemux0                                              122         8,133.9
h264parse0                                            236         4,228.0
capsfilter0                                            80        12,427.9
decodebin-link0                                       103         9,639.1
axtransform-colorconvert0                          54,198            18.5
inference-task0:libtransform_resize_cl_0            1,384           722.3
inference-task0:libtransform_padding_0              2,239           446.5
inference-task0:inference                          50,148            19.9
inference-task0:Inference latency               1,207,065             n/a
inference-task0:libdecode_yolov8_0                  3,422           292.2
inference-task0:libinplace_nms_0                       27        35,793.8
inference-task0:Postprocessing latency              6,842             n/a
inference-task0:Total latency                   1,497,217             n/a
=========================================================================
End-to-end average measurement                                       18.0
=========================================================================
(venv) ~/voyager-sdk$ ./inference.py yolov8s-coco-onnx media/traffic1_480p.mp4 --show-stream-timing 
INFO    : Core Temp : 39.0°C                                                                                                                                                                                       
INFO    : CPU % : 14.3%
INFO    : End-to-end : 20.8fps
INFO    : Jitter : 87.0ms
INFO    : Latency : 1867.4ms

Hmm your fps seems incredibly low for yolov8s as well. What kind of host are you using? And which Metis product? Are you getting any warning?

If I run yolov8s on a single m.2 device and an x86 i5, I get around 390 fps.

Hi ​@sara ,

If I remember correctly you were using Raspberry Pi 5, correct?

If so, please see the following info from https://support.axelera.ai/hc/en-us/articles/26362016484114-Bring-up-Voyager-SDK-in-Raspberry-Pi-5 : 

If you see that the performance is lower than expected, this is something that can be solved.

The solution is setting AXELERA_USE_DOUBLE_BUFFER=0 at the beginning of the inference.py command. For example:

AXELERA_USE_DOUBLE_BUFFER=0 ./inference.py yolov8s-coco-onnx ./media/traffic1_480p.mp4 

For reference, by setting AXELERA_USE_DOUBLE_BUFFER=0, yolov8s-coco-onnx improved from 5.2 FPS end-to-end (which is a bug) to 80.3 FPS end-to-end, which is expected.

Note that this is an existing bug we are investigating. However, with the workaround I shared, performance is almost not affected.

Please, let us know if that solves the issue.

I also recommend you to check, just in case, the section “Note about MVM utilisation limit” in https://support.axelera.ai/hc/en-us/articles/26362016484114-Bring-up-Voyager-SDK-in-Raspberry-Pi-5.

Best,

Victor

Hi ​@sara, not precisely what you're asking for, but this latency can be reduced significantly if you want to. This latency of 2s is added “artificially” to allow some buffering for a smoother stream. If you are in a controlled environment (private, dedicated network) 40ms should be enough to keep a smooth stream.

Check the RTSP Latency section in the Voyager reference:

https://github.com/axelera-ai-hub/voyager-sdk/blob/release/v1.3/docs/tutorials/application.md#rtsp-latency

Internally, this configures the “latency” property in the rtspsrc GStreamer element.

My two cents:

- Michael

https://ridgerun.ai