Need Help: RTSP → YOLOv8 Inference → Annotated Video → Live Web Streaming (Axelera PCIe + Voyager SDK)

Hi ​@WGPravin 

At DeGirum (a SW partner of Axelera), we developed PySDK, a python package, that simplifies application development. We made a working script for your use case: axelera_examples/examples/021_rtsp_smart_camera.ipynb at main · DeGirum/axelera_examples. Please see if you find it useful.

Thank you Stashi!

For curiosity I also tried getting this to work using gstreamer, and I got this working.  

First, generate the low level gstreamer pipeline you want using the application framework, 

$ ./inference.py yolov8s-coco media/traffic3_480p.mp4  --save-compiled mypipeline.yaml --frames=1

This yaml is an Axelera internal format we use to define gstreamer pipelines - it’s just a bit easier to read than the usual gst-launch but functionally very similar - which can be converted to a gst-launch using this gist https://gist.github.com/sampartington/9f3ca6c9afc766dd1b76f82c49798c1b

$ create_gst_launch.py mypipeline.yaml
gst-launch-1.0 filesrc location=media/traffic3_480p.mp4 ! \
  decodebin force-sw-decoders=true caps='video/x-raw(ANY)' expose-all-streams=false ! \
  axinplace name=decodebin-link0 lib=libinplace_addstreamid.so mode=meta options=stream_id:0 ! \
  axtransform lib=libtransform_colorconvert_cl.so options=format:rgb ! \
  queue max-size-buffers=4 max-size-time=0 max-size-bytes=0 ! \
  axinferencenet name=inference-task0 model=build/yolov8s-coco/yolov8s-coco/4/model.json devices=metis-0:1:0,metis-0:4:0 double_buffer=true dmabuf_inputs=true dmabuf_outputs=true num_children=0 preprocess0_lib=libtransform_resize_cl.so preprocess0_options='width:640;height:640;padding:114;letterbox:1;scale_up:1;to_tensor:1;mean:0.,0.,0.;std:1.,1.,1.;quant_scale:0.003921568859368563;quant_zeropoint:-128.0' preprocess1_lib=libtransform_padding.so preprocess1_options='padding:0,0,1,1,1,15,0,0;fill:0' preprocess1_batch=4 postprocess0_lib=libdecode_yolov8.so postprocess0_options='meta_key:detections;classes:80;confidence_threshold:0.25;scales:0.07014410197734833,0.06737857311964035,0.07247863709926605,0.10210584104061127,0.14912167191505432,0.17474472522735596;padding:0,0,0,0,0,0,0,0|0,0,0,0,0,0,0,0|0,0,0,0,0,0,0,0|0,0,0,0,0,0,0,48|0,0,0,0,0,0,0,48|0,0,0,0,0,0,0,48;zero_points:-67,-58,-43,134,114,104;topk:30000;multiclass:0;classlabels_file:/tmp/tmpky6k4ld2;model_width:640;model_height:640;scale_up:1;letterbox:1' postprocess0_mode=read postprocess1_lib=libinplace_nms.so postprocess1_options='meta_key:detections;max_boxes:300;nms_threshold:0.45;class_agnostic:1;location:CPU' ! \
  axinplace lib=libinplace_draw.so mode=write ! \
  videoconvert ! \
  x264enc ! \
  mp4mux ! \
  filesink location=output_video.mp4

To run this you need to set GST_PLUGIN_PATH=$PWD/operators/lib 

I’d test this is working properly, then edit the launch to use an rtsp or usb source (or rerun app framework with the desired source to regenerate the low level gst yaml) and edit the last two elements to be mpegtsmux ! hlssink 

For the web server I just used python -m http.server --dir=$PWD running in a different terminal. Here it is running in my browser