Voyager-sdk on singularity

Question

Dear all,

I am trying to set up the metis PCIe accelerator on a intel workstation. Later we will move to a full risc-v embedded setup.

In out lab workstation we are forced to use singularity instead of docker, so i am trying to translate the docker container into a singularity one.

I think i am very close to make everything work. However, i am now stuck with a problem i can’t figure how to solve. in particular, i get a weird error when i try to run a simple ./inference.py with one of the pre-trained models.

Below the full description on how i create, run the container, and how i run the example from the voyager sdk

my `voyager-sdk-1.2.5.def` file

Bootstrap: docker
From: ubuntu:22.04

%labels
    Author You
    Version voyager-sdk-1.2.5

%environment
    export DISPLAY=$DISPLAY

%post
    apt-get update && apt-get install -y \
        sudo \
        git \
        pciutils \
        lsb-release \
        x11-utils
        curl \
        wget \
        gnupg2 \
        lsb-release \
        software-properties-common \
        xauth \
        libgl1-mesa-glx \
        libxrender1 \
        libxtst6 \
        libxi6 \
        && apt-get clean \
        && touch /dev/metis0 \
        && update-pciids

    echo "eINFO] Cloning Voyager SDK repo..."
    git clone https://github.com/axelera-ai-hub/voyager-sdk.git

    echo "eINFO] Running Voyager SDK install script..."
    cd voyager-sdk
    ./install.sh --YES --no-driver --media --user llamberti@iis.ee.ethz.ch --token <whatever>
    echo "Voyager SDK base container setup complete."

%runscript
    echo "Starting Voyager SDK container shell..."
    exec /bin/bash

i built the .sif file like this

sudo singularity build voyager-sdk-1.2.5.sif voyager-sdk-1.2.5.def

and i run the container like this:

singularity shell \
  --fakeroot \
  --writable-tmpfs \
  --no-home \
  --home /scratch/llamber/singularity-home:/root \
  --bind /dev/dri:/dev/dri \
  --bind /lib/modules:/lib/modules \
  --bind /scratch/llamber/tmp:/tmp \
  --bind /run:/run \
  --bind "$HOME/.Xauthority:/root/.Xauthority" \
  --env DISPLAY=$DISPLAY \
  voyager-sdk-1.2.5.sif

i source the environment

source /root/.cache/axelera/venvs/93f45ae3/bin/activate
export AXELERA_FRAMEWORK=/scratch/llamber/singularity-workspace-axelera/voyager-sdk

last, i try to run a simple inference

./inference.py yolov5s-v7-barrel-onnx media/traffic1_1080p.mp4 -vv --disable-opencl --disable-opengl --no-display -vv

and i get the following error

ERROR   : RuntimeError: Failed to create element of type axinplace (axinplace-addstreamid0)

below the full log. Do you guys know why this function call is failing? my guess is that it could be trying to write somewhere not writeable within the singularity container, but i could not trace down the error..

Do you have any guess on what i should check?

I tried to see if i can find this axinplace, but the following commands return me nothing.

>find / -name "libaxinplace.so" 2>/dev/null

>gst-inspect-1.0 axinplace

No such element or plugin 'axinplace'

Thanks a lot if you have the patience to read!

full error log:

(venv) Singularity> ./inference.py yolov5s-v7-barrel-onnx media/traffic1_1080p.mp4 -vv --disable-opencl --disable-opengl --no-display -vv
DEBUG   :axelera.app.inf_tracers: HostTracer will write to /tmp/tmp1o1998fj
INFO    :axelera.app.device_manager: Using device metis-0:6:0
DEBUG   :axelera.app.network: Create network from ax_models/reference/image_preprocess/yolov5s-v7-barrel-onnx.yaml
TRACE   :axelera.app.utils: Changing weight_path from weights/yolov5s-v7.onnx to /root/.cache/axelera/weights/yolov5s-v7-coco-onnx/yolov5s-v7.onnx
TRACE   :axelera.app.utils: Changing template_path from $AXELERA_FRAMEWORK/pipeline-template/yolo-letterbox.yaml to /scratch/llamber/singularity-workspace-axelera/voyager-sdk/pipeline-template/yolo-letterbox.yaml
TRACE   :axelera.app.utils: Changing class_path from $AXELERA_FRAMEWORK/ax_models/base_onnx.py to /scratch/llamber/singularity-workspace-axelera/voyager-sdk/ax_models/base_onnx.py
TRACE   :axelera.app.utils: Changing weight_path from weights/yolov5s-v7.onnx to /scratch/llamber/singularity-workspace-axelera/voyager-sdk/ax_models/reference/image_preprocess/weights/yolov5s-v7.onnx
TRACE   :axelera.app.utils: Changing anchors_path from training_yamls/yolov5s.yaml to /scratch/llamber/singularity-workspace-axelera/voyager-sdk/ax_models/reference/image_preprocess/training_yamls/yolov5s.yaml
TRACE   :axelera.app.utils: Changing class_path from $AXELERA_FRAMEWORK/ax_datasets/objdataadapter.py to /scratch/llamber/singularity-workspace-axelera/voyager-sdk/ax_datasets/objdataadapter.py
TRACE   :axelera.app.utils: Changing labels_path from $AXELERA_FRAMEWORK/ax_datasets/labels/coco.names to /scratch/llamber/singularity-workspace-axelera/voyager-sdk/ax_datasets/labels/coco.names
TRACE   :axelera.app.utils: Changing repr_imgs_dir_path from $AXELERA_FRAMEWORK/data/coco2017_400_b680128 to /scratch/llamber/singularity-workspace-axelera/voyager-sdk/data/coco2017_400_b680128
TRACE   :axelera.app.utils: Changing template_path from $AXELERA_FRAMEWORK/pipeline-template/yolo-letterbox.yaml to /scratch/llamber/singularity-workspace-axelera/voyager-sdk/pipeline-template/yolo-letterbox.yaml
TRACE   :axelera.app.utils: Changing class_path from $AXELERA_FRAMEWORK/ax_models/base_onnx.py to /scratch/llamber/singularity-workspace-axelera/voyager-sdk/ax_models/base_onnx.py
TRACE   :axelera.app.utils: Changing anchors_path from training_yamls/yolov5s.yaml to /scratch/llamber/singularity-workspace-axelera/voyager-sdk/ax_models/reference/image_preprocess/training_yamls/yolov5s.yaml
TRACE   :axelera.app.utils: Changing class_path from $AXELERA_FRAMEWORK/ax_datasets/objdataadapter.py to /scratch/llamber/singularity-workspace-axelera/voyager-sdk/ax_datasets/objdataadapter.py
TRACE   :axelera.app.utils: Changing labels_path from $AXELERA_FRAMEWORK/ax_datasets/labels/coco.names to /scratch/llamber/singularity-workspace-axelera/voyager-sdk/ax_datasets/labels/coco.names
TRACE   :axelera.app.utils: Changing repr_imgs_dir_path from $AXELERA_FRAMEWORK/data/coco2017_400_b680128 to /scratch/llamber/singularity-workspace-axelera/voyager-sdk/data/coco2017_400_b680128
TRACE   :axelera.app.utils: Changing class_path from $AXELERA_FRAMEWORK/ax_models/decoders/yolo.py to /scratch/llamber/singularity-workspace-axelera/voyager-sdk/ax_models/decoders/yolo.py
DEBUG   :axelera.app.network: Register custom operator 'decodeyolo' with class DecodeYolo from /scratch/llamber/singularity-workspace-axelera/voyager-sdk/ax_models/decoders/yolo.py
TRACE   :axelera.app.network: 1. build/yolov5s-v7-barrel-onnx/yolov5s-v7-coco-onnx/model_info.json: Available
TRACE   :axelera.app.network: 1. build/yolov5s-v7-barrel-onnx/yolov5s-v7-coco-onnx/1/manifest.json: Available
DEBUG   :axelera.app.device_manager: Reconfiguring devices with ddr_size=2.0GB, device_firmware=1, mvm_utilisation_core_0=100%, clock_profile_core_0=800MHz, mvm_utilisation_core_1=100%, clock_profile_core_1=800MHz, mvm_utilisation_core_2=100%, clock_profile_core_2=800MHz, mvm_utilisation_core_3=100%, clock_profile_core_3=800MHz
DEBUG   :axelera.app.inf_tracers: Host FPS multiplier: 4
DEBUG   :axelera.app.utils: $ vainfo
INFO    :axelera.app.utils: Could not exec vainfo: Command ' 'vainfo']' returned non-zero exit status 3.
DEBUG   :axelera.app.utils: libva info: VA-API version 1.14.0
DEBUG   :axelera.app.utils: libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
DEBUG   :axelera.app.utils: vaInitialize failed with error code -1 (unknown libva error),exit
INFO    :axelera.app.pipe.manager: Network type: NetworkType.SINGLE_MODEL
INFO    :axelera.app.pipe.manager: Input
INFO    :axelera.app.pipe.manager:   └─detections
TRACE   :axelera.app.transforms: composite_expansion transformed:
TRACE   :axelera.app.transforms:   TorchToTensor(input_layout='NHWC', output_layout='NCHW', datatype='float32', scale=True)
TRACE   :axelera.app.transforms:  to:
TRACE   :axelera.app.transforms:   ToTensor()
TRACE   :axelera.app.transforms:   PermuteChannels(input_layout=<TensorLayout.NHWC: 2>, output_layout=<TensorLayout.NCHW: 1>)
TRACE   :axelera.app.transforms:   TypeCast(datatype='float32')
TRACE   :axelera.app.transforms:   Normalize(mean='0', std='255.0', tensor_layout=<TensorLayout.NCHW: 1>, format='RGB')
TRACE   :axelera.app.transforms: ax_letterbox_to_tensor_and_inplace transformed:
TRACE   :axelera.app.transforms:   Letterbox(height=640, width=640, scaleup=True, half_pixel_centers=False, pad_val=114, image_width=1000000, image_height=1000000)
TRACE   :axelera.app.transforms:   ToTensor()
TRACE   :axelera.app.transforms:   PermuteChannels(input_layout=<TensorLayout.NHWC: 2>, output_layout=<TensorLayout.NCHW: 1>)
TRACE   :axelera.app.transforms:   TypeCast(datatype='float32')
TRACE   :axelera.app.transforms:   Normalize(mean='0', std='255.0', tensor_layout=<TensorLayout.NCHW: 1>, format='RGB')
TRACE   :axelera.app.transforms:  to:
TRACE   :axelera.app.transforms:   LetterboxToTensorAndNormalise(height=640, width=640, mean=d0.0], std=a1.0], datatype='float32', scaleup=True)
DEBUG   :yolo: Model Type: YoloFamily.YOLOv5 (YOLOv5 pattern:
DEBUG   :yolo: - 3 output tensors (anchor-based)
DEBUG   :yolo: - Each output has 255 channels
DEBUG   :yolo:   = 3 anchors × (4 box + 1 obj + 80 classes)
DEBUG   :yolo:   = 3 × (4 + 1 + 80) = 255
DEBUG   :yolo: - Shapes: yo1, 40, 40, 255], ,1, 20, 20, 255], ,1, 80, 80, 255]])
TRACE   :axelera.app.pipe.manager: Parse input and output options
DEBUG   :axelera.app.pipe.io: FPS of /root/.cache/axelera/media/traffic1_1080p.mp4: 60
DEBUG   :axelera.app.operators.inference: Enabled 4x1 inference queues for yolov5s-v7-coco-onnx because model_cores=1 and num_cores=4
DEBUG   :axelera.app.operators.inference: Using inferencenet name=inference-task0 model=/scratch/llamber/singularity-workspace-axelera/voyager-sdk/build/yolov5s-v7-barrel-onnx/yolov5s-v7-coco-onnx/1/model.json devices=metis-0:6:0 double_buffer=True dmabuf_inputs=True dmabuf_outputs=True num_children=4
DEBUG   :axelera.app.pipe.gst: GST representation written to build/yolov5s-v7-barrel-onnx/logs/gst_pipeline.yaml
TRACE   :axelera.app.pipe.gst: environment at gst pipeline construction:                                                                                                                                                             
TRACE   :axelera.app.pipe.gst: {'AXELERA_DEVICE_DIR': '/opt/axelera/device-1.2.5-1/omega',
TRACE   :axelera.app.pipe.gst:  'AXELERA_EXAMPLES_DIR': '/opt/axelera/compiler-examples-1.2.5-1',
TRACE   :axelera.app.pipe.gst:  'AXELERA_FRAMEWORK': '/scratch/llamber/singularity-workspace-axelera/voyager-sdk',
TRACE   :axelera.app.pipe.gst:  'AXELERA_RISCV_TOOLCHAIN_DIR': '/opt/axelera/riscv-gnu-newlib-toolchain-409b951ba662-7',
TRACE   :axelera.app.pipe.gst:  'AXELERA_RUNTIME_DIR': '/opt/axelera/runtime-1.2.5-1',
TRACE   :axelera.app.pipe.gst:  'AXE_LZ_PERF_LOG_PATH': '/tmp/tmp1o1998fj',
TRACE   :axelera.app.pipe.gst:  'AXE_PROFILING_CONFIG': 'HOST_TEMPLATE'}
TRACE   :axelera.app.pipe.gst_helper: Setting property filesrc0.location=/root/.cache/axelera/media/traffic1_1080p.mp4                                                                                                               
TRACE   :axelera.app.pipe.gst_helper: Creating filesrc0 from filesrc                                                                                                                                                                 
TRACE   :axelera.app.pipe.gst_helper: Setting property queue_in0.force-sw-decoders=True                                                                                                                                              
TRACE   :axelera.app.pipe.gst_helper: Setting property queue_in0.caps=video/x-raw(ANY)                                                                                                                                               
TRACE   :axelera.app.pipe.gst_helper: Setting property queue_in0.expose-all-streams=False                                                                                                                                            
TRACE   :axelera.app.pipe.gst_helper: Creating queue_in0 from decodebin                                                                                                                                                              
ERROR   : Failed to create element of type axinplace (axinplace-addstreamid0)                                                                                                                                                        
ERROR   : Traceback (most recent call last):
ERROR   :   File "/scratch/llamber/singularity-workspace-axelera/voyager-sdk/./inference.py", line 93, in <module>
ERROR   :     stream = init(args, tracers)
ERROR   :   File "/scratch/llamber/singularity-workspace-axelera/voyager-sdk/./inference.py", line 40, in init
ERROR   :     return InferenceStream(pipemanager, frames=args.frames, timeout=args.timeout)
ERROR   :   File "/scratch/llamber/singularity-workspace-axelera/voyager-sdk/axelera/app/stream.py", line 129, in __init__
ERROR   :     self.pipe_mgr.init_pipe()
ERROR   :   File "/scratch/llamber/singularity-workspace-axelera/voyager-sdk/axelera/app/pipe/manager.py", line 404, in init_pipe
ERROR   :     self._pipeline.init()
ERROR   :   File "/scratch/llamber/singularity-workspace-axelera/voyager-sdk/axelera/app/pipe/base.py", line 64, in init
ERROR   :     self._loopfn = self.init_loop()
ERROR   :   File "/scratch/llamber/singularity-workspace-axelera/voyager-sdk/axelera/app/pipe/gst.py", line 661, in init_loop
ERROR   :     self.pipeline_cache = gst_helper.build_gst_pipelines(
ERROR   :   File "/scratch/llamber/singularity-workspace-axelera/voyager-sdk/axelera/app/pipe/gst_helper.py", line 403, in build_gst_pipelines
ERROR   :     raise RuntimeError(f"Failed to create element of type {instance} ({name})")
ERROR   : RuntimeError: Failed to create element of type axinplace (axinplace-addstreamid0)

here’s my env variables:

🔎 Voyager SDK Environment Variables
------------------------------------
AXELERA_FRAMEWORK:           /scratch/llamber/singularity-workspace-axelera/voyager-sdk
AXELERA_DEVICE_DIR:          /opt/axelera/device-1.2.5-1/omega
AXELERA_RUNTIME_DIR:         /opt/axelera/runtime-1.2.5-1
AXELERA_RISCV_TOOLCHAIN_DIR: /opt/axelera/riscv-gnu-newlib-toolchain-409b951ba662-7
PYTHONPATH:                  /scratch/llamber/singularity-workspace-axelera:/opt/axelera/runtime-1.2.5-1/tvm/tvm-src:/scratch/llamber/singularity-workspace-axelera:/opt/axelera/runtime-1.2.5-1/tvm/tvm-src
LD_LIBRARY_PATH:             /opt/axelera/runtime-1.2.5-1/lib:/scratch/llamber/singularity-workspace-axelera/operators/lib:/opt/axelera/runtime-1.2.5-1/lib:/scratch/llamber/singularity-workspace-axelera/operators/lib:/.singularity.d/libs
HOME:                        /root

Page 1 / 1

Hi Lorenzo,

Usually this error means that the Axelera GST operators have not been compiled. Normally, this is part of the installation process. Would you be able to try running

make clobber-libs && make gst-operators

from inside the Voyager SDK container and virtual environment (source venv/bin/activate).

If that doesn’t work- please share the logs of trying to compile the operators with this command

Thanks,

David

I have the exact same issue and after running I get this error:

/usr/bin/ld: /lib/x86_64-linux-gnu/libstdc++.so.6: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
>88/141] Building CXX object customers/tu...iles/inplace_tu9.dir/src/inplace_tu9.cc.o
ninja: build stopped: subcommand failed.
makee1]: *** Makefile:54: gst_ops_install] Error 1
makee1]: Leaving directory '/home/aris/Workspace/voyager-sdk/operators'
make: *** Makefile:80: operators] Error 2

Will try chatgpt’s fix and let you know if it works or do you have suggestions?

Reply

might-be-david · Answer

Hi Lorenzo,Usually this error means that the Axelera GST operators have not been compiled. Normally, this is part of the installation process. Would you be able to try runningmake clobber-libs && make gst-operatorsfrom inside the Voyager SDK container and virtual environment (source venv/bin/activate).If that doesn’t work- please share the logs of trying to compile the operators with this commandThanks,David

Reply

Sign up

Log in, or create an Axelera AI account

Login to the community

Log in, or create an Axelera AI account

Scanning file for viruses.

This file cannot be downloaded