Problems with network inference. Uio wait kernel failed with return code -1406.

Hi there ​@IAK, welcome on board!

Hmm, since you’ve seen this on both the Raspberry Pi and your laptop, it might be related to the runtime environment in Docker rather than the board firmware?

First quick thing to try: can you run this command inside your Docker container and share the output?

glxinfo | grep "OpenGL renderer"

This will help confirm whether OpenGL ES is available and functioning properly, which is required for inference display on the Pi5. If it’s working, missing or misconfigured, we can go from there 👍

Hi ​@Spanner 

Thanks, glad to be here!
After running the glxinfo | grep "OpenGL renderer" command inside my docker I have the following output:
⬢ [Docker] ❯ glxinfo | grep "OpenGL renderer"
OpenGL renderer string: V3D 7.1

Guess it is configured fine, isn’t it?

Ah, that’s good news! Well, not good news that you’re having an issue, but it’s good that we’ve confirmed that OpenGL is working 😄 Thanks for the info.

Maybe we should check that the Metis card is being recognised correctly? If you could run the following command inside the container and share the output:

axdevice

That should give us some info about its status, firmware, etc.

Hello ​@IAK 

May I ask which RPi HAT are you using?

I also recommend you to take a look at our troubleshooting guide in https://support.axelera.ai/hc/en-us/articles/26261515755538-PCIe-Troubleshooting-Guide

Regards,

Victor

axdevice

Facing the same issue.
Device 0: metis-1:1:0 board_type=m2 fwver='v1.2.5' clock=800MHz(0-3:800MHz) mvm=0-3:100%

using Waveshare pcie to m.2 hat
https://thepihut.com/collections/waveshare/products/pcie-to-m-2-adapter-for-raspberry-pi-5
 

OUTPUT OF dmesg -T
Tue May 27 14:49:08 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:49:09 2025] axl 0001:01:00.0: DMA WR CH0 queue timeout (00000000f2589e77 status 0)
[Tue May 27 14:49:09 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:49:10 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:49:11 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:49:12 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:49:13 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:49:42 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:49:43 2025] axl 0001:01:00.0: DMA WR CH0 queue timeout (00000000fdc5a1e2 status 0)
[Tue May 27 14:49:43 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:49:44 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:49:45 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:49:46 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:49:47 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:55:52 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:55:53 2025] axl 0001:01:00.0: DMA WR CH0 queue timeout (00000000cf307059 status 0)
[Tue May 27 14:55:53 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:55:54 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:55:55 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:55:56 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:55:57 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:56:19 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:56:20 2025] axl 0001:01:00.0: DMA WR CH0 queue timeout (00000000c592c395 status 0)
[Tue May 27 14:56:20 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:56:21 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:56:22 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:56:23 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:56:24 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 14:58:11 2025] axl 0001:01:00.0: Unregister directory 0001:01:00.0
[Tue May 27 14:58:11 2025] axl 0001:01:00.0: Unregistered triton-1:1:0 (0 0)
[Tue May 27 14:58:11 2025] axl 0001:01:00.0: Release dma mem triton-1:1:0
[Tue May 27 14:58:11 2025] pci_bus 0001:01: busn_res: [bus 01] is released
[Tue May 27 14:58:12 2025] triton: debugfs root directory triton removed 
[Tue May 27 14:58:13 2025] pci 0001:00:00.0: [14e4:2712] type 01 class 0x060400 PCIe Root Port
[Tue May 27 14:58:13 2025] pci 0001:00:00.0: PCI bridge to [bus 01]
[Tue May 27 14:58:13 2025] pci 0001:00:00.0:   bridge window [mem 0x1b80000000-0x1b82ffffff]
[Tue May 27 14:58:13 2025] pci 0001:00:00.0: PME# supported from D0 D3hot
[Tue May 27 14:58:13 2025] brcm-pcie 1000110000.pcie: clkreq-mode set to safe
[Tue May 27 14:58:13 2025] brcm-pcie 1000110000.pcie: link up, 8.0 GT/s PCIe x1 (!SSC)
[Tue May 27 14:58:13 2025] pci 0001:01:00.0: [1f9d:1100] type 00 class 0x120000 PCIe Endpoint
[Tue May 27 14:58:13 2025] pci 0001:01:00.0: BAR 0 [mem 0x1b82010000-0x1b82010fff 64bit]
[Tue May 27 14:58:13 2025] pci 0001:01:00.0: BAR 2 [mem 0x1b80000000-0x1b81ffffff]
[Tue May 27 14:58:13 2025] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff pref]
[Tue May 27 14:58:13 2025] pci 0001:01:00.0: supports D1
[Tue May 27 14:58:13 2025] pci 0001:01:00.0: PME# supported from D0 D1 D3hot
[Tue May 27 14:58:13 2025] pci 0001:01:00.0: 7.876 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x1 link at 0001:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[Tue May 27 14:58:13 2025] pci 0001:00:00.0: bridge window [mem 0x1b80000000-0x1b82ffffff]: assigned
[Tue May 27 14:58:13 2025] pci 0001:01:00.0: BAR 2 [mem 0x1b80000000-0x1b81ffffff]: assigned
[Tue May 27 14:58:13 2025] pci 0001:01:00.0: ROM [mem 0x1b82000000-0x1b8200ffff pref]: assigned
[Tue May 27 14:58:13 2025] pci 0001:01:00.0: BAR 0 [mem 0x1b82010000-0x1b82010fff 64bit]: assigned
[Tue May 27 14:58:13 2025] pci 0001:00:00.0: PCI bridge to [bus 01]
[Tue May 27 14:58:13 2025] pci 0001:00:00.0:   bridge window [mem 0x1b80000000-0x1b82ffffff]
[Tue May 27 14:58:13 2025] pcieport 0001:00:00.0: PME: Signaling with IRQ 39
[Tue May 27 14:58:13 2025] pcieport 0001:00:00.0: AER: enabled with IRQ 39
[Tue May 27 14:58:13 2025] pci 0001:01:00.0: Found target device: TRITON_OMEGA_DEVICE_ID
[Tue May 27 14:58:13 2025] pci 0001:01:00.0: Found target device: 0001:01:00.0
[Tue May 27 14:58:13 2025] pcieport 0001:00:00.0: Found bridge device: 0001:00:00.0
[Tue May 27 14:58:13 2025] pcieport 0001:00:00.0: Bridge window: [mem 0x1b80000000-0x1b82ffffff]
[Tue May 27 14:58:13 2025] pcieport 0001:00:00.0: Decoded memory behind bridge: 1b80000000-1b82ffffff
[Tue May 27 14:58:13 2025] pcieport 0001:00:00.0: Memory behind bridge is sufficient. Skipping reset.
[Tue May 27 14:58:13 2025] triton: root directory for triton
[Tue May 27 14:58:13 2025] axl 0001:01:00.0: MSI registered 32 (32)
[Tue May 27 14:58:13 2025] axl 0001:01:00.0: irq vec number 186
[Tue May 27 14:58:13 2025] axl 0001:01:00.0: Register directory 0001:01:00.0
[Tue May 27 14:58:13 2025] Triton Linux Driver, version 0.07.16, init OK
[Tue May 27 15:02:24 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 15:02:25 2025] axl 0001:01:00.0: DMA WR CH0 queue timeout (000000001768cf1f status 0)
[Tue May 27 15:02:25 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 15:02:26 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 15:02:27 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 15:02:28 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 15:02:29 2025] axl 0001:01:00.0: DMA RD CH0 timeout (irq 4, ctrl 0x3)
[Tue May 27 15:03:54 2025] axl 0001:01:00.0: Unregister directory 0001:01:00.0
[Tue May 27 15:03:54 2025] axl 0001:01:00.0: Unregistered triton-1:1:0 (0 0)
[Tue May 27 15:03:54 2025] axl 0001:01:00.0: Release dma mem triton-1:1:0
[Tue May 27 15:03:54 2025] pci_bus 0001:01: busn_res: [bus 01] is released
[Tue May 27 15:03:55 2025] triton: debugfs root directory triton removed 
[Tue May 27 15:03:56 2025] pci 0001:00:00.0: [14e4:2712] type 01 class 0x060400 PCIe Root Port
[Tue May 27 15:03:56 2025] pci 0001:00:00.0: PCI bridge to [bus 01]
[Tue May 27 15:03:56 2025] pci 0001:00:00.0:   bridge window [mem 0x1b80000000-0x1b82ffffff]
[Tue May 27 15:03:56 2025] pci 0001:00:00.0: PME# supported from D0 D3hot
[Tue May 27 15:03:56 2025] brcm-pcie 1000110000.pcie: clkreq-mode set to safe
[Tue May 27 15:03:56 2025] brcm-pcie 1000110000.pcie: link up, 8.0 GT/s PCIe x1 (!SSC)
[Tue May 27 15:03:56 2025] pci 0001:01:00.0: [1f9d:1100] type 00 class 0x120000 PCIe Endpoint
[Tue May 27 15:03:56 2025] pci 0001:01:00.0: BAR 0 [mem 0x1b82010000-0x1b82010fff 64bit]
[Tue May 27 15:03:56 2025] pci 0001:01:00.0: BAR 2 [mem 0x1b80000000-0x1b81ffffff]
[Tue May 27 15:03:56 2025] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff pref]
[Tue May 27 15:03:56 2025] pci 0001:01:00.0: supports D1
[Tue May 27 15:03:56 2025] pci 0001:01:00.0: PME# supported from D0 D1 D3hot
[Tue May 27 15:03:56 2025] pci 0001:01:00.0: 7.876 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x1 link at 0001:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[Tue May 27 15:03:56 2025] pci 0001:00:00.0: bridge window [mem 0x1b80000000-0x1b82ffffff]: assigned
[Tue May 27 15:03:56 2025] pci 0001:01:00.0: BAR 2 [mem 0x1b80000000-0x1b81ffffff]: assigned
[Tue May 27 15:03:56 2025] pci 0001:01:00.0: ROM [mem 0x1b82000000-0x1b8200ffff pref]: assigned
[Tue May 27 15:03:56 2025] pci 0001:01:00.0: BAR 0 [mem 0x1b82010000-0x1b82010fff 64bit]: assigned
[Tue May 27 15:03:56 2025] pci 0001:00:00.0: PCI bridge to [bus 01]
[Tue May 27 15:03:56 2025] pci 0001:00:00.0:   bridge window [mem 0x1b80000000-0x1b82ffffff]
[Tue May 27 15:03:56 2025] pcieport 0001:00:00.0: PME: Signaling with IRQ 39
[Tue May 27 15:03:56 2025] pcieport 0001:00:00.0: AER: enabled with IRQ 39
[Tue May 27 15:03:56 2025] pci 0001:01:00.0: Found target device: TRITON_OMEGA_DEVICE_ID
[Tue May 27 15:03:56 2025] pci 0001:01:00.0: Found target device: 0001:01:00.0
[Tue May 27 15:03:56 2025] pcieport 0001:00:00.0: Found bridge device: 0001:00:00.0
[Tue May 27 15:03:56 2025] pcieport 0001:00:00.0: Bridge window: [mem 0x1b80000000-0x1b82ffffff]
[Tue May 27 15:03:56 2025] pcieport 0001:00:00.0: Decoded memory behind bridge: 1b80000000-1b82ffffff
[Tue May 27 15:03:56 2025] pcieport 0001:00:00.0: Memory behind bridge is sufficient. Skipping reset.
[Tue May 27 15:03:56 2025] triton: root directory for triton
[Tue May 27 15:03:56 2025] axl 0001:01:00.0: MSI registered 32 (32)
[Tue May 27 15:03:56 2025] axl 0001:01:00.0: irq vec number 186
[Tue May 27 15:03:56 2025] axl 0001:01:00.0: Register directory 0001:01:00.0
[Tue May 27 15:03:56 2025] Triton Linux Driver, version 0.07.16, init OK
 

Hello ​@dev.manek ,

Did you follow the guide in https://www.waveshare.com/wiki/PCIe_TO_M.2_HAT+ ?

Specifically this is very important:

1: Enable PCIe Interface:

The PCIe interface is not enabled by default on PI5B. To enable it, add the following configuration in /boot/firmware/config.txt:dtparam=pciex1

2: PCIE is gen2 by default, if you need to enable PCIE gen3, add it in /boot/firmware/config.txt:

dtparam=pciex1_gen=3

Please enable PCIE gen3 as described in step 2.

At the moment, we have tested Raspberry Pi 5 official M.2 HAT and Seeed Studio Dual M.2 HAT, both working successfully. We know of more users succesfully testing other HATs.

Thank you for the reply I will try it with the seed hat and update you

Is your RPi5 set to PCIe Gen3, ​@dev.manek?

dtparam=pciex1_gen=3

dtparam=pciex1_gen=3

Yes I did. I can confirm that it works with seed dual m.2 hat. But still no stream.
WARNING : Failed to get OpenCL platforms : clGetPlatformIDs failed: PLATFORM_NOT_FOUND_KHR
EDIT:- I am dumb I had no display on. Everything works. Thank you so much guys.

dtparam=pciex1_gen=3

Ah nice work! It’s easy to overlook these things 😄 Just glad it’s up and running!

What’s the project you’re building with it ​@dev.manek?

Hello everyone!

I’m still having this issue. I’ve tried to solve it with this guide: https://support.axelera.ai/hc/en-us/articles/26362016484114-Bring-up-Voyager-SDK-in-Raspberry-Pi-5 and some research but the result is still unsatisfying, this is the output of the inference trials:

(venv) root@iak:/voyager-sdk# AXELERA_OPENGL_BACKEND=gles,3,1 ./inference.py yolov8s-coco-onnx ./media/traffic1_480p.mp4
INFO    : Using device metis-1:1:0
WARNING : Failed to get OpenCL platforms : clGetPlatformIDs failed: PLATFORM_NOT_FOUND_KHR
WARNING : Please check the documentation for installation instructions
INFO    : Default OpenGL backend gl,3,3 overridden, using gles,3,1
INFO    : Network type: NetworkType.SINGLE_MODEL
INFO    : Input
INFO    :   └─detections
Stream Paused:  57%|████████████████████████████████████████                              | 4/7 [00:00<00:00, 106.96/s][ERROR][axeWaitForCommandList]: Uio wait kernel failed with return code -1600.
[ERROR][axeCommandQueueExecuteCommandListsAsync]: Waiting for command lists failed: 0x70010001.
terminate called after throwing an instance of 'std::runtime_error'
  what():  axr_run_model failed with Error at zeCommandQueueExecuteCommandLists(cmdqueue, n_cmdlists, cmdlists, nullptr): cmdqueue_run_cmdlists: 309: Exit with error code: 0x70010001 : ZE_RESULT_ERROR_NOT_AVAILABLE
Aborted (core dumped)
 

So my guess is that I cannot access the OpenCL platform which is needed for inference. 

However when I run the OpenCL demo with gears (the glxgears command from the guide), it runs just fine.
also when I run clinfo command, it shows that I have 0 platforms:
(venv) root@iak:/voyager-sdk# clinfo
Number of platforms                               0
 

Any ideas on the solution directions from here?

Best wishes,

Ivan

We’ll check this with ​@Victor Labian, but I don’t think OpenCL is support on the Pi5, in the Voyager SDK context. I think there’s been some experimentation around enabling it, but it’s not straight forward (or officially supported) yet - I could be wrong though.

During some quick research, I did see that --pipe=torch-aipu has been used to skip OpenCL entirely - might be worth a shot in the meantime? I’ve not tried this (?!) so forgive me if it wastes any time (I don’t have a spare Pi5 to test), but I’m guessing that’d be…

./inference.py yolov8s-coco-onnx ./media/traffic1_480p.mp4 --pipe=torch-aipu

Presumably slower, but might work at least, in the meantime. Let me know how it goes, and we’ll get our Raspberry Ninja (that’s Victor 😆) to check in too 👍

Hello ​@Spanner ​@IAK ,

OpenCL is not officially supported on Raspberry Pi 5. We are exploring possibilities of how to enable OpenCL in RPi 5 and then be able to work on Voyager SDK with it on RPi5 but that is something that we cannot disclosure more about at the moment.

However, OpenCL is not the issue, it is only a warning, as inference should also be able to run without OpenCL.

As I mentioned before to ​@dev.manek we have tested Raspberry Pi 5 official M.2 HAT and Seeed Studio Dual M.2 HAT, both working successfully. For other HATs you might need to read the documentation for the HAT and/or contact the manufacturer. I suggest that you test any of the two HATs we have confirmed that work well.

​@IAK :

• which HAT are you using?
• Can you run inside the (venv) the command axdevice --refresh and share the output with us?

Can you also share the output of running the following:

• uname -r
• cat /sys/class/metis/version
• lsmod | grep metis 

Best,

Victor

Hiya,

I’m on the same team as ​@IAK. We’re running it on the official RPI M.2 HAT+. The result from the commands is:

(venv) root@iak:/voyager-sdk# axdevice --refresh
0001:01:00.0 : Axelera AI Metis AIPU (rev 02)
Device 0: metis-1:1:0 board_type=m2 fwver='v1.2.5' clock=800MHz(0-3:800MHz) mvm=0-3:100%

(venv) root@iak:/voyager-sdk# uname -r
6.12.25+rpt-rpi-2712

(venv) root@iak:/voyager-sdk# cat /sys/class/metis/version
0.07.16

(venv) root@iak:/voyager-sdk# lsmod | grep metis 
metis                  65536  0

When I run it with the pipe=torch-aipu (with -v as well), I get the following results:

The first time I ran it, I got this. It seems to me it managed some detections but then got stuck somewhere? Or that it was querying from None. It did get further than without the torch-aipu, as a little window with the Axelera logo opened up which hadn’t happened before.

(venv) root@iak:/voyager-sdk# ./inference.py yolov8s-coco-onnx ./media/traffic1_480p.mp4 --pipe=torch-aipu -v
DEBUG   :axelera.app.inf_tracers: HostTracer will write to /tmp/tmp36tq5ajq
INFO    :axelera.app.device_manager: Using device metis-1:1:0
DEBUG   :axelera.app.network: Create network from /voyager-sdk/ax_models/zoo/yolo/object_detection/yolov8s-coco-onnx.yaml
DEBUG   :axelera.app.network: Register custom operator 'decodeyolo' with class DecodeYolo from /voyager-sdk/ax_models/decoders/yolo.py
DEBUG   :axelera.app.device_manager: Reconfiguring devices with ddr_size=1.0GB, device_firmware=1, mvm_utilisation_core_0=100%, clock_profile_core_0=800MHz
DEBUG   :axelera.app.inf_tracers: Host FPS multiplier: 1
WARNING :axelera.app.utils: Failed to get OpenCL platforms : clGetPlatformIDs failed: PLATFORM_NOT_FOUND_KHR
WARNING :axelera.app.utils: Please check the documentation for installation instructions
WARNING :axelera.app.utils: pyglet could not access the display, OpenGL is not available: Could not create GL context
INFO    :axelera.app.pipe.manager: Network type: NetworkType.SINGLE_MODEL
INFO    :axelera.app.pipe.manager: Input
INFO    :axelera.app.pipe.manager:   └─detections
DEBUG   :yolo: Model Type: YoloFamily.YOLOv8 (YOLOv8 pattern:
DEBUG   :yolo: - 6 output tensors (anchor-free)
DEBUG   :yolo: - 3 regression branches (64 channels)
DEBUG   :yolo: - 3 classification branches (80 channels)
DEBUG   :yolo: - Channel pattern: [64, 64, 64, 80, 80, 80]
DEBUG   :yolo: - Shapes: [[1, 80, 80, 64], [1, 40, 40, 64], [1, 20, 20, 64], [1, 80, 80, 80], [1, 40, 40, 80], [1, 20, 20, 80]])
DEBUG   :axelera.app.pipe.io: FPS of /root/.cache/axelera/media/traffic1_480p.mp4: 60
INFO    :axelera.app.torch_utils: Using CPU based torch                                                                                     
DEBUG   :axelera.app.pipe.io: Create image generator from VideoCapture                                                                      
DEBUG   :axelera.app.operators.inference: Loaded model : /voyager-sdk/build/yolov8s-coco-onnx/yolov8s-coco-onnx/1/model.json                
DEBUG   :axelera.app.operators.inference: Available ONNX runtime providers: ['AzureExecutionProvider', 'CPUExecutionProvider']              
WARNING :axelera.app.stream: Timeout for querying an inference                                                                              
DEBUG   :axelera.app.operators.inference: Expected core model output shape: [[1, 64, 80, 80], [1, 64, 40, 40], [1, 64, 20, 20], [1, 80, 80, 80], [1, 80, 40, 40], [1, 80, 20, 20]]
ERROR   :axelera.app.display: Exception in inference thread: InferenceThread                                                                
ERROR   :axelera.app.display: Traceback (most recent call last):
ERROR   :axelera.app.display:   File "/voyager-sdk/axelera/app/display.py", line 270, in _target
ERROR   :axelera.app.display:     target(*args, **kwargs)
ERROR   :axelera.app.display:   File "/voyager-sdk/./inference.py", line 48, in inference_loop
ERROR   :axelera.app.display:     for frame_result in tqdm(
ERROR   :axelera.app.display:   File "/root/.cache/axelera/venvs/fd8d4664/lib/python3.10/site-packages/tqdm/std.py", line 1181, in __iter__
ERROR   :axelera.app.display:     for obj in iterable:
ERROR   :axelera.app.display:   File "/voyager-sdk/axelera/app/stream.py", line 175, in __iter__
ERROR   :axelera.app.display:     raise RuntimeError('timeout for querying an inference') from None
ERROR   :axelera.app.display: RuntimeError: timeout for querying an inference
DEBUG   :axelera.app.meta.object_detection: Total number of detections: 6

The second time I ran it I got the following:

(venv) root@iak:/voyager-sdk# ./inference.py yolov8s-coco-onnx ./media/traffic1_480p.mp4 --pipe=torch-aipu -v   
DEBUG   :axelera.app.inf_tracers: HostTracer will write to /tmp/tmpp_o2jqie
INFO    :axelera.app.device_manager: Using device metis-1:1:0
DEBUG   :axelera.app.network: Create network from /voyager-sdk/ax_models/zoo/yolo/object_detection/yolov8s-coco-onnx.yaml
DEBUG   :axelera.app.network: Register custom operator 'decodeyolo' with class DecodeYolo from /voyager-sdk/ax_models/decoders/yolo.py
DEBUG   :axelera.app.device_manager: Reconfiguring devices with ddr_size=1.0GB, device_firmware=1, mvm_utilisation_core_0=100%, clock_profile_core_0=800MHz
DEBUG   :axelera.app.inf_tracers: Host FPS multiplier: 1
WARNING :axelera.app.utils: Failed to get OpenCL platforms : clGetPlatformIDs failed: PLATFORM_NOT_FOUND_KHR
WARNING :axelera.app.utils: Please check the documentation for installation instructions
WARNING :axelera.app.utils: pyglet could not access the display, OpenGL is not available: Could not create GL context
INFO    :axelera.app.pipe.manager: Network type: NetworkType.SINGLE_MODEL
INFO    :axelera.app.pipe.manager: Input
INFO    :axelera.app.pipe.manager:   └─detections
DEBUG   :yolo: Model Type: YoloFamily.YOLOv8 (YOLOv8 pattern:
DEBUG   :yolo: - 6 output tensors (anchor-free)
DEBUG   :yolo: - 3 regression branches (64 channels)
DEBUG   :yolo: - 3 classification branches (80 channels)
DEBUG   :yolo: - Channel pattern: [64, 64, 64, 80, 80, 80]
DEBUG   :yolo: - Shapes: [[1, 80, 80, 64], [1, 40, 40, 64], [1, 20, 20, 64], [1, 80, 80, 80], [1, 40, 40, 80], [1, 20, 20, 80]])
DEBUG   :axelera.app.pipe.io: FPS of /root/.cache/axelera/media/traffic1_480p.mp4: 60
INFO    :axelera.app.torch_utils: Using CPU based torch                                                                                     
DEBUG   :axelera.app.pipe.io: Create image generator from VideoCapture                                                                      
DEBUG   :axelera.app.operators.inference: Loaded model : /voyager-sdk/build/yolov8s-coco-onnx/yolov8s-coco-onnx/1/model.json                
DEBUG   :axelera.app.operators.inference: Available ONNX runtime providers: ['AzureExecutionProvider', 'CPUExecutionProvider']              
Detecting...                               :   0%|                                                            | 0/13363 [00:02<?, ?frames/s][ERROR][axeLoadAndExecuteQueueElf]: Uio wait kernel failed with return code -1406.
[ERROR][axeRunQueueCSourceCommandLists]: Load and execute queue elf failed: 0x70010001.
[ERROR][axeCommandQueueExecuteCommandListsSync]: Run command lists failed: 0x70010001.
ERROR   :axelera.runtime: Error at zeCommandQueueExecuteCommandLists(cmdqueue, n_cmdlists, cmdlists, nullptr): cmdqueue_run_cmdlists: 309: Exit with error code: 0x70010001 : ZE_RESULT_ERROR_NOT_AVAILABLE
ERROR   :axelera.app.pipe.torch: TorchPipe terminated due to RuntimeError at /root/.cache/axelera/venvs/fd8d4664/lib/python3.10/site-packages/axelera/runtime/objects.py:74: AXR_ERROR_RUNTIME_ERROR: Error at zeCommandQueueExecuteCommandLists(cmdqueue, n_cmdlists, cmdlists, nullptr): cmdqueue_run_cmdlists: 309: Exit with error code: 0x70010001 : ZE_RESULT_ERROR_NOT_AVAILABLE
ERROR   :axelera.app.pipe.torch: Full traceback:                                                                                            
ERROR   :axelera.app.pipe.torch:   File "/voyager-sdk/axelera/app/pipe/torch.py", line 55, in _loop
ERROR   :axelera.app.pipe.torch:     image, result, meta = model_pipe.inference.exec_torch(
ERROR   :axelera.app.pipe.torch:   File "/voyager-sdk/axelera/app/operators/inference.py", line 750, in exec_torch
ERROR   :axelera.app.pipe.torch:     self._axr_modeli.run(inputs, outputs)
ERROR   :axelera.app.pipe.torch:   File "/root/.cache/axelera/venvs/fd8d4664/lib/python3.10/site-packages/axelera/runtime/objects.py", line 510, in run
ERROR   :axelera.app.pipe.torch:     _raise_error(self.context, res.value)
ERROR   :axelera.app.pipe.torch:   File "/root/.cache/axelera/venvs/fd8d4664/lib/python3.10/site-packages/axelera/runtime/objects.py", line 74, in _raise_error
ERROR   :axelera.app.pipe.torch:     raise exc(f"{err}: {msg}")
INFO    : Core Temp : 39.0°C                                                                                                                
INFO    : prep : 37.9us
INFO    : Memcpy-in : 562.5us
INFO    : Kernel : 34663.2us
INFO    : Memcpy-out : 459.3us
INFO    : Host : 28.0fps
INFO    : CPU % : 15.5%
INFO    : End-to-end : 0.0fps

When running it with --pipe=torch it does work, but seems to be running on CPU with ~1 FPS.

Thanks for running those check ​@GeertL! This is good stuff.

So it looks like all system-level and SDK checks pass. The board is visible, the firmware and driver are correctly installed, and the Pi environment looks to be configured appropriately. The inference error may then be related to runtime resource issues, subtle PCIe instability, or pipeline issues?

Let’s ask ​@Victor Labian if he has any thoughts on the most practical next step!

Hello ​@GeertL 

Couple of points:

• Have you enable PCIe gen 3 in RPi5 as in https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#pcie-gen-3-0 ?

Also, if you do inference.py --help you can see what each command line options means. In this case:

  --pipe {gst,torch,torch-aipu,quantized}

                        specify pipeline type:

                          - gst: C++ pipeline based on GStreamer (uses AIPU)

                          - torch-aipu: PyTorch pre/post-processing with model on AIPU

                          - torch: PyTorch pipeline in FP32 for accuracy baseline (uses ONNXRuntime for ONNX models)

The fact that you see low FPS is expected as:

• With torch-aipu you are running non-optimized PyTorch pre/post-processing, that is really demanding on the host, and the RPi5 doesn’t have a very powerful CPU.
• With torch you are running the whole pipeline in the CPU which is extremely slow.

You should use gst, which is the default and using optimised pre/post processing and Metis for inference.

Note that even Metis is very fast, the Raspberry Pi 5 will be the bottleneck as it does the pre/post processing.

FYI ​@Spanner 

Best,

Victor

Hi everyone!

Some update from my side:

1. We’ve decided to try deploying the model on another machine - a PC with Ubuntu 20.04 on it. We then installed the Docker container there and ​@GeertL was able to run the inference.py script on it. Big Win.
2. As for Raspberry Pi 5 host, we still face same issues as before. Answering the question above, we have enabled the PCIE gen. 3 in the raspi-config. Any ideas on where to move from here?
3. I have noticed a weird bug while working on Ubuntu 20.04 machine. After rebooting the machine, the accelerator is not recognized, and lspci doesn’t list it. I’ve tried changing the GRUB config as was mentioned in this post 

   However, it didn’t help. However if the system is completely powered off, and turned on afterwards, the chip is recognized and after running sudo update-pciids, it has the proper naming. Is there solution for this issue?

Best,

Ivan

OK one more update:

the inference on out PC host doesn’t work now too. the error is that it fails to deploy it. Here’s the verbosed command output: (Sorry for the image posting move, can’t access this page from my host PC)

Any ideas for the solution?

Best,

Ivan

I wonder if the dataset hasn’t been downloaded? Perhaps if you could take a look at this, and let me know if that helped? Then we can pick up from there if not 👍

https://github.com/axelera-ai-hub/voyager-sdk/blob/release/v1.3/download_prebuilt.py