DMABUF error

Hi ​@mato!

Hmm, the kernel line:

axl 0001:01:00.0: DMA error WR CH0

might suggest a low-level DMA/PCIe/device-state issue rather than the model itself, I wonder...

As a first thing to try, a full power-off shutdown of the Raspberry CM5 + Metis M.2 is always worthwhile. Remove power entirely, give it a minute, then boot up again. If you’re running inside Docker, restarting the container before running inference again is often helpful.

This is different from using axdevice --reboot or a normal warm reboot, and it’s often kicked things back into action for people.

Let me know if that does anything, and we can go from there!

Hi ​@Spanner,

yes, power restart does get the device back in a good state, but since we use the Metis device in a real time control system having to restart the device to recover is really bad for us. We would prefer to somehow prevent this error from happening.

Just curious if you have any insight or previous experience with this error?

The error occurs even if we configure pcie gen2 or gen3 in the config.txt in linux.
 

Yeah, it’s useful for diagnostic purposes, but you can’t rely on being able to cold restart is as part of its daily operation 😅

Certainly I think setting it to PCIe gen3 is the way to go, but it sounds like you’ve tried that. It’s not a simple power issue, is it? I know the RPi can get temperemental in some weird ways if it’s not getting just enough juice. I suspect you’ve checked that, but just in case.

 Let’s see if ​@Victor Labian has any insights, too - he’s done a lot of awesome work with a Metis and RPi5 combo. 👍

Thanks for including ​@Victor Labian I hope he gives us some good insight.

Originally I used PCIe gen3 and the default power save mode in  Active State Power Management:
/sys/module/pcie_aspm/parameters/policy. With this combo I was getting a lot of recoverable PCIe errors (11k / min). Switching the ASPM to “performance” mode it cleared the recoverable errors to zero. But the fatal one that crashes the runtime persisted, Switching to PCIe gen2 produced same results. I can double check the power supply to the Metis.

Hi ​@mato  ​@Spanner ,

If it runs and interrupts after some minutes I suspect it can be related to power peaks.

Let us know if the following works:

Power delivery on the Raspberry Pi 5 can be limited, which may affect some models. If inference crashes, limit MVM utilisation on Metis. From our internal testing we recommend a 20% limit on the Raspberry Pi 5 (tested on Voyager SDK 1.3):

AXELERA_CONFIGURE_BOARD=,20 ./inference.py yolov8l-coco-onnx ./media/traffic1_480p.mp4 --no-display

Thank you for the advice, I will try it out and get back to you as soon as possible.

​@Spanner  ​@Victor Labian 
So I applied the 20% limit on all 4 cores and so far have not recieved the  PCIe error. I even ran one long 1 hour run. One important note is also I did not report any performance degradation.

I will track this issue in the upcoming days to see if the error reoccurs with this configuration. But for now it seems fine.

Thank you both for the support!

Awesome news! So glad it's up and running! 

What's the project you're working on, by the way?

​@mato Thank you for letting us know that it is working now! Glad I could help :)