Hello together we are Sebastian Krödel and Lorenzo Gualniera and we are super excited to having been selected for the Pioneer 10 Project Challenge. Sebastian is a machine learning engineer and Lorenzo is an embedded machine learning engineer, both with a background in the biomedical field.
What are we Building?
We will build See & Hear, a table-top companion for two-to-four–person meetings.
The headphone wearer receives a private, noise-reduced feed of the talker they are looking at – and can instantly switch sides by simply raising the matching hand (left hand → left seat, right hand → right seat).
This gives interpreters, note-takers and non-native speakers a fatigue-free, captioned audio channel while the room conversation remains untouched.
Why Does This Project Matter to Us?
For seven years Sebastian developed beam-forming and noise-reduction systems that help people with hearing loss follow conversations. Users, and even perfectly normal-hearing colleagues, sink into fatigue during fast, multilingual meetings because they can’t isolate the voice that matters.
See & Hear is supposed to solve these issues with no custom earbuds, no cloud, just look or lift a hand and listen.
First Steps
Once the board arrives we will start to work on running the vision models we need (YOLO and Retinaface) before we look into the audio chain and integration topics.
Such a great, creative solution to a problem that so many people face, but aren’t even necessarily aware of! I also love that it’s an AI project based around audio that isn’t just TTS or STT.
Years ago I helped set up a board room at the company I was at with a custom-made telepresence system. It wasn’t particularly clever! Big screen TVs around one side of a board table, with cameras pointing back at the people who were there in person. Then a simple panel with three buttons - one for each of the other offices. Press the button, and it opened a video call.
It worked okay, and it was the early days of video calls. It’d probably be much better now. But either way, it would have benefited massively for the inclusion of a system like See and Hear! Best of luck guys!
I am trying to run the hello world example of the voyager SDK.
./inference.py yolov5s-v7-coco media/traffic1_1080p.mp4
This works but when I try to use a USB device, I am not sure which device ID to set…
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 007 Device 002: ID 0461:3f41 Primax Electronics, Ltd HP Wired Desktop 320K Keyboard
Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 004: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC
Bus 002 Device 003: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC
Bus 002 Device 002: ID 1a40:0101 Terminus Technology Inc. Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 006: ID 03f0:1198 HP, Inc HID-compliant mouse
Bus 001 Device 005: ID 0bda:8179 Realtek Semiconductor Corp. RTL8188EUS 802.11n Wireless Network Adapter
Bus 001 Device 004: ID 046d:08e5 Logitech, Inc. HD Pro Webcam C920
Bus 001 Device 003: ID 2886:0018 Seeed Technology Co., Ltd. ReSpeaker 4 Mic Array (UAC1.0)
Bus 001 Device 002: ID 05e3:0608 Genesys Logic, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hu
Which number does it refer to ? For instance I want to select
Bus 001 Device 004: ID 046d:08e5 Logitech, Inc. HD Pro Webcam C920
Additionally I would be happy if somebody knows of an example of running two models in parallel on the same input. In the tutorials I have just seen cascaded (sequential) models.
Hi @skroedel!
For using a USB device with the Voyager SDK’s inference.py, I think you’d reference it as usb:<device_id>. For example, for the Logitech xebcam, the format would be:
./inference.py yolov5s-v7-coco usb:0
(Here, usb:0 refers to the first USB video device detected by the system)
Thanks @Spanner makes sense but I am getting some strange error now:
(venv) aetina@aetina:~/voyager-sdk$ ./inference.py yolov5s-v7-coco usb:0
No target device found in lspci output
Failed to create device manager: AXR_ERROR_CONNECTION_ERROR: No target device found in lspci output
WARNING : Unsupported tracer: core_temp: valid tracers are: cpu_usage, end_to_end_fps, stream_timing
ERROR : No target device found in lspci output
ERROR : Failed to create device manager: AXR_ERROR_CONNECTION_ERROR: No target device found in lspci output
arm_release_ver: g13p0-01eac0, rk_so_ver: 9
WARNING : pyglet could not access the display, OpenGL is not available: No standard config is available.
ERROR : Unable to set the pipeline to the ready state
terminate called after throwing an instance of 'std::runtime_error'
what(): axr_device_connect failed : No device found
Aborted
I have seen it in a different thread but restarting the system did not help.
lspci | grep 1f9d does not show anything
Hmm, was that a full power cycle you did, @skroedel? That typically helps, but let me know if not!
@Spanner : If you are talking about disconnecting the power cord and then re-attaching it, yes I have done that multiple times.
after rebooting with the tiny black button I am getting all sorts of strange errors:
(venv) aetina@aetina:~/voyager-sdk$ ./inference.py yolov5s-v7-coco usb:0
arm_release_ver: g13p0-01eac0, rk_so_ver: 9
WARNING : pyglet could not access the display, OpenGL is not available: No standard config is available.
ERROR : Unable to set the pipeline to the ready state
>ERROR]]zeModuleCreate]: An unexpected exception occurred: std::bad_alloc
>ERROR]]zeKernelCreate]: Failed with error code: 2013265925
terminate called after throwing an instance of 'std::runtime_error'
what(): axr_load_model_instance failed : Error at zeKernelCreate(module_handle, &desc, &kernel_handle): kernel_create: 409: Exit with error code: 0x78000005 : ZE_RESULT_ERROR_INVALID_NULL_HANDLE
Aborted
OR
(venv) aetina@aetina:~/voyager-sdk$ ./inference.py yolov5s-v7-coco usb:0
arm_release_ver: g13p0-01eac0, rk_so_ver: 9
WARNING : pyglet could not access the display, OpenGL is not available: No standard config is available.
ERROR : Unable to set the pipeline to the ready state
Assembler messages:
Fatal error: can't create /tmp/level-zero_build_1799/9563450766924294387/0/axeCmdListHeader.o: No such file or directory
bERROR]Rcompile]: Compile command failed. command="riscv64-unknown-elf-gcc -march=rv64imafdc -Werror -Wall -Wextra -mabi=lp64d -mcmodel=medany -ffast-math -ffunction-sections -fdata-sections -fno-builtin-printf -fno-common -O2 -I /opt/axelera/device-1.3.1-1/omega/infra/include -c /tmp/level-zero_build_1799/9563450766924294387/0/axeCmdListHeader.c -o /tmp/level-zero_build_1799/9563450766924294387/0/axeCmdListHeader.o -MD -MT /tmp/level-zero_build_1799/9563450766924294387/0/axeCmdListHeader.o -MF /tmp/level-zero_build_1799/9563450766924294387/0/axeCmdListHeader.o.d", ret=256
bERROR]RaxeCommandQueueCreate]: Create command list header failed.
terminate called after throwing an instance of 'std::runtime_error'
what(): axr_load_model_instance failed : Error at zeCommandQueueCreate(context, device, &desc, &cmdqueue): cmdqueue_create: 305: Exit with error code: 0x70000004 : ZE_RESULT_ERROR_MODULE_BUILD_FAILURE
Aborted
Hi @skroedel ,
- Inside the venv inside the Docker container, can you try
axdevice --refresh and then share the output of lspci ? - Can you also try to do
sudo reboot and then share the output of lspci ? - Please also share the output of:
cat /sys/class/metis/versionlsmod | grep metis
- Which Voyager SDK version are you using?
Thanks beforehand,
Victor
@skroedel
If the previous didn’t work, please also try:
There should be a fix_pcie.sh (or smth similar) script in home that @Steven Hunsche placed in these boards for you.
If not follow this:
Remove and rescan PCIe bridge manually
PCIe bridge to Metis
Find the port addresses of your PCIe bridges with lspci and lspci -tv
Remove the bridge to which Metis should be connected:
echo 1 | sudo tee /sys/bus/pci/devices/0000:xx:00.0/remove > /dev/null
Then rescan by doing:
echo 1 | sudo tee /sys/bus/pci/rescan > /dev/null
Before Reboot and after Reboot the output is the same:
(venv) aetina@aetina:~/voyager-sdk$ axdevice --refresh
0000:01:00.0 : Axelera AI Metis AIPU (rev 02)
Device 0: metis-0:1:0 1GiB m2 flver=1.2.0-rc2 bcver=1.0 clock=800MHz(0-3:800MHz) mvm=0-3:100%
(venv) aetina@aetina:~/voyager-sdk$ lspci
00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
01:00.0 Processing accelerators: Axelera AI Metis AIPU (rev 02)
The other one:
(venv) aetina@aetina:~/voyager-sdk$ cat /sys/class/metis/version
1.0.2
(venv) aetina@aetina:~/voyager-sdk$ lsmod | grep metis
metis
I am using Voyager SDK 1.3.1
Now I stumbled about Docker Containers you mentioned … So far I dont run this command from within a container.
Hi @skroedel ,
Thanks for sharing. axdevice --refresh simply does a PCIe bridge removal and rescan. That usually helps when for some reason the card was not enumerated at boot time.
If you encounter the same issue, run axdevice --refresh and things should work. Your driver looks also correctly loaded.
Everything looks correct now, so you should be able to run inference. Please let us know.
Best,
Victor
@skroedel
If things work well for you without using Docker container, feel free not to use Docker container. We have tested that things work well on the Aetina without using Docker container.
I see that you are probably not using Docker, so that’s great. You can work the way you prefer.
Also, I use this message to share this guide with you just in case: https://support.axelera.ai/hc/en-us/articles/25556514922514-Solving-Storage-Space-Issues-on-Aetina-Evaluation-System
Best,
Victor
@Victor Labian :
Thanks I finally managed to get it running. The problem was the wrong usb source.
I ran:
v4l2-ctl --list-devices
and saw that my webcam was not actually /dev/video0 but actually /dev/video20 …
No idea why its enumerated like this, there is a bunch of devices listed that I dont know about:
rkisp-statistics (platform: rkisp):
/dev/video18
/dev/video19
rkcif-mipi-lvds2 (platform:rkcif):
/dev/media0
rkisp_mainpath (platform:rkisp0-vir0):
/dev/video11
/dev/video12
/dev/video13
/dev/video14
/dev/video15
/dev/video16
/dev/video17
/dev/media1
Anyways I am happy it runs now, thanks for your help !
Hi @skroedel ,
Thank for letting us know! Yes, it is known that the USB device in the Aetina appears on /dev/video20.
We should add that to our documentation for Aetina, in case we haven’t done that.
@Spanner let’s take this action offline as something internal for us.
Glad the issue is solved @skroedel and best of luck with the challenge! We are looking forward to seeing your project :)
Best,
Victor
@Victor Labian : I am having again issues with the PCI … Now I am getting this output when doing axdevice --refresh:
ls: cannot access '/sys/bus/pci/devices/*/pci_bus/0000:01': No such file or directory
Any ideas what could be the problem ?
Hi @skroedel ,
Have you done or tried any updates and that is the reason it stopped working?
Or it simply suddenly stopped working with no reason?
Please try also doing a reboot and also try doing a full power cycle (unplug and plug) and then see if Metis is listed in lspci -tv.
Let us know how it goes.
Please also check the output of:
cat /sys/class/metis/versionlsmod | grep metis
Best,
Victor
@Victor Labian : I am having again issues with the PCI … Now I am getting this output when doing axdevice --refresh:
ls: cannot access '/sys/bus/pci/devices/*/pci_bus/0000:01': No such file or directory
Any ideas what could be the problem ?
It’s not something a quick axdevice --refresh might solve? I’m being basic here, and Victor’s advice is probably a lot better, but just in case 
thanks guys… I am having some problems from time to time, my go to solution is now full-reboot and then axdevice --refresh… It does not always work but most of the times.
