Firmware loading error on Windows

Hi ​@nicov ! Thanks for the detailed logs, it always helps to see those.

I think our key clue is that every value the SDK reads back is 0xff / 0xffffffffffffffff (firmware version, board DDR size, the bad magic 0xffff line), which suggests the PCIe link to the card not coming up properly. Windows enumerates the device and binds the driver, but it doesn’t look like the host can actually read the card's memory over PCIe. And if that’s the case, it’s not so much that the firmware is missing as the SDK can't talk to the card to read or load it.

That’d tally with the PC refusing to reboot with the card installed and running slowly when it does, then being fine once you’ve removed it. So I’m feeling like it’s the M.2 slot / PCIe configuration on that specific mini PC...

Here’s something quick to try first:

• Go into the BIOS and check the M.2 slot's mode. A quick search suggests a lot of mini-PC M.2 slots are set to SATA/NVMe-storage-only or limited PCIe lanes. If there's a setting for PCIe mode / lane configuration / "M.2 mode" or something for that slot, set it to PCIe (and the widest lane width available), then reboot and re-run axdevice -v.

Let’s see f that helps us narrow it down and then go from there! 👍

Thanks ​@Spanner ! I found pci express configurations in the bios. There are two mutually exclusive settings ‘pcie x4 dt slot power enable’ and ‘m.2 ssd1 power enable’, but strangely enough both are enabled (there has never been a m.2 card in the second slot so this may have not been a problem in the past. There is a separate ‘m.2 ssd0 power enable’ that is enabled, but that is probably for the boot m.2 nvme drive. There are no settings related to pcie lanes on the slot, nor sata/nvme settings. I will try in the comings days if changing the mutually exclusive settings make any difference. If that doesnt make a difference i’ll try how the situation looks like in Linux.

So I got some progress: disabling the ‘m.2 ssd1 power enable’ and keeping  ‘pcie x4 dt slot power enable’  enabled resulted in successful axdevice output: (I also ran the cmd prompt as admin, don’t know if this made a difference) 

axdevice -v

INFO: Found PCI device: Instance ID: PCI\VEN_1F9D&DEV_1100&SUBSYS_11001F9D&REV_00\4&c1676ec&0&0011 Device Description: Metis Device Class Name: ComputeAccelerator Class GUID: {f01a9d53-3ff6-48d2-9f97-c8a7004be10c} Manufacturer Name: Axelera AI B.V. Status: Started Driver Name: oem27.inf INFO: Found AIPU driver: oem27.inf INFO: Current firmware version v1.3.2+bl1-stage0 != required version v1.7.0 INFO: Device firmware version is not compatible, loading now INFO: Using device metis-0:0:0 Device 0: metis-0:0:0 1GiB metis-m2 flver=1.3.2 bcver=1.4 clock=800MHz mvm=100%   name=metis-0:0:0   device_runtime_firmware=v1.7.0   board_controller_board_type=ortles   sw_throttling: 200°C, hysteresis 5°C, throttle rate:12%   hw_throttling: 105°C, hysteresis 10°C   freq_downscaling: 110°C, hysteresis 5°C   pvt_warning_threshold: 95°C

But when I try to run the sample it gives hardware related errors again: axrunmodel C:\users\nicov\Code\voyager-sdk\build\resnet50-imagenet-onnx\resnet50-imagenet-onnx\1\model.json C:\users\nicov\Code\voyager-sdk\build\resnet50-imagenet-onnx\resnet50-imagenet-onnx\1\model.json, |▎ [libaxldev.c:142] Failed to wait for MSI 0: No error [libaxldev.c:142] Failed to wait for MSI 1: No error [ERROR][waitForQueueBinaryCompletion]: Wait kernel failed with return code -1. [libaxldev.c:142] Failed to wait for MSI 3: No error [ERROR][waitForQueueBinaryCompletion]: Wait kernel failed with return code -1. [libaxldev.c:142] Failed to wait for MSI 2: No error [ERROR][waitForQueueBinaryCompletion]: Wait kernel failed with return code -1. [ERROR][waitForQueueBinaryCompletion]: Wait kernel failed with return code -1. [ERROR][axeCommandQueueExecuteCommandLists]: Failed with error code: 1879048193 [on 0.7%: ERROR:axelera.runtime:Error at zeCommandQueueExecuteCommandLists(cmdqueue, n_cmdlists, cmdlists, nullptr): cmdqueue_run_cmdlists: 320: Exit with error code: 0x70000001 : ZE_RESULT_ERROR_DEVICE_LOST omC:\users\nicov\Code\voyager-sdk\build\resnet50-imagenet-onnx\resnet50-imagenet-onnx\1\model.json, |▎ mandQueueExecuteCommandLists]: Failed with error code: 1879048193 [ERERROR:axelera.runtime:Error at zeCommandQueueExecuteCommandLists(cmdqueue, n_cmdlists, cmdlists, nullptr): cmdqueue_run_cmdlists: 320: Exit with error code: 0x70000001 : ZE_RESULT_ERROR_DEVICE_LOST ROR][axeCommandQueueExecuteCommandLists]: Failed with error code: 1879048193 [ERROR:axelera.runtime:Error at zeCommandQueueExecuteCommandLists(cmdqueue, n_cmdlists, cmdlists, nullptr): cmdqueue_run_cmdlists: 320: Exit with error code: 0x70000001 : ZE_RESULT_ERROR_DEVICE_LOST ERROR][axeCommandQueueExecuteCommandLists]: Failed with error code: 1879048193 ERROR:axelera.runtime:Error at zeCommandQueueExecuteCommandLists(cmdqueue, n_cmdlists, cmdlists, nullptr): cmdqueue_run_cmdlists: 320: Exit with error code: 0x70000001 : ZE_RESULT_ERROR_DEVICE_LOST [utils.c:264] sysctl_get_datastream_area: bad magic 0xffff (expected 0xf0f0) ERROR:axelera.runtime:Invalid log source: aicore0 C:\users\nicov\Code\voyager-sdk\build\resnet50-imagenet-onnx\resnet50-imagenet-onnx\1\model.json,FAIL: AXR_ERROR_INVALID_ARGUMENT: Invalid log source: aicore0

If I then run axdevice again I get the original errors again: 

axdevice -v INFO: Found PCI device: Instance ID: PCI\VEN_1F9D&DEV_1100&SUBSYS_11001F9D&REV_00\4&c1676ec&0&0011 Device Description: Metis Device Class Name: ComputeAccelerator Class GUID: {f01a9d53-3ff6-48d2-9f97-c8a7004be10c} Manufacturer Name: Axelera AI B.V. Status: Started Driver Name: oem27.inf INFO: Found AIPU driver: oem27.inf INFO: Current firmware version \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff != required version v1.7.0 INFO: Device firmware version is not compatible, loading now [libaxldev.c:419] Firmware load address 0xffffffffffffffff exceeds 32-bit address space on Metis. WARNING: Failed to load runtime stage0: C:\AxeleraDevice-1.7.0\opt\axelera\device\omega\bin\start_axelera_runtime_stage0.bin INFO: Using device metis-0:0:0 [AxeleraDevice.cpp:455] Failed to get board DDR size. WARNING: Failed to get SW throttling from device metis-0:0:0 WARNING: Failed to get HW throttling from device metis-0:0:0 WARNING: Failed to get freq downscaling from device metis-0:0:0 Device 0: metis-0:0:0 0 metis-pcie flver= bcver= clock=0MHz mvm=0%   name=metis-0:0:0 Traceback (most recent call last):   File "<frozen runpy>", line 203, in _run_module_as_main   File "<frozen runpy>", line 88, in *run_code   File "c:\Users\nicov\Code\voyager-sdk\venv-win\Scripts\axdevice.exe_*main**.py", line 5, in <module>     sys.exit(entrypoint_main())              ~~~~~~~~~~~~~~~^^   File "c:\Users\nicov\Code\voyager-sdk\venv-win\Lib\site-packages\axelera\runtime\axdevice.py", line 1135, in entrypoint_main     result = args.func(args, extras)   File "c:\Users\nicov\Code\voyager-sdk\venv-win\Lib\site-packages\axelera\runtime\axdevice.py", line 872, in main     print_device_info(c, found_devices, args.verbose > 0, args.verbose > 0 or therm)     ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "c:\Users\nicov\Code\voyager-sdk\venv-win\Lib\site-packages\axelera\runtime\axdevice.py", line 457, in print_device_info     print(f" device_runtime_firmware={d.firmware_version}")                                        ^^^^^^^^^^^^^^^^^^   File "c:\Users\nicov\Code\voyager-sdk\venv-win\Lib\site-packages\axelera\runtime\objects.py", line 228, in firmware_version     return self._info.firmware_version.decode("utf-8")            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

After that, when I shutdown the computer and restart again, it doesn't boot anymore and I have to remove the Metis card.

Any ideas?

After disabling secure boot in the bios i can run the sample successfully.

The computer still wont start after a shutdown, however if i keep it unplugged for some time I can boot it again.

So the firmware/link/SDK chain is now good. 👍 Excellent work there ​@nicov !

Sounds like a mini PC power/rail or PERST# reset-timing problem on the M.2 slot. I have one that does that. There aren’t any power settings in the BIOS specfic to the M.2 slot that might help with this, so you don’t have to wait for the PSU to bleed down before restarting?