My Axelera Embedded 111c (Metis Compute Board) cannot communicate with the Metis AIPU. I need help solving the issue .
Board details
- BSP: 1.3.3, flashed over USB with upgrade_tool (kernel 6.1.148-rockchip-standard)
- Driver: patched to 1.6.2 by installing kernel-module-metis-6.1.148-rockchip-standard_v1.8.0-r0_arm64.deb and axelera-container_1.0-r0_arm64.deb (as described in the SDK 1.8.0 release notes). /sys/class/metis/version shows 1.6.2.
- Voyager SDK: 1.8.0 (axversion reports 1.8.0, axversion --driver reports 1.6.2)
- Did axdevice ever work on this board before? no
Symptoms
- axdevice reports: "Device communication timed out", "Failed to get valid board type for device metis-0:1:0 got 12", and board_type=unknown (not responding).
- axdevice --refresh -v reports: "USR_DMA_XFER failed: Connection timed out", "Failed to write ELF to device memory", and "Failed to load runtime stage0".
- lspci shows the device: 01:00.0 Processing accelerators [1200]: Axelera AI Metis AIPU (rev 02) [1f9d:1100]. The PCIe link on fe150000.pcie comes up at Gen3 x4.
- dmesg shows "IRQ MSI timeout (12 1)" and "DMA WR CH0 status -110" whenever axdevice runs.
- /proc/interrupts: all 32 msi-metis-0-1-0-* counters stay at 0.
- Even after a power cycle with the SDK container not started, dmesg shows the PCIe link going down and up about every 10 seconds, starting about 10 s after boot. About 5 s after each bounce, it logs "fwtrace: detached for firmware reload".
What I tried
- Full power cycles
- axdevice --refresh and axdevice --reboot
- [modprobe metis single_msi=1 dma_poll=1, if you tried it]
[ 81.613983] axl 0000:01:00.0: Link up
[ 86.748455] axl 0000:01:00.0: vmsi not available
[ 86.748556] axl 0000:01:00.0: fwtrace: detached for firmware reload
[ 91.674615] axl 0000:01:00.0: Link down
[ 91.762589] axl 0000:01:00.0: Link up
[ 96.985923] axl 0000:01:00.0: vmsi not available
[ 96.985985] axl 0000:01:00.0: fwtrace: detached for firmware reload
[ 101.832886] axl 0000:01:00.0: Link down
[ 101.920880] axl 0000:01:00.0: Link up
[ 106.968997] axl 0000:01:00.0: vmsi not available
[ 106.969137] axl 0000:01:00.0: fwtrace: detached for firmware reload
[ 385.372998] axl 0000:01:00.0: Unregister directory 0000:01:00.0
[ 385.373050] axl 0000:01:00.0: Cleaning up firmware trace consumer
[ 385.373740] axl 0000:01:00.0: Unregistered metis-0-1-0 (0 0)
[ 385.373762] axl 0000:01:00.0: Release dma mem metis-0-1-0
[ 385.411782] axl_aipu: debugfs root directory axl_aipu removed
[ 389.285369] axl_aipu: root directory for axl_aipu
[ 389.285791] axl 0000:01:00.0: All AER errors masked
[ 389.285807] axl 0000:01:00.0: Memory windows prefetch 0x0000000000000000 0x0000000000000000
[ 389.285819] axl 0000:01:00.0: Memory windows no-prefetch 0x0000000900000000 0x0000000004010000
[ 389.288545] axl 0000:01:00.0: vmsi not available
[ 389.288651] axl 0000:01:00.0: DMA trace buffer: 4096 entries (288 KB)
[ 389.288997] axl 0000:01:00.0: Data Link Layer Link Active Reporting capability
[ 389.289442] axl 0000:01:00.0: MSI registered 1 (1)
[ 389.289481] axl 0000:01:00.0: irq vec number 132
[ 389.289492] axl 0000:01:00.0: Initializing Metis MSI (nmsi=1, max_msi=32)
[ 389.290261] axl 0000:01:00.0: fwtrace: datastream area not available yet
[ 389.290274] axl 0000:01:00.0: Firmware trace consumer initialized (kfifo=1024 KB, dma=256 KB, poll=1000 ms)
[ 389.290314] axl 0000:01:00.0: Register directory metis-0000:01:00.0
[ 573.819106] axl 0000:01:00.0: IRQ MSI timeout (12 1)
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
13: 33011 11733 8671 5408 5957 6324 20651 5587 GICv3 26 Level arch_timer
14: 34365 11000 8587 6761 17056 14987 15799 6623 GICv3 321 Level rk_timer
20: 0 0 0 0 0 0 0 0 GICv3 455 Edge debug-signal
21: 0 0 0 0 0 0 0 0 GICv3 365 Level debug
22: 0 0 0 0 0 0 0 0 GICv3 429 Level rockchip_thermal
23: 0 0 0 0 0 0 0 0 GICv3 23 Level arm-pmu
24: 0 0 0 0 0 0 0 0 GICv3 425 Level rockchip_usb2phy
25: 2 0 0 0 0 0 0 0 GICv3 423 Level rockchip_usb2phy
26: 2 0 0 0 0 0 0 0 GICv3 424 Level rockchip_usb2phy
27: 0 0 0 0 0 0 0 0 GICv3 426 Level rockchip_usb2phy
28: 10 0 0 0 0 0 0 0 GICv3 295 Level pcie-sys
30: 0 0 0 0 0 0 0 0 GICv3 118 Level fea10000.dma-controller
31: 0 0 0 0 0 0 0 0 GICv3 119 Level fea10000.dma-controller
32: 0 0 0 0 0 0 0 0 GICv3 120 Level fea30000.dma-controller
33: 0 0 0 0 0 0 0 0 GICv3 121 Level fea30000.dma-controller
34: 0 0 0 0 0 0 0 0 GICv3 122 Level fed10000.dma-controller
35: 0 0 0 0 0 0 0 0 GICv3 123 Level fed10000.dma-controller
37: 0 0 0 0 0 0 0 0 GICv3 142 Level fdab9000.iommu, fdab0000.npu
38: 0 0 0 0 0 0 0 0 GICv3 143 Level fdab9000.iommu, fdab0000.npu
39: 0 0 0 0 0 0 0 0 GICv3 144 Level fdab9000.iommu, fdab0000.npu
40: 0 0 0 0 0 0 0 0 GICv3 150 Level fdb50800.iommu
41: 0 0 0 0 0 0 0 0 GICv3 146 Level fdb60f00.iommu, rga3
42: 0 0 0 0 0 0 0 0 GICv3 147 Level fdb70f00.iommu, rga3
43: 0 0 0 0 0 0 0 0 GICv3 162 Level fdb90480.iommu
44: 0 0 0 0 0 0 0 0 GICv3 153 Level fdba0800.iommu
45: 0 0 0 0 0 0 0 0 GICv3 155 Level fdba4800.iommu
46: 0 0 0 0 0 0 0 0 GICv3 157 Level fdba8800.iommu
47: 0 0 0 0 0 0 0 0 GICv3 159 Level fdbac800.iommu
48: 0 0 0 0 0 0 0 0 GICv3 149 Level fdbb0800.iommu, fdbb0000.iep
49: 0 0 0 0 0 0 0 0 GICv3 131 Level fdbdf000.iommu
50: 0 0 0 0 0 0 0 0 GICv3 132 Level fdbdf000.iommu
51: 0 0 0 0 0 0 0 0 GICv3 134 Level fdbef000.iommu
52: 0 0 0 0 0 0 0 0 GICv3 135 Level fdbef000.iommu
53: 0 0 0 0 0 0 0 0 GICv3 128 Level fdc38700.iommu
54: 0 0 0 0 0 0 0 0 GICv3 130 Level fdc48700.iommu
55: 0 0 0 0 0 0 0 0 GICv3 141 Level fdca0000.iommu
56: 12309 0 0 0 0 0 0 0 GICv3 188 Level fdd97e00.iommu, fdd90000.vop
57: 0 0 0 0 0 0 0 0 GICv3 193 Level fde50000.dp
58: 0 0 0 0 0 0 0 0 GICv3 305 Level ahci-dwc[fe210000.sata]
59: 0 0 0 0 0 0 0 0 GICv3 306 Level ahci-dwc[fe220000.sata]
60: 4799 0 0 0 0 0 0 0 GICv3 360 Level feb20000.spi
61: 0 0 0 0 0 0 0 0 rockchip_gpio_irq 7 Level rk806
62: 0 0 0 0 0 0 0 0 rk806 0 Edge rk805_pwrkey_fall
63: 0 0 0 0 0 0 0 0 rk806 1 Edge rk805_pwrkey_rise
64: 0 0 0 0 0 0 0 0 rk806 7 Level rk806_vb_low
65: 0 0 0 0 0 0 0 0 GICv3 266 Level eth0
66: 0 0 0 0 0 0 0 0 GICv3 265 Level eth0
67: 296 0 0 0 0 0 0 0 GICv3 259 Level eth1
68: 0 0 0 0 0 0 0 0 GICv3 258 Level eth1
69: 0 0 0 0 0 0 0 0 GICv3 252 Level dwc3
70: 2706 0 0 0 0 0 0 0 GICv3 253 Level xhci-hcd:usb1
71: 29 0 0 0 0 0 0 0 GICv3 247 Level ehci_hcd:usb3
72: 32 0 0 0 0 0 0 0 GICv3 250 Level ehci_hcd:usb5
73: 0 0 0 0 0 0 0 0 GICv3 248 Level ohci_hcd:usb4
74: 0 0 0 0 0 0 0 0 GICv3 251 Level ohci_hcd:usb6
75: 13114 0 0 0 0 0 0 0 GICv3 349 Level fd880000.i2c
76: 62 0 0 0 0 0 0 0 GICv3 350 Level fea90000.i2c
77: 0 0 0 0 0 0 0 0 GICv3 354 Level fead0000.i2c
78: 73 0 0 0 0 0 0 0 GICv3 355 Level fec80000.i2c
79: 0 0 0 0 0 0 0 0 rockchip_gpio_irq 27 Level fsc_interrupt_int_n
80: 12 0 0 0 0 0 0 0 GICv3 356 Level fec90000.i2c
81: 0 0 0 0 0 0 0 0 GICv3 175 Level rockchip-mipi-csi2-hw
82: 0 0 0 0 0 0 0 0 GICv3 176 Level rockchip-mipi-csi2-hw
83: 0 0 0 0 0 0 0 0 GICv3 177 Level rockchip-mipi-csi2-hw
84: 0 0 0 0 0 0 0 0 GICv3 178 Level rockchip-mipi-csi2-hw
85: 0 0 0 0 0 0 0 0 GICv3 179 Level rockchip-mipi-csi2-hw
86: 0 0 0 0 0 0 0 0 GICv3 180 Level rockchip-mipi-csi2-hw
87: 0 0 0 0 0 0 0 0 GICv3 181 Level rockchip-mipi-csi2-hw
88: 0 0 0 0 0 0 0 0 GICv3 182 Level rockchip-mipi-csi2-hw
89: 0 0 0 0 0 0 0 0 GICv3 183 Level rockchip-mipi-csi2-hw
90: 0 0 0 0 0 0 0 0 GICv3 184 Level rockchip-mipi-csi2-hw
91: 0 0 0 0 0 0 0 0 GICv3 185 Level rockchip-mipi-csi2-hw
92: 0 0 0 0 0 0 0 0 GICv3 186 Level rockchip-mipi-csi2-hw
93: 8004 0 0 0 0 0 0 0 GICv3 237 Level mmc0
94: 8358 0 0 0 0 0 0 0 GICv3 430 Level fec10000.saradc
95: 0 0 0 0 0 0 0 0 GICv3 217 Level i2s
96: 0 0 0 0 0 0 0 0 GICv3 212 Level i2s
97: 0 0 0 0 0 0 0 0 GICv3 148 Level rga2
98: 0 0 0 0 0 0 0 0 GICv3 151 Level fdb51000.avsd-plus, fdb50400.vdpu
101: 0 0 0 0 0 0 0 0 GICv3 152 Level fdb50000.vepu
102: 0 0 0 0 0 0 0 0 GICv3 154 Level fdba0000.jpege-core
103: 0 0 0 0 0 0 0 0 GICv3 156 Level fdba4000.jpege-core
104: 0 0 0 0 0 0 0 0 GICv3 158 Level fdba8000.jpege-core
105: 0 0 0 0 0 0 0 0 GICv3 160 Level fdbac000.jpege-core
106: 0 0 0 0 0 0 0 0 GICv3 161 Level fdb90000.jpegd
107: 0 0 0 0 0 0 0 0 GICv3 127 Level fdc38100.rkvdec-core
108: 0 0 0 0 0 0 0 0 GICv3 129 Level fdc48100.rkvdec-core
109: 0 0 0 0 0 0 0 0 GICv3 133 Level fdbd0000.rkvenc-core
110: 0 0 0 0 0 0 0 0 GICv3 136 Level fdbe0000.rkvenc-core
111: 0 0 0 0 0 0 0 0 GICv3 140 Level fdc70000.av1d
113: 3 0 0 0 0 0 0 0 GICv3 392 Level dw-hdmi-qp-hpd
114: 0 0 0 0 0 0 0 0 GICv3 201 Level fde80000.hdmi
115: 0 0 0 0 0 0 0 0 GICv3 203 Level fde80000.hdmi
116: 34 0 0 0 0 0 0 0 GICv3 204 Level fde80000.hdmi
117: 2633 0 0 0 0 0 0 0 GICv3 124 Level fb000000.gpu
118: 1 0 0 0 0 0 0 0 GICv3 125 Level fb000000.gpu
119: 2343 0 0 0 0 0 0 0 GICv3 126 Level fb000000.gpu
120: 0 0 0 0 0 0 0 0 GICv3 235 Level dw-mci
121: 11 0 0 0 0 0 0 0 GICv3 105 Level dmc
131: 0 0 0 0 0 0 0 0 ITS-MSI 8 Edge PCIe PME
132: 0 0 0 0 0 0 0 0 ITS-MSI 524288 Edge msi-metis-0-1-0-0
IPI0: 523 399 1033 340 2192 281 1801 240 Rescheduling interrupts
IPI1: 16938 19788 15981 16084 17243 10886 21807 12593 Function call interrupts
IPI2: 0 0 0 0 0 0 0 0 CPU stop interrupts
IPI3: 0 0 0 0 0 0 0 0 CPU stop (for crash dump) interrupts
IPI4: 4848 1197 3616 1900 5515 1098 1725 2397 Timer broadcast interrupts
IPI5: 423 292 230 212 343 286 297 297 IRQ work interrupts
IPI6: 0 0 0 0 0 0 0 0 CPU wake-up interrupts
Err: 0
INFO: Found AIPU driver: metis 147456 0
[libaxldev_linux.c:1678] Device communication timed out: device did not respond within 1 seconds. (0)
WARNING: Failed to get valid board type for device metis-0:1:0 got 12
INFO: Ignoring device 0(metis-0:1:0) as it has not returned a valid board type: 12
INFO: Using device metis-0:1:0
Device 0: metis-0:1:0 board_type=unknown (not responding)
