Dear Zefir,

Thanks for sharing this detailed description with us and I hope (besides the booting issue) you already managed to get some models deployed on Metis PCIe.

On the second issue:
Can you try “axdevice --refresh”? Sometimes this helps to identify the device.

On the first issue:
Of course there is always more information, which is helpful in understanding what’s going on here: When you run the axdevice command after a cold reboot in a freshly activated container, what is the output that you get?
(I expect it should be something like “Firmware version: v1.2.xxx”)
 

Hi ​@jonask-ai 

no, the commands you suggested fail due to same issue:

(venv) root@zefir-PC:~/voyager-sdk# axdevice -v

INFO:axelera.runtime:Found PCI device: 05:00.0 Processing accelerators: Axelera AI Metis AIPU (rev 02)

INFO:axelera.runtime:Found AIPU driver: metis                  90112  0

WARNING:axelera.runtime:4PCI device count mismatch: lspci=1, triton=0
 

This is after cold-boot and device showing up in the lspci list.

I skimmed through the metis driver source code and if I understand correctly, this is what happens:

1. it identifies the PCI bridge it is attached to

o    3.874876] pci 0000:05:00.0: Found target device: TRITON_OMEGA_DEVICE_ID
_    3.874883] pci 0000:05:00.0: Found target device: 0000:05:00.0
0    3.874886] pcieport 0000:03:02.0: Found bridge device: 0000:03:02.0
:    3.874894] pcieport 0000:03:02.0: Bridge window: :mem 0xf8000000-0xfa0fffff]
0    3.874898] pcieport 0000:03:02.0: Decoded memory behind bridge: f8000000-fa0fffff

2. it checks whether the PCI bridge has at least 47MB and since not, tries a rescan

i    3.874903] pcieport 0000:03:02.0: Removing bridge device: 0000:03:02.0
   3.881586] pci_bus 0000:05: busn_res: ]bus 05] is released
s    3.886408] pci 0000:03:02.0: 1022:57a3] type 01 class 0x060400 PCIe Switch Downstream Port
I    3.886451] pci 0000:03:02.0: PCI bridge to bus 05]
2    3.886465] pci 0000:03:02.0:   bridge window mem 0xf8000000-0xfa0fffff]
o    3.886796] pci 0000:03:02.0: PME# supported from D0 D3hot D3cold

3. assumption here is that the bridge would resize the memory to match its attached nodes’, which it does not

n    3.900213] pci 0000:05:00.0: /1f9d:1100] type 00 class 0x120000 PCIe Endpoint
t    3.900282] pci 0000:05:00.0: BAR 0 mem 0xfa010000-0xfa010fff 64bit]
A    3.900287] pci 0000:05:00.0: BAR 2 bmem 0xf8000000-0xf9ffffff]
:    3.900296] pci 0000:05:00.0: ROM fmem 0xfa000000-0xfa00ffff pref]
0    3.900417] pci 0000:05:00.0: supports D1
�    3.900420] pci 0000:05:00.0: PME# supported from D0 D1 D3hot
   3.916294] pci 0000:03:02.0: PCI bridge to bbus 05]
9    3.916367] pci 0000:03:02.0: bridge window ]mem size 0x03000000]: can't assign; no space
n    3.916371] pci 0000:03:02.0: bridge window amem size 0x03000000]: failed to assign
r    3.916376] pci 0000:05:00.0: BAR 2 lmem size 0x02000000]: can't assign; no space
0    3.916379] pci 0000:05:00.0: BAR 2 smem size 0x02000000]: failed to assign
0    3.916382] pci 0000:05:00.0: ROM ]mem size 0x00010000 pref]: can't assign; no space
.    3.916385] pci 0000:05:00.0: ROM amem size 0x00010000 pref]: failed to assign
0    3.916388] pci 0000:05:00.0: BAR 0 emem size 0x00001000 64bit]: can't assign; no space
0    3.916391] pci 0000:05:00.0: BAR 0 mem size 0x00001000 64bit]: failed to assign
0    3.916394] pci 0000:03:02.0: PCI bridge to bus 05]
i    3.921373] pci 0000:03:02.0: Bridge removal and PCIe rescan completed successfully
 

What I don’t quite understand, where does the requirement to have 47MB memory at the bridge (EXPECTED_MEM_BEHIND_BRIDGE_SIZE) come from, since the two BARs of the metis card are  smaller than that (0xfa010000-0xfa010fff, 0xf8000000-0xf9ffffff)?

Anyway, since it seems to work on other platforms, it might be that my mainboard is not compatible. I might retire the PCIe device and pre-order a Metis Compute Board instead.

Would there be something else I could try to help narrowing down the issue?

Yeah, as you say, there could be an inherent compatibility issue with the mainboard. Although one thing that just sprang to mind (which may or may not help as it was for a different issue! ) is someone said a while ago that some AMD boards ship with “Above 4G Decoding” turned off in the BIOS. Turning it on can help the system to assign resources more appropriately when multiple devices are sharing limited space.

Have you seen a setting for that in your BIOS ​@zefir? Might be worth a quick test, if so?

Finally!

This was a tough one and in a desperate monkey-testing session I finally succeeded to have proper PCI access to the Metis card. It is a very system-specific issue which might not affect many users out there, but I’m posting it just in case.

As written above, my system is based on AMD X570 chipset, here in a ASUS PRIME X570-P board. That one has two PCIEx16 slots and distributes 16 lanes to that slots based on how they are populated. What is buried deep in the manual is this: they are not independent but meant to power either one or two GPU cards. Turned out that when the first slot has a GPU installed, the second slot must either remain empty or have a second GPU attached. Other cards are silently disabled, which makes them visible in lspci but their memory access remains disabled.

So the first part of the solution was: put the Metis in the primary slot, put the VGA card in the second

At BIOS side I had to play around until a workable combination was found stable:

• CPU PCIE ASPM Mode: Auto → Disabled
• PCIEx16_1 Bandwidth Bifurcation Configuration: Auto → x8/x8
• PCIE Above 4G Decoding: Disabled → Enabled
• PCIE Resize BAR Support: Disabled → Auto
• PCIE SR-IOV Support: Auto → Disabled

With those BIOS setting for my system it is not required to add boot parameters to the kernel anymore.

As noted in the other thread, the warm-boot failure persists, which is already a known issue.

Thanks for the support so far.

Thanks for sharing the solution ​@zefir !

​@Victor Labian - worth adding to the WIP PCI troubleshooting guide

Hello ​@zefir ,

Thank you for sharing your case and solution.

Regarding the warm boot I have a few things we can check. Can you please help us by testing them one by one and see if one of these makes it work?

(If 1. works, no need to test 2. or 3.)

1. Metis device not detected after startup, but is after reboot

This can be due to the pcie-rescan script not being enabled on boot. Run the following to enable it:

systemctl enable pcie-check.service

Then in any future boots the service will be enabled.

(I don’t expect this to solve your issue but it is worth checking)

2. Enable PCIe Shutdown Mode on BIOS

Check if your BIOS has any of these options and enable it.

Look for an option such as:

• "PCIe Slot Power Control"

• "PCIe Power-On Reset"

• "PCIe Cold Reset"

• "PCIe Shutdown Mode"

PCIe Shutdown Mode (Cold Reset) is a feature on some boards that cuts power entirely to a PCIe slot during a reboot or reset. We have seen hosts in which enabling this feature helped.

3. Cold boot via PCIe vuses triton_multi_ctx]

If you stablished a PCIe connection with the Metis card but you face communication issues, performing a cold boot via PCIE might help.

Go to your voyager-sdk folder and activate the virtual environment with source venv/bin/activate command.

Execute the following command:

triton_multi_ctx --cold-boot 3

Let us know if any of these helped.

Best,

Victor

Wow, well done on figuring that complex combo out ​@zefir! Amazing work! And thanks for sharing - who knows, it may turn out not to be as fringe an issue as we think, as more and more host systems are brought into action.

I just invented a System Sensei badge for exactly this kind of thing, and I think you’ve more than earned it!

Yo, that’s a huge badge - was definitively worth the work :)  Thanks ​@Spanner 

Also thanks ​@Victor Labian for the suggestions regarding warm-boot failure.

Tried the first, the service is now running but the card still not listed after warm-boot.

As for the second, my BIOS has billions of settings on how to overclock this and that, but unfortunately nothing regarding PCIe slot handling at reboot - priorities Asus...

The last might come handy once I observe the card hangs at later stages, but as long as it does not show up at the bus, all tools obviously don’t work.

For now I can live with the limitation and power-cycle the PC to get access. Maybe you can make it to add a warm-boot detection into a future FW to handle the reset at card side.

BTW, the two micro-switches at the upper end of the PCB (SW1, SW2) by chance are no HW-reset ones, right?