Can not Enable Update

Hi there ​@meva-ysf !

Hmm, odd that it’s not working for any of them. Looks like a PCIe communication issue with the host. To help us dig deeper, can you provide the following?

• What host systemare you using?
• What's your OS and kernel version?
• Driver version: cat /sys/class/metis/version
• Is the driver loaded? lsmod | grep metis
• And are all three in one host system, or in individual computers?

👍

Thank you for your interest ​@Spanner 

I connected the M2 Metis device to my Lenovo ThinkStation P2 Tower computer. All three devices are installed in this computer.

Ubuntu Version : 22.04.5 LTS x86_64

Kernel version   : 6.8.0-90-generic.

CPU                   : Intel i7-14700K (28) @ 5.500GHz. 

ysf@ysf:~$ cat /sys/class/metis/version -> 1.4.4

ysf@ysf:~$ lsmod | grep metis -> metis                 147456  0

Hi ​@meva-ysf , can you provide the output of running `sudo dmesg` as a txt file here so we can analyse it and look for errors?

Hi, thank you for your interest ​@Victor Labian 

The output of command “sudo dmesg” is in the dmesg_output.txt file. 

Morning ​@meva-ysf , thanks for sharing that!

​@Victor Labian is waaaaaay better at this than I am, but I had a quick butcher’s at the output file, and from a glance it looks like only one of the three M.2 boards is being enumerated - are all three slots turned on in the system BIOS? Seems probably that they are, but worth double checking maybe.

Alternatively, I wonder if it’s worth rotating them, to see if it’s a specific card that’s working, or a specific slot?

And a full power cycle reboot followed by an axdevice --refresh never hurts, although I suspect you’ve done that multiple times already 😅

Good morning! ​@Spanner  I hope you're having a good day. I’ve currently installed only one of the three Metis cards. When i tried the other two one by one, I got the same result for all of them. I'm swapping the slot for the current card and re-sharing the dmesg output. Here is also the output for axdevice --refresh: 

(venv) ysf@ysf:~/Desktop/voyager-sdk$ axdevice --refresh 
[sudo] password for ysf: 
0000:02:00.0 : Axelera AI Metis AIPU (rev 02)
[libtriton_linux.c:1313] Device communication timed out: device did not respond within 1 seconds. (0)
WARNING: Failed to get valid board type for device metis-0:2:0 got 8
[libtriton_linux.c:1313] Device communication timed out: device did not respond within 1 seconds. (0)
ERROR: Failed to load runtime stage0: /opt/axelera/device-1.5.2-1/omega/bin/start_axelera_runtime_stage0.bin
ERROR: AXR_ERROR_CONNECTION_ERROR: Failed to load runtime stage0: /opt/axelera/device-1.5.2-1/omega/bin/start_axelera_runtime_stage0.bin
 

Hi ​@meva-ysf ,

Thank you for sharing the output of sudo dmesg.

I have inspected it and I can see that:

  - Driver loads successfully: "Triton Linux Driver, version 1.4.4, init OK" (line 1215)
  - Device detected and registered correctly
  - MSI interrupts registered (32 vectors)
  - Memory BARs mapped successfully
  - No DMA errors, no timeout errors, no communication failures in dmesg

Can you share the output for the following command with us?

sudo lspci -vvv -s 02:00.0
 

​@meva-ysf Do you have by any chance another PC in which you can test the Metis card to see if there is the same issue?

Hi ​@Victor Labian ,

Output of the command is : 

02:00.0 Processing accelerators: Axelera AI Metis AIPU (rev 02)
    Subsystem: Axelera AI Metis AIPU (rev 02)
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 165
    Region 0: Memory at 81010000 (64-bit, non-prefetchable) [size=4K]
    Region 2: Memory at 82000000 (32-bit, non-prefetchable) [size=32M]
    Expansion ROM at 81000000 [disabled] [size=64K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable+ Count=32/32 Maskable+ 64bit+
        Address: 00000000fee00818  Data: 0000
        Masking: 00000000  Pending: 00000000
    Capabilities: [70] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
        DevCtl:    CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
            RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <16us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl:    ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 8GT/s (ok), Width x4 (ok)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
             10BitTagComp- 10BitTagReq- OBFF Via message, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
             EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [100 v2] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq+ ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        CEMsk:    RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+
        AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [148 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn- PerformEqu-
        LaneErrStat: 0
    Capabilities: [168 v1] Latency Tolerance Reporting
        Max snoop latency: 3145728ns
        Max no snoop latency: 3145728ns
    Capabilities: [170 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
              PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
               T_CommonMode=0us LTR1.2_Threshold=108544ns
        L1SubCtl2: T_PwrOn=42us
    Capabilities: [180 v1] Vendor Specific Information: ID=0006 Rev=0 Len=018 <?>
    Kernel driver in use: axl
    Kernel modules: metis

Also when i try to setup Voyager SDK 1.5.2 to my Aetina Board, lsmod | grep metis command returns nothing. So i can not run my Axelera Metis device on Aetina Board. 

Can you help me to solve this problem ? 

Hi ​@meva-ysf , 

Sorry for the delay in replying, we have been a bit busy.

The lspci output reveals the root cause:

LnkSta: Speed 8GT/s (ok), Width x4 (ok)
        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

DLActive- indicates that the PCIe Data Link Layer is NOT active. This prevents any communication between the host and device, even though the physical link appears trained.

There are still some things we can try:

lspci -tv and bridge removal _+ pcie rescan

Can you please share with us the output of lspci -tv ?

I want to see the pcie topology to identify the bridge in which the Metis card is connected to so I can then provide you the commands to do a bridge removal and a rescan and see if that helps.

BIOS settings

Please try to make sure the BIOS is updated, that may cause issue sometimes.

Then check:

• Check PCIe Slot Configuration (if available)

  • Look for "PCIe Bifurcation" settings
  • Ensure slots are configured for x4 or x8 mode

Make sure you safe and exit.

Kernel Boot Parameters

1. Edit GRUB configuration

   sudo nano /etc/default/grub
   

2. Modify the kernel command line

   Find the line starting with GRUB_CMDLINE_LINUX_DEFAULT and change it to:

   • intel_iommu=off: Disables IOMMU (already present, keep it)
   • pcie_aspm=off: Disables PCIe Active State Power Management

3. Update GRUB and reboot

   sudo update-grub
   sudo reboot

Hi ​@Victor Labian 
           
There is no Above 4G Decoding and Resize Bar Support Option in my BIOS, but i enabled SRIOVSupport function : 

ysf@ysf:~$ sudo cat /sys/class/firmware-attributes/thinklmi/attributes/SRIOVSupport/current_value
Enabled

Also i configured PCIe Slot Speeds as Auto : 
ysf@ysf:~$ sudo cat /sys/class/firmware-attributes/thinklmi/attributes/PCIe16xSlotSpeed/current_value
Auto
ysf@ysf:~$ sudo cat /sys/class/firmware-attributes/thinklmi/attributes/PCIe4xSlotSpeed/current_value
Auto
ysf@ysf:~$ sudo cat /sys/class/firmware-attributes/thinklmi/attributes/PCIe1xSlot1Speed/current_value
Auto
ysf@ysf:~$ sudo cat /sys/class/firmware-attributes/thinklmi/attributes/PCIe1xSlot2Speed/current_value
Auto

ASPM option is disabled : 
ysf@ysf:~$ sudo cat /sys/class/firmware-attributes/thinklmi/attributes/ASPMSupport/current_value
Disabled

Fast Boot is disabled : 
ysf@ysf:~$ sudo cat /sys/class/firmware-attributes/thinklmi/attributes/FastBoot/current_value
Disabled

Content of the /etc/default/grub file is that  and i updated the grub: 
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=off pcie_aspm=off"
GRUB_CMDLINE_LINUX=""

After setup all of this setting lspci -tv command output is that : 

-[0000:00]-+-00.0  Intel Corporation Raptor Lake-S 8+12 - Host Bridge/DRAM Controller
           +-01.0-[01]--+-00.0  NVIDIA Corporation AD107 [GeForce RTX 4060]
           |            \-00.1  NVIDIA Corporation AD107 High Definition Audio Controller
           +-02.0  Intel Corporation Raptor Lake-S GT1 [UHD Graphics 770]
           +-04.0  Intel Corporation Raptor Lake Dynamic Platform and Thermal Framework Processor Participant
           +-14.0  Intel Corporation Alder Lake-S PCH USB 3.2 Gen 2x2 XHCI Controller
           +-14.2  Intel Corporation Alder Lake-S PCH Shared SRAM
           +-16.0  Intel Corporation Alder Lake-S PCH HECI Controller #1
           +-16.3  Intel Corporation Alder Lake-S Keyboard and Text (KT) Redirection
           +-17.0  Intel Corporation Alder Lake-S PCH SATA Controller [AHCI Mode]
           +-1a.0-[02]----00.0  Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
           +-1b.0-[03]----00.0  Axelera AI Metis AIPU (rev 02)
           +-1f.0  Intel Corporation W680 Chipset LPC/eSPI Controller
           +-1f.3  Intel Corporation Alder Lake-S HD Audio Controller
           +-1f.4  Intel Corporation Alder Lake-S PCH SMBus Controller
           +-1f.5  Intel Corporation Alder Lake-S PCH SPI Controller
           \-1f.6  Intel Corporation Ethernet Connection (17) I219-LM

Hi ​@meva-ysf 

Thank you for sharing all that information.

Can you now go to voyager-sdk, activate the virtual environment and try the following commands:

axdevice -v

axdevice --refresh

Please share the outputs with us.

Can you also share lspci

(without any options)

Kind regards,

Victor

Hi ​@Victor Labian 

Output of the axdevice -v is that : 
INFO: Found PCI device: 03:00.0 Processing accelerators: Axelera AI Metis AIPU (rev 02)
INFO: Found AIPU driver: metis                 147456  0
[libtriton_linux.c:1313] Device communication timed out: device did not respond within 1 seconds. (0)
WARNING: Failed to get valid board type for device metis-0:3:0 got 8
INFO: Ignoring device 0(metis-0:3:0) as it has not returned a valid board type: 8
INFO: Using device metis-0:3:0
Device 0: metis-0:3:0 board_type=unknown (not responding)

Output of the axdevice --refresh is that :
0000:03:00.0 : Axelera AI Metis AIPU (rev 02)
[libtriton_linux.c:1313] Device communication timed out: device did not respond within 1 seconds. (0)
WARNING: Failed to get valid board type for device metis-0:3:0 got 8
[libtriton_linux.c:1313] Device communication timed out: device did not respond within 1 seconds. (0)
ERROR: Failed to load runtime stage0: /opt/axelera/device-1.5.2-1/omega/bin/start_axelera_runtime_stage0.bin
ERROR: AXR_ERROR_CONNECTION_ERROR: Failed to load runtime stage0: /opt/axelera/device-1.5.2-1/omega/bin/start_axelera_runtime_stage0.bin

Output of the lspci is that : 
00:00.0 Host bridge: Intel Corporation Raptor Lake-S 8+12 - Host Bridge/DRAM Controller (rev 01)
00:01.0 PCI bridge: Intel Corporation Raptor Lake PCI Express 5.0 Graphics Port (PEG010) (rev 01)
00:02.0 Display controller: Intel Corporation Raptor Lake-S GT1 [UHD Graphics 770] (rev 04)
00:04.0 Signal processing controller: Intel Corporation Raptor Lake Dynamic Platform and Thermal Framework Processor Participant (rev 01)
00:14.0 USB controller: Intel Corporation Alder Lake-S PCH USB 3.2 Gen 2x2 XHCI Controller (rev 11)
00:14.2 RAM memory: Intel Corporation Alder Lake-S PCH Shared SRAM (rev 11)
00:16.0 Communication controller: Intel Corporation Alder Lake-S PCH HECI Controller #1 (rev 11)
00:16.3 Serial controller: Intel Corporation Alder Lake-S Keyboard and Text (KT) Redirection (rev 11)
00:17.0 SATA controller: Intel Corporation Alder Lake-S PCH SATA Controller [AHCI Mode] (rev 11)
00:1a.0 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #25 (rev 11)
00:1b.0 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #21 (rev 11)
00:1f.0 ISA bridge: Intel Corporation W680 Chipset LPC/eSPI Controller (rev 11)
00:1f.3 Audio device: Intel Corporation Alder Lake-S HD Audio Controller (rev 11)
00:1f.4 SMBus: Intel Corporation Alder Lake-S PCH SMBus Controller (rev 11)
00:1f.5 Serial bus controller: Intel Corporation Alder Lake-S PCH SPI Controller (rev 11)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (17) I219-LM (rev 11)
01:00.0 VGA compatible controller: NVIDIA Corporation AD107 [GeForce RTX 4060] (rev a1)
01:00.1 Audio device: NVIDIA Corporation AD107 High Definition Audio Controller (rev a1)
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
03:00.0 Processing accelerators: Axelera AI Metis AIPU (rev 02)

Best regards,

Yusuf

Hi ​@meva-ysf ,

Thank you for sharing that. 

Let’s try one last test.

Please execute the following commands to remove the PCIe bridge and rescan:

echo 1 | sudo tee /sys/bus/pci/devices/0000:00:1b.0/remove

echo 1 | sudo tee /sys/bus/pci/rescan

Then try again to run axdevice -v

If this doesn’t work I will communicate internally with my team, as we might need to send a replacement card to check if the cards you received had by mistake an old FW or if it is specific issue with this host system (FYI ​@Spanner ).

Hi, ​@Victor Labian 

The output is that : 

(venv) ysf@ysf:~/Desktop/voyager-sdk/venv/bin$ echo 1 | sudo tee /sys/bus/pci/devices/0000:00:1b.0/remove
1
(venv) ysf@ysf:~/Desktop/voyager-sdk/venv/bin$ echo 1 | sudo tee /sys/bus/pci/rescan
1
(venv) ysf@ysf:~/Desktop/voyager-sdk/venv/bin$ axdevice -v
INFO: Found PCI device: 03:00.0 Processing accelerators: Axelera AI Metis AIPU (rev 02)
INFO: Found AIPU driver: metis                 147456  0
[libtriton_linux.c:1313] Device communication timed out: device did not respond within 1 seconds. (0)
WARNING: Failed to get valid board type for device metis-0:3:0 got 8
INFO: Ignoring device 0(metis-0:3:0) as it has not returned a valid board type: 8
INFO: Using device metis-0:3:0
Device 0: metis-0:3:0 board_type=unknown (not responding)

If you can send a replacement card immediately we will be very happy about that because we want to try these cards on our products.

Thank you for your support.