Hi, I just bought 113m and trying to profile to find the bottlenecks.
Following is my setting.
Are there ways to enable per layer level profiling?
Environment
- Board: Axelera Embedded 113m (Metis M.2), PCIe 03:00.0
- Host A: Radxa Dragon Q8B, Ubuntu 26.04, kernel 7.0.11-qcom (aarch64)
- Host B: x86_64 Ubuntu 26.04, kernel 7.0.0-43-generic, Secure Boot enabled
- Voyager SDK 1.9.0 (
axelera-rt,axelera-devkit[all]) - Firmware:
flver=1.9.0,bcver=7.4(host B);flver=1.8.0flashed with runtime firmwarev1.9.0loaded at connect (host A) - Driver:
metis-dkms1.7.16 (Ubuntu 24 apt suite on Ubuntu 26.04)
What I’m trying to do
Get per-layer or per-kernel device timing for a compiled model (a 1×1 conv + ReLU, 64 channels, int8).
What I tried
- Pipeline
OperatorProfiler/axzoo benchmark yolov8n-coco: works, but only per operator (AxRuntimeModelis one row, flagged DEVICE WAIT). axruntimeinstance withdevice_profiling=Trueand/orhost_profiling=True: no device-side output found. Host trace (AXE_LZ_PERF_LOG_PATH,AXE_PROFILING_CONFIG=HOST_TEMPLATE) works and gives per-frame kernel time.axtrace --atrace 0 --atrace-target 0:datapathand0:megakernel, running 200 frames: the output file stays at 0 bytes on both hosts, with and without the instance profiling flags.axtrace --debugshows only the device selection.axlogdevice --atrace 0 --atrace-target 0:inf: the device returns an error forCMD_TRACE_AICORE.axlogdevice --alog 0 --alog-level 0:inf:runtime: no output while frames run.axlogdevice --filefails witha bytes-like object is required, not 'str'.- Level Zero tools API (
zetGetMetricGroupProcAddrTable,zetGetTracerExpProcAddrTable,zetGetKernelProcAddrTable,zetGetCommandListProcAddrTable): calls return success, but every function pointer in those tables is NULL.
Questions
- Is AI-core trace (
axtrace --atrace) supported on Metis M.2 with firmware 1.9.0 and driver 1.7.16? If yes, what must be enabled for it to record (firmware build, property, or boot option)? - Does
device_profiling=Truehave any effect on the device? If it’s supposed to produce per-kernel timings, where do they go? - Is there a supported way to get per-layer or per-kernel device timing on Metis M.2 with SDK 1.9? If not, is one planned?
- Is the
axlogdevice --filestring/bytes error known?
I can share the probe programs and the logs if they help.
