Low Utilization on Portenta

Hi ​@nick !

Hmm, with utilisation at ~6% even with pre/post-processing removed and a dummy zero input, that looks like it rules out the host pipeline and points at the compile config. Good work there.

By default Voyager compiles models for single-core execution if I recall, so only one core does work while axmonitor reports per-core usage across all four (which is consistent with what you're seeing).

How about recompiling for multi-core (Batch-1), so the runtime instantiates a copy per core and dispatches frames across all four?

compilation_config:
  aipu_cores_used: 1
  resources_used: 0.25   # 1.0 / 4 cores

(The same settings can be passed via CompilerConfig with the Python API.)

If you specifically want lowest single-stream latency on one heavy model, try Batch-4 instead (aipu_cores_used: 4, resources_used: 1.0).

Let me know if this moved the needle at all!

Thanks for your reply!

I have updated the board to v1.7 according to the published guide. I have recompiled the model on the newer sdk version using the config for multi-core you’ve provided, performance is the same though. I had some issues with the pip install crashing a couple of times due to OOM, so I think my install is not complete as axmonitor is missing. But when running inference I get the same throughput as before. 

I have however since found out about `axmodeltool`. When running it on the model I have, it does say some kernels are not pre-compiled. Could this be the reason for the slower inference? I am using a U-Net architecture with MobileNetV2 as the encoder. Is there a known issue with this architecture?
 

axmodeltool .
L2: 7.66MiB DDR: 57.51MiB

--------------------------------------------------------------------------------------
  Tensor Name     Unpadded Shape Padding          Quant Scale/Zp Size                 
--------------------------------------------------------------------------------------
< var_input_ifd0  1,512,512,1    0:0,1:1,1:63,0:0 0.007542/5     289.12KiB (296,064)  
> output_mmio_var 1,512,512,1    0:0,0:0,0:0,0:63 1.096289/-95   16.00MiB (16,777,216)
  TOTAL                                                          16.28MiB (17,073,280)
--------------------------------------------------------------------------------------

Some of the kernels in the model have not been pre-compiled.
Consider fixing using

  axmodeltool --update-kernels .

Running the command with the update kernels flag, on either the portenta or the machine I used for compilation, gives the following error:

Failed to run axkernelcc to get version: [Errno 2] No such file or directory: 'axkernelcc'
Error: Cannot update kernels because axkernelcc version is unknown

During model compilation there are no errors printed, except that it does multiple attempts for LowerTIR.

I’m currently compiling the model in batch-4, I will post here if it makes a difference.

Small update:
I managed to get axmonitor working. The utilization seems to have dropped even further to 2% on four cores using batch-1. Also, upon closer inspection, my previous comment about the throughput being the same is false. It is almost exactly half of what it was on v1.4.

batch-4 also utilizes 2% across 4 cores. I had to change compilerflag MulticoreMode to batch in order to compile it for batch-4 since modes cooperative and pipeline failed to start compilation.

ValueError: Unsupported multicore mode: cooperative
ValueError: Unsupported multicore mode: pipeline

CPU usage is also lower than with version 1.4, but the python process spikes to 400% occasionally for just one update cycle. I think these spikes last only a couple milliseconds, as the CPU monitor updates every 100 milliseconds.
Before inference starts, it logs this warning:

INFO:axelera.runtime:Found AIPU driver: metis                 143360  2
INFO:axelera.runtime:Current firmware version v1.3.0-stage0 != required version v1.7.0
INFO:axelera.runtime:Device firmware version is not compatible, loading now

cat /sys/class/metis/version
1.5.5

I am not completely sure, but I think earlier today I did not get this message when running inference, which runs exactly as described above after this message. Version 1.5.5 is shown both in the container and on the host yocto image.

I have included some more logs from axmonitor and axmodeltool.
Let me know if there are any more useful details I can share. Thanks in advance for the help. 

Thanks for the thorough update, ​nick! A few leads come to mind, that it might be interesting to test out.

First, I'd take that 6% with a pinch of salt on this board. The Portenta X8 shows up in the v1.7 release notes' known issues, that AxMonitor isn't supported on "single-MSI hosts," with the Portenta X8 Mini given as an example. So the low reading might be more about the host than your model sitting idle. I'd lean on the System FPS from axrunmodel/inference.py as your source of truth instead. 👍

Second, the thing I found myself staring at in your axmodeltool output is the output tensor. It's a chunky FP32 tensor (~16 MB a frame), which for a segmentation model is often just raw per-pixel scores coming back every frame. It could be that shifting more of that final step onto the device is where the time's actually going, which would also fit with commenting out pre/post making no difference. I think the segmentation decoder has options for doing that reduction on-device (something around a threshold for binary/1-class cases), so that might be a rabbit hole worth getting lost in! 😄

On the axkernelcc side, given your install hit those OOM hiccups and axmonitor went missing too, I’m wondering if that's an incomplete install thing rather than your model? If possible, I'd retry --update-kernels from a clean, full install on your build machine (with some swap if memory's tight); if it still complains, that's one to dig into deeper with the team. The repeated LowerTIR messages look like the compiler just working through its normal stages, so I wouldn't worry there.

Keen to hear what you find 👍

Thanks for the quick response!

How much fps should I expect to get from the Metis on a 512x512 segmenting model?

The output appears to be in int8 format already, the large size is due to applied padding. Taking the padded size gives the exact amount of bytes axmodeltool shows: `1 × 512 × 512 × 64 = 16777216`. I’ve attached the manifest json, which contains more info on the input and output shapes. 

Is there any way to slice off these channels on the Metis?

Checking the PCIe bus speed confirms your suspicion on the frame size being the problem. lspci output for the metis device:

LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <16us
LnkSta:	Speed 2.5GT/s (downgraded), Width x1 (downgraded)

The new image I compiled is probably forcing PCIe gen 1, while the original image was using PCIe gen 2. That, together with the 16 MB framesize, would match the PCIe speeds.

Perhaps I can modify the model such that less padding is required. Is there any documentation on how the compiler decides how tensors are padded? 

I retried the update kernel flag on my build machine with a new venv, but it fails in the same way. This build machine has plenty of memory and installation goes well, the OOM was on the portenta. For installation I followed all steps of the pip installation guide on github, including step 3 for the kernel module.