The Prompt Challenge - Port visual ROI tracking algorithm to Metis - SiamFC

Backstory:

It has been on my wish-list since last year to port a visual ROI tracking algorithm on Metis. I have posted about this before.

To be clear, visual ROI tracking algorithms are those that do not require an object detector, meaning you should be able to point them at anything and they track the object inside selected ROI (region of interest) in subsequent frames. You can imagine these are very useful for cases when you encounter objects that are not in the trained classes of your typical detection models.

Significance:

Since Metis is neither a general-purpose processor nor a GPU, accelerating typical OpenCV tracking algorithms on it is not feasible. Fortunately, researchers have developed neural networks for a wide range of tasks, including tracking. Metis is small, incredibly fast, and highly efficient, and its multiple cores can run several models simultaneously. So when it's already part of the system, why not use it to accelerate multiple tasks and leave the CPU to do easier things? 

When Axelera released Wingman (I was an early tester), I was excited because I could finally try porting, among other things, an ROI tracking algorithm to Metis.

And so began my journey...

Prompt(s):

I started with the most basic prompt. I even let the wingman help me choose:

I want to implement a visual object tracking algorithm on the Metis. To be clear, I am not talking about an object tracker that uses an object detector first, but a purely visual ROI tracker. My requirement is being able to track at least one selected ROI at a time (more is better but not necessary) and speed is critical. I want something that can be accelerated and run with very high FPS and low latency on the Metis. Review all the available options, cross check their architecture against what best suits the Metis architecture, and then tell me options sorted by most to least desirable. 2-3 top options at max. Any selected option must be possible to implement on Metis without any major roadblock.

Wingman came back with a table of options, with the most recommended family being “Siamese fully-convolutional (SiamFC, SiamRPN, SiamRPN++)”. It told me this was because the backbone is fully convolutional and head is cross-correlation of two feature maps, which was an excellent fit for Metis. 

At the time I started this, the on-device option was not there (only chat), so my next prompt was:

Let's deploy and test SiamFC on my RK3588 board with metis M.2. Give me step-by-step procedure for this. I will copy-paste to my board and follow along.

After that I was mostly being wingman’s wingman - i.e., helped copy-paste and run the commands on the actual device and report back on errors/progress. A few approach changes happened along the way, but eventually we had a working ROI tracker accelerated on Metis. 

Note: What the video below doesn’t capture too well is that the ROI is manually selected in frame 1. The selection could have been any ROI in the frame. 

Porting journey:

Following is high-level summary of what all was done during the porting process:

• Chose SiamFC — fully convolutional, static shapes, single-ROI native → ideal for Metis INT8 MVM.

• Converted pretrained checkpoint to Metis-friendly ONNX:

  • Expanded grouped convolutions (conv2/4/5, groups=2) to equivalent dense block-diagonal kernels.
  • Folded BatchNorm (conv1–4) into preceding convolutions; kept identity BN layers for clean export graph.
  • Compensated conv1 weights ×255 (checkpoint trained on 0–255; Voyager supplies 0–1).
  • Exported two static graphs: template [1,3,127,127], search [1,3,255,255].

• Deployed with Voyager SDK:

  • Two YAMLs (AxONNXModel, identity normalization).
  • Calibration on ROI crops (~150 images per branch).
  • Quantized to INT8, compiled for Metis.

• Runtime tracker (single Python file, low-level AxRuntime API):

  • Template kernel cached; three-scale search per frame.
  • Zero-mean + L2-normalized valid cross-correlation (per-channel sum), upsampled 16× with Hann window. 

    

    Damped position/scale updates; interactive re-select (r), quit (Esc).

• Validated with fixed-ROI diagnostic runs. Achieved ~390 FPS backbone / ~50–80 FPS full tracker on Metis M.2.

•  

Final thoughts:

Overall, I was extremely impressed with wingman. During this whole process I had not needed to read a single documentation page - wingman helped me with everything. Something that would have taken me weeks to implement otherwise, I was able to accomplish in few days. This has changed the game for future experimentation and development on Axelera’s accelerators.

I have recorded all relevant files and detailed process of compiling the model via voyager in my github project:

github.com/SaadTiwana/siamfc-metis

What’s next:

My success with porting the SiamFC to Metis has given me the confidence boost I needed to pursue this further. Since visual ROI tracking is a topic that interests me (and I have actual uses for it), I plan to next go through the more start of art algorithms from the VOT (Visual Object Tracking) challenge and select some networks that are well suited for porting to Metis’ op-set. Once ported, this will give me even better algorithms for ROI tracking that don’t need to run on CPU or GPU and yet can be accelerated to insane speed on the Metis. 

I also do hope my models get picked for inclusion in the model zoo one day...for bragging rights ;)