Zero-Shot Monocular Depth Estimation on Axelera Metis

When a robot moves about minding its business, seeing is only half the problem. The other half is how far. Monocular depth estimation solves this with just a single camera. Feed it an image, and it produces a dense depth value for every pixel in the scene.

A recent zero shot model Zipdepth, solved this problem for tiny embedded deivces. From NPUs to even mobile phones, realtime at ridiculous speeds.

I attempted to run ZipDepth on my newly bought Axelera Metis. A chip the size of my index finger, running at under 5W. The result: 30 FPS end to end, real-time inference with minimal accuracy loss compared to FP32.

The deployment was not easy to put it mildly. As always, anything that fails to compile on the accelerator moves to the host. The model had to be split into 4 parts, with 2 running on the AIPU, and the other 2 on the host. This was my first attempt to run a cascaded model on Axelera Metis, where the output of one compiled graph feeds directly into the next. It worked surprisingly well. Another key optimization was pinning the encoder and decoder to separate cores, keeping both busy across frames..

Another new technique, I attempted to use non-image ( embedding ) input to one of the AIPU models. The compiler and quantization tool from Voyager SDK churned the non-image input and got lossless inference compared to FP32. Simply crazy how robust the overall compilation frameowkr is. The AIPU parts that cover 95% of the model works at 110FPS. Only the host parts running on CPU brings it t o 30FPS end to end.

This model can be used in robotic navigation with a single camera sensor for obstacle avoidance, floor plane estimation for safe traversal, and bin picking where the robot needs to judge which object sits closest before attempting a grasp.

In the demo, depth inference runs on a recorded drone flight using a single camera feed. The depth map correctly identifies the closer pillar in dark orange and the farther one in lighter yellow, accurately resolving the depth ordering between two objects at different distances in the same scene.

Many more interesting robotics perception models, deployed on Axelera Metis coming up.

Upwards and onwards

Github : https://github.com/sanket-pixel/zipdepth-metis