The Problem
Autonomous vehicles and robots need to understand the world in 3D. The industry standard is BEVFusion, fusing multiple cameras and LiDAR into a unified bird's-eye-view representation for 3D object detection. Every production implementation runs on NVIDIA hardware. The cost, power draw, and vendor lock-in are significant barriers for robotics at scale.
What I Want to Build
A fully working BEVFusion pipeline on a single Axelera Metis PCIe card. Six camera backbones, LiDAR processing, BEV feature fusion, and a 3D detection head, running in real time without any NVIDIA hardware.
The model is FastBEV https://github.com/ymlab/advanced-fastbev), a CNN-only BEV fusion architecture with no transformer attention, designed specifically for efficient hardware deployment. No attention means no DPU fusion issues. Pure Conv-BN-ReLU throughout.
The deployment plan:
- Six camera backbones run on six parallel AIPU instances, one per core. Metis has 4 cores on the M.2 and 16 cores on the PCIe card.
- Each backbone processes one camera view simultaneously, true parallelism across cores.
- BEV feature lifting runs on the host CPU.
- LiDAR voxelization and feature extraction on remaining AIPU cores or host.
- Fusion and detection head on AIPU.
Why This Matters
This is the entire autonomy perception stack, not a demo, not a toy. A working BEVFusion deployment on Metis means a self-driving or robotic system can run its full perception pipeline on a single chip drawing under 30W, at a fraction of the cost of an Orin or H100 cluster.
Why I Can Do This
I have already deployed two models on Axelera Metis from scratch:
MonoCon, single-camera 3D detection at 30 FPS: https://github.com/sanket-pixel/monocon-metis
ZipDepth, monocular depth estimation at 30 FPS on a 5W M.2 card: https://github.com/sanket-pixel/zipdepth-metis
Both involved systematic compiler debugging, custom C++ inference libraries, multi-segment AIPU pipelines, and split graph compilation with CPU bridges. I understand the Voyager compiler constraints, the DPU fusion rules, and the axruntime C++ API in detail.
Extending to multi-camera is the natural next step. The backbone architecture is identical to what I have already compiled. The difference is running six instances in parallel across cores, which the Metis PCIe card with 16 cores is purpose-built for.
What the Submission Will Include
- FastBEV exported and compiled for Metis, all six camera backbones running simultaneously
- Real-time inference on nuScenes or custom dataset
- C++ inference library with multi-core pipeline
- Full Voyager Wingman prompt log showing the build process
- Live demo video with 3D bounding boxes in bird's-eye view
- Open source repo
The Bigger Picture
NVIDIA owns autonomous vehicle perception today. This project is a proof of concept that the entire stack, not individual models but the full multi-sensor fusion pipeline, can run on Axelera hardware at real-time speeds. That changes what is possible for robotics companies who cannot afford or do not want Orin-based systems.
THE PROMPT
Alright Wingman, here is the mission. We are building BEVSight: a full autonomous vehicle perception system running live on Axelera Metis PCIe, processing six cameras and one LiDAR simultaneously and publishing 3D detections in bird's-eye-view. Build it in stages, walk me through each one, and ask me before assuming anything about the camera configuration or dataset format.
1. Single camera backbone. Generate a Voyager pipeline for one camera backbone from FastBEV, a CNN-only BEV fusion model with no transformer attention, running on one Metis AIPU core. Input is a single camera image at the resolution used in the nuScenes dataset. Output is the feature map before BEV projection. Show me a live latency breakdown per stage from the card itself and validate the output matches the float32 ONNX reference before moving on.
2. Six cameras in parallel. Scale the backbone pipeline to six camera instances running simultaneously, one per AIPU core. All six share a single axrContext. Output is six feature maps produced in parallel within one inference cycle. Show me wall-clock latency for the full six-camera pass and confirm no core contention.
3. BEV feature lifting on host. Add the BEV projection stage that lifts the six camera feature maps into a unified bird's-eye-view grid using the camera intrinsics and extrinsics from nuScenes. This stage runs on the host CPU. Output is a single BEV feature volume ready for fusion.
4. LiDAR voxelization and feature extraction. Add LiDAR point cloud processing. Voxelize the input, run a compact CNN-based voxel feature encoder on a spare AIPU core if it compiles cleanly, otherwise run it on host. Fuse the LiDAR BEV features with the camera BEV features from stage 3.
5. Fusion and 3D detection head. Add the BEV fusion neck and 3D detection head on AIPU. Output is 3D bounding boxes with class, position, size, and orientation in the LiDAR coordinate frame. Validate against nuScenes ground truth and report mAP before and after INT8 quantization so I can see the accuracy cost clearly.
6. Tune it. Once the full cascade runs end to end, help me optimize: compiler configurations, core assignment, resolution trade-offs, and pipeline ordering. Target is real-time inference at the nuScenes sweep rate of 12.5 Hz, and show me where every millisecond goes across cameras, LiDAR, BEV lifting, fusion, and detection.
