The Problem
More than 2.2 billion people live with vision impairment. Current assistive tools either provide very limited information (white cane), rely on the cloud (raising latency and privacy issues), or lack robust real-time hazard detection — especially for moving objects, overhead obstacles, sudden drop-offs, and complex indoor/outdoor environments. Users need a system that can detect dangers before they become critical, describe the scene, and do it entirely offline.
The Idea
SightCompanion is a fully on-device assistive vision system built with Voyager Wingman on Axelera Metis hardware. Its primary focus is real-time multi-layer hazard detection and prioritization, combined with contextual scene understanding and audio feedback.
Core capabilities:
- Multi-class hazard detection (moving people/vehicles, static obstacles, overhead obstacles, floor-level hazards, sudden elevation changes)
- Distance-aware risk prioritization (closer + faster-moving objects get highest priority)
- Semantic scene understanding and concise natural-language descriptions
- Selective OCR for signs and text
- Fully offline operation — zero cloud dependency for maximum privacy and reliability
Why this is a strong fit for Voyager Wingman + Axelera
This project requires cascading multiple models, real-time multi-stream processing, custom prioritization logic, and tight latency control — exactly the kind of complex pipeline where Wingman can shine. I will use natural language prompts to rapidly generate, iterate, and optimize the full pipeline on Metis, allowing me to focus on hazard prioritization logic and real-world robustness rather than low-level YAML and compiler issues.
Technical Pipeline Overview
1. Input: Live camera stream (USB or MIPI)
2. Primary Vision Stage (Metis):
- YOLOv8 / YOLOv9-style object detection fine-tuned for hazard classes (person, vehicle, bicycle, chair, pole, overhanging object, stairs, curb, etc.)
- Semantic segmentation for floor / free-space / obstacle regions
- Monocular depth estimation for distance and time-to-contact calculation
3. Hazard Reasoning Layer (host + Metis):
- Fuse detection + segmentation + depth
- Classify risk level based on distance, relative speed, and object class
- Priority queue: Critical (immediate collision risk) → Warning → Informational
4. Secondary Tasks (triggered selectively to save compute):
- OCR on detected text regions
- Lightweight scene captioning for richer context when no critical hazards are present
5. Output: Priority-based audio feedback (urgent hazards first, spoken clearly and concisely)
I plan to leverage multiple Metis AI cores to run detection, segmentation, and depth estimation in parallel where possible, targeting real-time performance (≥15–20 FPS end-to-end) with low latency audio alerts.
Development Approach with Wingman
I will start with high-level prompts to generate the base multi-model pipeline, then iteratively refine:
- Hazard class selection and prioritization logic
- Depth fusion and risk scoring
- Latency optimization and core allocation
- Audio feedback formatting
All prompts, intermediate pipelines, and iterations will be documented as required by the challenge.
Expected Impact
A working prototype that demonstrates how Axelera’s high-performance, power-efficient edge AI can deliver genuinely useful, privacy-preserving assistive technology — with a strong emphasis on proactive hazard detection that current solutions still struggle with.
I’m excited to push Voyager Wingman on a demanding, multi-stage, real-time computer vision pipeline that has clear social value.
Looking forward to building this and sharing the journey with the community.
