Hi Community!
I’m incredibly proud to present the final update and the full reveal of Project GAIUS. It’s been an amazing, challenging, and deeply rewarding journey over the past few months, and I want to share the complete story of how this system evolved from a simple idea into a fully functional spatial smart home interface.
The Original Vision
From the very beginning, my goal was to bring a slice of the "Iron Man" experience into the real world. I wanted to build a system where you could walk into a room, point at a device (like a TV or a lamp), and control it with intuitive hand gestures—without remotes, without voice commands, and completely offline for maximum privacy.
With the Axelera DevKit (Orange Pi 5 Plus + Metis M.2 NPU), I finally had the edge compute power to make this a reality.
The Journey and The Pivots
If you’ve followed my previous updates, you know this wasn't a straight path.
-
The Setup: Getting the hardware running was a breeze thanks to the community guides.
-
The Struggle: I initially went down a deep rabbit hole trying to build a custom 2-stage dynamic cropping pipeline using YOLOv8-pose and the low-level
axelera.runtimeAPI. While educational, the CPU overhead was too high, and tracking 2D keypoints from 3 meters away proved unreliable. -
The Breakthrough: I took a step back, redesigned the architecture, and pivoted to a dedicated, highly robust gesture recognition model (trained on the HaGRID dataset). By migrating back to Voyager's high-level API, the pipeline became incredibly fast, efficient, and reliable.
The Final Evolution: True 3D Spatial Understanding
In my last post, I mentioned successfully connecting gestures to Home Assistant. But since then, the project has taken a massive leap forward. GAIUS is no longer just a 2D gesture recognizer; it is a true 3D spatial engine.
Here is what the final architecture looks like today:
-
Multi-Camera Triangulation: The system takes 2D pose data from multiple RTSP cameras and uses OpenCV and epipolar geometry to triangulate a 3D skeleton of the user in real-time.
-
Arm-based Raycasting: Instead of just recognizing gestures, GAIUS knows what you are looking at. By calculating a 3D vector from your Shoulder through your Elbow and Wrist, it casts a virtual "ray" into the room to detect exactly which real-world object you are pointing at.
-
Custom GUI Room Designer: I built a Tkinter-based visual mapping tool. You can simply drag and drop your cameras and smart devices (like bounding boxes for your TV or lights) into a digital twin of your physical room.
-
Advanced Integrations: Beyond standard Home Assistant light controls, GAIUS directly connects to LG WebOS TVs via WebSockets. It translates your 3D wrist movements into smooth 2D on-screen cursor movements (a virtual mouse) and sends visual Toast notifications right to your screen when you lock onto a target!
To handle the natural jitter of human movement, I also implemented a robust "Three-Tier Armor" system featuring spike filters, heavy Exponential Moving Average (EMA) smoothing, and deadzones. The result is a smooth, futuristic interaction.
Open Source & Documentation
I am thrilled to announce that the entire project—complete with an in-depth Architecture breakdown, Setup Guide, and Calibration Manual—is now published and open source!
You can check out the details, code, the 3D math, and the release files here:
https://github.com/KatusDavid/Axelera_Project_GAIUS
Here is a demo video:
Thank You!
I want to express my deepest gratitude to the Axelera team for selecting me as one of the 10 participants and providing this incredible hardware. Exploring the capabilities of the Voyager SDK has pushed my boundaries as a developer.
Edge AI is the future of smart homes, and I can't wait to see what we all build next!
Cheers, David Katus
