HomeCare-Wingman: Multimodal Edge-AI, Wearable Sensor Fusion & Telegram Incident Response for Senior Safety

1. Project Overview As global populations age, millions of seniors live independently or spend long hours alone at home while family members are at work. Accidents like sudden falls or nocturnal disorientation pose severe health risks if not detected and responded to immediately.

HomeCare-Wingman is a privacy-first, multimodal Edge-AI safety ecosystem designed for senior homecare. Powered by Axelera Metis, the system combines real-time vision pose estimation, local voice command recognition, smart IoT room actuation, wearable sensor fusion, and instant mobile incident management to prevent falls before they occur and deliver immediate emergency alerts when they do.

2. The Problem with Current Solutions Existing cloud-connected home security cameras suffer from processing latency, internet dependency, and severe privacy invasion (seniors strongly object to having raw video feeds of their bedrooms uploaded to third-party cloud servers). Furthermore, single-sensor systems suffer from high false-alarm rates or critical coverage blindspots (e.g., bathrooms).

3. Key Features & System Architecture

• Early Bed-Egress Alert (Vision): Uses keypoint pose estimation to detect when a senior sits at the edge of the bed or attempts to stand up in low-light conditions, automatically triggering pathway lighting before they take a step.
• Local Voice Command Recognition (Audio): Allows seniors to speak natural commands (e.g., "I want to go to the bathroom") processed 100% locally to activate smart room lighting and guidance systems without struggling to find physical wall switches.
• Instant Fall Detection & Telegram Alert: Continuously tracks sudden changes in body pose vectors and velocity. Detects floor falls instantly and triggers lightweight webhook notifications (via a dedicated Telegram Emergency Bot) with privacy-masked snapshot alerts sent directly to family members or partner homecare nurses.
• Zero-Camera Toilet Safety (ESP32 DevWatch Blindspot Protection): Toilets and bathrooms represent the highest fall-risk areas for seniors, yet placing cameras inside them is a severe violation of privacy. A wrist-worn ESP32 DevWatch serves as a wearable fall detector in camera-free zones, routing IMU impact signals to the central Axelera Edge hub.
• Privacy-Preserving Skeleton View: Converts live camera feeds into real-time skeleton pose overlays or blurred video streams locally. Family members can verify the senior's safety from a dashboard/bot without compromising their dignity or privacy.
• IoT Physical Actuation: Connects seamlessly with an ESP32 microcontroller via MQTT/HTTP to trigger physical relays, smart LED nightlights, and local warning sirens.
• Two-Way Emergency Deactivation & Acknowledgment: Solves the alarm-reset loop by offering dual deactivation methods an inline response button on the Telegram Bot ([Acknowledge & Mute]) and a physical reset button on the ESP32 node. Once assistance is provided, pressing either trigger instantly silences the siren and resets the system to normal monitoring mode.

4. The Central Role of Axelera Metis (The Intelligence Hub) While wearable sensors provide data from camera-free zones like bathrooms, standalone IMU sensors are notorious for high false-alarm rates (e.g., hitting a table or dropping the watch).

Axelera Metis serves as the Master AI Verifier and Central Decision Engine. It handles the most computationally intensive tasks:

1. Continuous Real-Time Vision: Processing high-FPS video streams to analyze 3D pose vectors and body velocity at zero latency.
2. Multi-Modal Sensor Fusion: Cross-verifying signals from the ESP32 DevWatch with live vision analytics to validate genuine emergencies and eliminate false alarms.
3. Local Natural Language Processing: Running edge audio pipelines to interpret voice commands locally without cloud delays.

The ESP32 acts merely as a lightweight edge node, while Axelera Metis provides the heavy-duty processing power required to run the entire multi-stream safety pipeline locally.

5. How Voyager Wingman Will Be Used (Step-by-Step Iteration Journey) As a solo builder, I will rely on Voyager Wingman as my primary AI engineering collaborator to construct, optimize, and iterate the entire pipeline on Axelera Metis silicon in record time. Rather than generating a single monolithic script, I will engage Wingman through a structured, multi-step prompting journey:

• Step 1: Core Vision & Pose Estimation Pipeline"Voyager Wingman, generate a real-time keypoint pose estimation pipeline on Axelera Metis for an overhead camera stream. Track body joint coordinates and calculate velocity vectors to distinguish between normal lying/sitting positions, bed-egress maneuvers (sitting at the edge of the bed), and sudden floor fall events."
• Step 2: Multimodal Audio Integration (Local Voice Commands)"Extend the existing Metis vision pipeline to include a concurrent local audio processing stream. Implement a lightweight keyword spotting model to detect local voice triggers such as 'I want to go to the bathroom' or 'Help', running 100% locally with zero cloud latency."
• Step 3: Wearable Sensor Fusion & IoT Actuation (ESP32 Integration)"Integrate an MQTT client into the Python pipeline to interface with external ESP32 nodes. Create logic to cross-verify fall events by fusing vision metrics with incoming IMU/acceleration telemetry from an ESP32 DevWatch. When bed-egress or voice triggers are detected, publish MQTT commands to trigger ESP32 relays for smart nightlights and sirens."
• Step 4: Privacy Masking, Telegram Dispatcher & Alarm Reset Loop"Add a privacy-preserving rendering layer that converts raw camera frames into skeleton pose overlays or blurred streams. When a confirmed fall occurs, capture a privacy-masked snapshot and dispatch an emergency alert via Telegram Bot API with inline response buttons ([Acknowledge & Mute]). Handle incoming webhook/polling callbacks from Telegram and physical ESP32 reset buttons to silence sirens and reset the system state to normal monitoring mode."

6. Submission & Demonstration Plan Testing and video demonstration will be conducted in a dedicated home bedroom/bathroom setup. The final submission will include:

• The complete Voyager Wingman prompt journey and code repository.
• A live working video demo showing real-time pose tracking, local voice response, ESP32 DevWatch sensor fusion, smart light activation, privacy masking, Telegram instant alerts, and alarm reset execution.

7. About Me I have a background in the healthcare sector, giving me deep domain knowledge of patient safety protocols, fall risks, and emergency response workflows. Combining healthcare domain expertise with Axelera’s high-performance Edge AI hardware will demonstrate how accessible, secure, and life-saving senior care can be.