FloorSight — a perception brain for the open-source robot vacuum [The Prompt submission]

The problem, personally.

My Roomba is old. Years of service have worn it down, and it still fails the same two ways it always has. First: it eats things. Charging cables, phone chargers, shoelaces — anything below the height its sensors can see just disappears under it until something jams. Second, and worse: there is one room in my house it must never disturb — our prayer room — and it barges in anyway, bumps into things, knocks items around, and gets itself stuck. App-drawn no-go rectangles don't cut it, because the robot doesn't actually see the room; it just trusts a map that drifts.

This isn't just my Roomba. The robot vacuum market ships over 12 million units a year, is worth more than $12B, and is still accelerating — IDC counted global shipments up 18.7% in the first three quarters of 2025 alone. Yet the dominant sensing stack — bump sensors plus a 2D LiDAR turret — is structurally blind below ~10cm. Over 70% of models launched since 2020 pack multiple navigation sensors (LiDAR, infrared), and none of them can tell a cable from a rug tassel, because navigation sensors map walls, not floor clutter. Cables, socks, LEGO, pet waste, spills: all invisible.

The industry knows this. Pet waste collisions were common enough that iRobot launched an official "P.O.O.P." guarantee (Pet Owner Official Promise) with the camera-equipped Roomba j7, promising a replacement robot for any poop-smearing incident. And camera-based AI obstacle avoidance works — flagship models like the Roborock S8 MaxV Ultra and Roomba j7+ hit ~98% avoidance rates on deliberately placed socks, cables, and pet toys in reviewer testing. But that capability lives exclusively in $1,000+ flagships running closed, vendor-locked perception stacks. Budget vacuums get bump sensors; the open-source community gets nothing at all.

The open-source OOMWOO project (4.9k stars, the community effort to build a fully open ROS2 robot vacuum) says it plainly in their design research: "never gets stuck" needs camera + AI sensor fusion, not LiDAR alone — LiDAR is blind below its turret. Their near-field obstacle avoidance module is sitting open, marked ready to start, waiting for someone to build it. What's missing is an open, local, affordable perception stack — which is exactly what a Metis card is for.

I want to be that someone, with a Metis card doing the seeing.

The idea.

FloorSight is a vacuum's-eye-view perception module: a low-mounted camera feeding a cascaded Metis pipeline that watches the floor ahead and emits simple, actionable events any vacuum — my future OOMWOO build, a ROS2-bridged commercial unit, or the person pushing the demo rig — can act on:

• hazard_entangle — cables, cords, socks, shoelaces → stop / avoid
• hazard_smear — pet waste, liquid spills → hard avoid, alert human
• hazard_ingest — LEGO, coins, small toys → avoid, log location
• keep_out_zone — a room the robot has been shown once and must never enter → retreat
• surface_change — carpet / hardwood / tile boundary → mode switch
• drop_off — stairs, ledges → stop
• all_clear — the default. Silence means clean floor ahead.

Detection (INT8 YOLO, fine-tuned) + floor-type semantic segmentation + monocular depth for distance-to-hazard, all on the Metis PCIe card, with the host doing only event logic and overlay rendering.

Keep-out zones that actually see.

The feature my Roomba made me want: instead of drawing rectangles in an app, you show FloorSight a room once — a handful of photos of its entrance and interior. An embedding model running on Metis matches the live view against those references, and the moment the robot faces that threshold, keep_out_zone fires and it retreats. No magnetic strips, no map drift, no re-localization failures. It works for a prayer room, a nursery, a workshop floor covered in screws — any space that deserves to be left alone.

The twist: a dataset that doesn't exist, generated synthetically.

There is no public "things robot vacuums die on" dataset. Nobody publishes ten thousand labeled photos of tangled cables at 10cm camera height. So FloorSight generates one: BlenderProc (open-source, free) procedurally renders domain-randomized floor scenes — random floor textures, lighting, camera height and tilt matching the real rig, 3D hazard assets scattered with physics. Perfect labels for free, thousands of images overnight, zero cloud dependency. Fine-tune YOLO on synthetic + a small hand-labeled real set, quantize to INT8, and measure the sim-to-real gap honestly: synthetic-only vs mixed accuracy is a headline number of the final write-up.

If the transfer works, that's the real result: you can bootstrap an edge perception product for a niche with no dataset, in a month, with one camera and a Wingman.

How it works (the pipeline Wingman builds).

A camera mounted ~10cm off the ground on a rolling rig (vacuum height, vacuum tilt) feeds the XE5:

1. INT8 YOLO detection over ~10 hazard classes (Metis)
2. Semantic segmentation of floor surfaces (Metis)
3. Monocular depth for distance / time-to-contact (Metis)
4. Embedding extraction for place recognition (Metis), matched on the host against enrolled keep-out references
5. Host-side event engine (my code, not Wingman's): fuses everything into the event stream above, journals events with frame snapshots, renders a live overlay with FPS and latency straight off the Metis card

The events are published as ROS2 messages, built against OOMWOO's documented software interfaces — so the module is upstream-ready from day one.

The one-month MVP, live on Metis.

Not "it would work if." A live demo where the rig rolls through a real, deliberately booby-trapped course — cables, socks, LEGO, a (fake!) pet-waste stand-in, a rug transition, a stair ledge — calling out every hazard with class, distance, and the fired event in real time. Then the finale: the rig approaches the enrolled keep-out room, keep_out_zone fires at the threshold, and the event log shows the retreat command a robot would obey. FPS and latency from the Metis card on-screen the whole run.

Week-by-week.

• Week 1 — Wingman generates the base pipeline (pretrained YOLO + segmentation) on the XE5; live camera in, detections out in real time. Build the floor rig. Start BlenderProc scene generation in parallel.
• Week 2 — Synthetic dataset v1 (~10k images, 9 classes), fine-tune YOLO, quantize, deploy through Wingman. First sim-to-real numbers.
• Week 3 — Depth + floor segmentation integration, keep-out zone enrollment and matching, event engine, ROS2 message publishing, overlay UI. Collect and label a small real-world eval set.
• Week 4 — The booby-trapped-course demo run, ablation numbers (synthetic-only vs mixed training), video, repo cleanup, full Wingman prompt journey write-up, and an upstream PR to OOMWOO's obstacle-avoidance contribution module.

Deliverables.

Open-source GitHub repo (pipeline configs, BlenderProc generation scripts, dataset recipe, event engine, ROS2 interface), the full Wingman prompt and every iteration, the synthetic dataset generator so anyone can reproduce it, demo video of the live run, the OOMWOO upstream PR, and a write-up of what Wingman generated vs what I hand-built — including the honest sim-to-real numbers, good or bad.

What I'm not claiming.

I'm not shipping a complete robot vacuum in a month — OOMWOO's hardware is still in development, and FloorSight is deliberately the perception layer, not SLAM, path planning, or motor control. The demo runs on a camera rig at vacuum height, with the ROS2 interface proving a real robot could consume the events the day one exists. The hazard list is a fixed set of ~10 classes, stated plainly. Keep-out recognition is few-shot on rooms I enroll, not general scene understanding. And synthetic data may not fully close the gap on every class — measuring exactly where it does and doesn't is part of the point, and I'll publish those numbers either way.

Why this project.

Honestly? My Roomba is lonely. It's been patrolling our floors solo for years, it's worn down, and it deserves a friend — ideally one that's open-source, cloud-free, and doesn't eat cables for breakfast. The open vacuum community is building exactly that robot, and they've already marked out the one module it's missing: the eyes. FloorSight is me building those eyes — for the community's robot, for my own future OOMWOO build, and for the old Roomba that's earned a capable companion. I'm a high school student teaching myself embedded systems, with no lab and no GPU cluster — just a camera, free tools, a Wingman, and a month. Which is kind of the point.

One more thing. I'm under 18 — a high school student — which the challenge guidelines say is no problem, just something to flag. Consider it flagged! My parents are on board, and we're happy to handle whatever extra paperwork is needed.

＄ＴＨＥ ＰＲＯＭＰＴ＄

Alright Wingman, here's the mission. We're building FloorSight: a vacuum's-eye-view hazard and keep-out perception system running live on Metis, publishing ROS2 events an open-source robot vacuum can consume. Build it in stages, walk me through each one, and ask me before assuming anything about the camera or classes.

1. Base pipeline. Generate a Voyager pipeline for a USB camera mounted ~10cm above the floor: pretrained INT8 YOLO object detection plus semantic segmentation, running on the Metis PCIe card, with a live annotated preview showing FPS and per-stage latency from the card itself.

2. Custom classes. Swap in my fine-tuned YOLO weights covering 9 floor-hazard classes (cables/cords, socks/textiles, LEGO/small toys, coins/jewelry, pet waste, liquid spills, shoes, bags/straps, rug tassels). Handle the INT8 quantization and validate accuracy against my eval set before deploying.

3. Floor segmentation. Add a segmentation stage classifying the ground plane into carpet / hardwood / tile / rug, and emit a surface_change event whenever the dominant class ahead of the rig changes.

4. Depth. Add monocular depth estimation and fuse it with detections so every hazard gets a distance estimate; flag anything inside 1 meter as imminent, and fire drop_off when the floor plane ahead shows a sudden depth discontinuity (stairs, ledges).

5. Embedding stage for keep-out zones. Add a feature-extraction/embedding model as a final pipeline stage on Metis, so every frame also outputs an embedding vector alongside detections. My host code will match those embeddings against reference photos of rooms I enroll — your job is the pipeline stage and helping me pick a zoo model that produces good scene embeddings at real-time rates.

6. Tune it. Once the cascade runs end-to-end, help me optimize: compiler configurations, stage ordering, resolution trade-offs — whatever gets the full pipeline to a frame rate a moving robot could act on, and show me where the milliseconds go.

The event engine (fusing pipeline output into hazard_*, keep_out_zone, surface_change, drop_off events published as ROS2 messages), the enrollment workflow, and the demo overlay are my code, not yours — but I'll lean on you for the SDK side of that boundary: how to consume the pipeline's inference stream from my own application, and debugging when the integration fights back.

Iterate with me on accuracy per class — I'll tell you which ones the synthetic data handled and which need more real examples.