Watchkeeper: a real-time edge-AI lookout that gives an autonomous boat eyes
Semantic segmentation of water / sky / obstacle on Metis, with a danger-zone collision alarm. Fully offline, boat-grade power budget, and validated on real footage I'll shoot sailing a busy Mediterranean shipping lane next month.
On the water there are no lanes and no signs. The one question that matters for any autonomous surface vehicle (USV) and for a shorthanded human crew is: is the way ahead clear, and if not, how close is the danger?
Watchkeeper answers exactly that, on-device. A forward camera watches the sea; the system segments every frame into water / sky / obstacle, turns each obstacle into a tracked box, and raises a warning the moment something enters the boat's danger zone. Wether it a buoy, a vessel, a pier, floating debris, or the highest-value alert of all: a person in the water.
I'm a sailor. This is a problem I actually have, and next month I'll be logging hours on one of the busiest routes in the Mediterranean, which makes it both a real testbed and a personal one.
The base model is U-Net FCN, the semantic-segmentation network from the Voyager Model Zoo, fine-tuned on MaSTr1325, a maritime dataset labeled per-pixel for exactly these three classes. Semantic segmentation is the right paradigm here (every pixel is water, sky or obstacle), and the water/sky boundary gives a clean horizon line as a rough distance cue, so a ship on the horizon reads differently from a buoy 10m off the bow. The perception front-end is deliberately swappable: the danger-zone logic consumes either a segmentation mask or detection boxes, so if measured FPS or robustness on Metis favours a detector, we drop in a fine-tuned YOLO (YOLOv8 / YOLO11) instead of U-Net. We pick the model empirically on the hardware. Segmentation runs on the AIPU; tracking and the danger-zone state machine stay on CPU.
What you'll physically see
Two demos, escalating in realism:
- Live on public data: the Metis pipeline running on USV datasets (MaSTr1325 / MODD2), segmentation overlay + danger-zone alerts in real time.
- Real Mediterranean footage: recorded from the deck during my sailing next month: ships, ferries, other yachts, buoys, swimmers, in genuine sea, light and weather. Used both to fine-tune and as a real-world validation set.
A video of the working demo will be part of the final submission.
Metis & Edge Synergy for our Maritime Vision
A boat at sea has no reliable connectivity meaning cloud inference is off the table. Collision avoidance is latency-critical and must run fully offline inside a small vessel's power budget. Metis runs the segmentation model in real time on the AIPU while the CPU handles tracking and alerting. The zoo's U-Net FCN already benchmarks comfortably above real-time on the Metis PCIe card, so there's headroom for the post-processing.
Why Wingman
Not "help me code": concrete jobs: generate the segmentation pipeline from plain language, wire webcam / RTSP / file inputs, load my fine-tuned weights via the custom-weights path, compile + quantize U-Net for the AIPU and report the mIoU drop, flag unsupported ops & validate Metis vs PyTorch output (for the stretch), profile FPS/latency and iterate on the danger-zone logic.
Main goal: port a maritime-specialist model
Port eWaSR: a domain-specialised, embedded-ready maritime segmentation network (Apache-2.0, pretrained on MaSTr1325): as a custom model onto Metis, and benchmark it against the fine-tuned U-Net: generic backbone vs purpose-built maritime net, on accuracy and FPS. A genuine test of Wingman on a real port (ONNX export → quantization → unsupported-op detection → PyTorch-vs-Metis validation). Scoped as upside, not core: the project lands even if this slips.
One-month plan
- Week 1: U-Net semantic segmentation live on Metis over public footage, measured FPS (core)
- Week 2: obstacle → box + tracking + horizon estimate + danger-zone alarm (core)
- Week 3: fine-tune on MaSTr1325 · build the water-tray rig · shoot & fold in Mediterranean footage
- Week 4: polish · record demo video · (stretch) port + benchmark eWaSR
Weeks 1–2 are the committed deliverable; everything after is upside.
Where it goes next, further applications and extensions
Watchkeeper is the perception layer for a broader assistance stack. Natural extensions, none promised for the challenge window:
- Real-time collision alerting for solo / shorthanded crews: audible + visual, day or night.
- Night operation with a low-light or thermal camera, feeding the same perception into the boat's autopilot (NMEA 2000/0183) to trigger a hold-off or avoidance manoeuvre.
- Fusion with AIS: AIS only shows vessels that transmit; vision catches the ones that don't (small craft, fishing boats, containers, debris): exactly the dangerous gap.
- Coupling with weather/routing data (GRIB): higher-level assistance for power and sail-plan decisions.
Datasets, models & hardware
Datasets: MaSTr1325 (1325 pixel-labeled images), MODD2 (video test), MODS (benchmark), public USV research sets from the ViCoS lab, plus my own Mediterranean footage
Models: U-Net FCN base, eWaSR stretch
Hardware: Dell XE5 + 16 GB Metis PCIe, USB webcam, GoPro/phone for field capture
Why us
I work in computer vision and I'm comfortable across the whole pipeline this project needs: training and fine-tuning detection and segmentation models (YOLO, U-Net) in PyTorch, converting and quantising them for edge inference, and wiring up the surrounding video I/O and logic. I've taken vision prototypes from raw dataset to a working real-time demo before: which is exactly the shape of this challenge.
Just as important, I'm a sailor with real time on the water this month, on one of the Mediterranean's busiest routes. That gives me genuine field access to capture and validate on the exact data this system is built for, not just a desk webcam. This is a project I'd build and finish whether or not it wins: it scratches a real itch I have every time I'm on a boat at dusk.
— Hugo Crochet :))
⚓⚓⚓
THE PROMPT
Build me a real-time maritime obstacle-detection pipeline for an
autonomous boat, running on the Metis PCIe card.
GOAL
Segment every video frame into three classes: water, sky, obstacle: then
warn when an obstacle is close to the boat.
MODEL
Use the U-Net FCN semantic-segmentation model from the Voyager Model Zoo as
the base, at 512x512. I will fine-tune it on the MaSTr1325 maritime dataset,
which is labeled for exactly these three classes: help me load my fine-tuned
weights through the custom-weights path, then compile and quantize the model
for the Metis AIPU. Report the mIoU after quantization and any unsupported
operations or host fallbacks.
INPUT
Start from a video file, then switch to a USB webcam and to an RTSP stream
using the same pipeline. Handle the input color format the model expects.
PIPELINE
Run the segmentation on Metis. In Python post-processing: take the obstacle
mask, extract connected components as bounding boxes, track them across
frames, estimate the horizon from the water/sky boundary as a rough distance
cue, and define a "danger zone" in the lower part of the frame. When a tracked
obstacle enters the danger zone, draw it in red and trigger an on-screen
COLLISION WARNING.
OUTPUT
Show the live video with the segmentation overlay, the tracked obstacle boxes,
the danger-zone band, and measured FPS and latency. Target at least 25 FPS at
512x512; if it is lower, suggest a lighter configuration.
WORKING STYLE
Propose a short plan before any major change, and do not alter the model
architecture without asking me first. If U-Net does not hit the FPS or
accuracy target on Metis, propose a YOLO-based detector (YOLOv8 / YOLO11)
fine-tuned on maritime data as an alternative front-end, and compare the two.
Preserve all code, configuration, prompts and instructions needed to reproduce
the project.
