RelateAnything runs faster than real time on Axelera Europa, ported and verified by Axelera Wingman
 

RelateAnything, the new open model that names what is happening in a video (person, riding, horse), now runs faster than real time on an Axelera Europa machine. Axelera Wingman, Axelera AI's agentic AI platform for developing applications on Axelera hardware, built the pipeline from a request typed in Hindi, moved the heavy models onto Europa in int8, and verified every one of them against an fp32 reference. The finished video comes out at up to 47.9 frames per second end to end, 1.6 times real time, and a clip shot in 4K runs faster than real time too.

What did Europa deliver?

• Faster than real time on all five scenes: 36.4 to 47.9 frames per second end to end, from opening the file to writing the labelled 1080p video, against source video of about 30 frames per second.
• int8 on Europa matches the fp32 reference: box IoU 0.959, mask IoU 0.934, backbone token cosine 0.899, and the same top relation on 37 of 40 test frames. Every stage passed tolerances that were fixed before the test ran.
• Well ahead of the host CPU: measured on the same machine in the same verification run, ResNet50 ran about 84 times faster on Europa (6.0 ms against 503.5 ms), the segmentation detector 2.6 times faster (22.0 ms against 57.1 ms), and the DINOv3 backbone at about 2.8 times the CPU's throughput (about 36 ms per frame against 101.5 ms).
• The heavy lifting is on Europa: the DINOv3 vision backbone, about 90 percent of RelateAnything's compute, runs there alongside detection, segmentation and a recognition check.
• Steady labels: on the horse-riding clip, relations that used to switch off and on 62 times in the first 300 frames now hold without a single flicker.

The film of the build: {{youtube:film-en}} (Hindi version: {{youtube:film-hi}}).

What does RelateAnything do?

Object detection tells you that a person and a horse are in the frame. RelateAnything tells you that the person is riding the horse. It predicts scored subject, relation, object triplets between regions of an image, and you give it the relation words at run time. A small text encoder reads them once, so nothing is retrained and no language model runs on every frame. That is what makes it practical on an edge accelerator.

The model has 53M parameters. It comes from Maëlic Neau (arXiv 2609.12552, 11 September 2026), and the code is at github.com/Maelic/RelateAnything under AGPL-3.0, with a commercial licence available from the author. In the paper's own GPU test, RelateAnything with YOLO-World runs 7.8 times faster than OvSGTR, the closest earlier open-vocabulary relation model.

How did Axelera Wingman build it?

Everything happened in one Build chat on an Axelera Europa machine with the Voyager SDK 1.9 pre-release, on 10 October 2026. We typed three prompts: the request in Hindi, a reminder to keep replying in Hindi, and one round of polish. Wingman ran the 825 commands: compiling the models for Europa, writing and tuning the pipeline, checking each stage against the CPU and rendering five public clips from Wikimedia Commons (horse riding, a blacksmith, mountain biking, jalebi being made in Bengaluru, and a guitarist filmed in 4K). Hindi was this demo's choice; Wingman takes the request in any language. This is the English translation of what we sent:

Run github.com/Maelic/RelateAnything on this machine's Axelera Europa at production quality. Do the new work in ~/relateanything-v2. The project's five videos are in ~/Videos/relateanything-reel/.

1. Recognition: use the project's own latest method (sections 7 and 8 of deploy/README.md), with light segmentation masks along with the boxes, as in the project's own reel. Every object must be named correctly, the guitar included.
2. Stability: add object tracking and relation smoothing (as in render_video.py) so boxes and relations do not blink.
3. Europa: run every model stage on Europa as far as possible, and check each stage's output against a CPU reference. For any stage that does not run correctly on Europa, report the measured reason.
4. Rendering: clean and sharp at 1920×1080, with thick boxes, light translucent masks, readable Hindi names and relations, and the average frames per second in a corner.
5. Speed: optimise for real time, measured end to end over the whole video.
6. Join the five results into one reel and show the reel, the videos and the speed table in a results viewer.

Wingman does not stop at a first draft. In Build, a second engine reviews the work and scores it out of 10, and Wingman keeps working while the score is below 8.5, for up to three automatic rounds. On this build the review sent the work back three times (scores 2, 7.5, 7.3, then 8), and Wingman kept improving it each time. The polish prompt then asked for smoother mask edges, masks on every added box and two naming fixes.

What runs where?

How was it verified?

Wingman compared Europa against a CPU reference on 40 evenly spaced frames, 8 from each video, with the tolerances written down before the comparison ran.

How fast is it?

Speed is measured end to end over the whole video, from opening the file to closing the encoder: decode, detection, the backbone, the relations, drawing and encoding. Relations are computed on every second frame, with Kalman tracking in between, as the paper's own video renderer does.

These are the English-label runs used in the English film. Wingman's original Hindi-label build ran the same pipeline while other jobs kept the machine busy (load average around 20) and still came out faster than real time on every scene, at 31.0 to 41.6 frames per second.

A tuning tip for multi-model pipelines on Europa

The biggest speed-up came from connections. With all six backbone instances sharing one Europa connection, they ran one after another at 463 ms per frame. Giving each instance its own connection brought that to 93.8 ms per frame. A process can open up to 8 Europa connections, so Wingman split them six for the backbone, one for YOLO and one for ResNet50. If you run several models in one pipeline, plan your connections the same way.

How do the labels stay steady?

The pipeline tracks boxes with a Kalman tracker and smooths each relation on calibrated scores, with separate on and off thresholds (0.45 and 0.35). In the first 300 frames of each clip, the number of times a relation switched off and back on fell from 62 to 0 on horse riding, from 2 to 0 on the blacksmith, from 115 to 12 on the jalebi clip and from 273 to 50 on the guitarist. On horse riding, a relation now stays on for 72.2 frames on average, against 5.8 before.

Try it

Relation prediction is useful wherever you need to know what is happening in a scene: sports coaching, workshop safety, retail, robotics, video search and accessibility. On Europa the video can stay on site. Axelera Wingman combines AI software development with knowledge of Axelera's hardware, compiler and Voyager SDK, so you can describe what you want in your own language and get a working, verified application on your Axelera hardware: chat.wingman.axelera.ai. Questions about the build are welcome in this thread.

Engineering notes

• The relation head (10.6 ms per frame, about 10 percent of the model's compute) runs on the host CPU. Its pairs are sampled dynamically, and the current compiler stops at the ModularizeLayerNormFunction pass.
• A 601-class Open Images v7 helper detector renames and adds boxes from a CPU background thread (28 ms). Its int8 Europa build needs more representative calibration data and returned no detections on these clips.
• On a single connection the backbone takes 161.3 ms per call. The throughput above comes from six connections in parallel.
• Of 82 reference boxes, 59 matched. The other 23 were low-confidence boxes (0.36 to 0.48 in fp32) that fell just under the threshold in int8. Three of the 40 relation checks differed, all on scores close to the threshold, and the smoothing keeps them out of the finished video.
• Class coverage: COCO has no guitar class, so a ResNet50 check on Europa confirms guitars and labels them (person, playing, guitar at 0.7 to 0.8). Jalebi is named from ImageNet's nearest class. No detector used here has classes for a blacksmith's tools, and the person is missed for about 100 frames when they turn away into the smoke.

I work on the Axelera Wingman team.