# RecoveryBench — Teaching Robots What to Do After They Fail
Most robot demos show the happy path: the object is where it should be, the grasp succeeds, and the task completes.
Real robots do not live on the happy path.
A gripper misses by a few millimetres. An object slips after contact. A box is slightly heavier than expected. An obstacle appears where the motion planner did not expect it. The arm reaches a poor approach angle and a simple retry repeats the same mistake.
The difficult question is not only:
> Can a robot detect that a task failed?
It is:
> Can it choose the most useful next recovery action?
That is the project I want to build with Voyager Wingman and Axelera Metis.
---
## The project
**RecoveryBench** is a ROS 2 and MuJoCo-based benchmark and closed-loop recovery system for robotic manipulation.
A simulated robot arm performs a simple pick-and-place task. When the task enters a failure state, a compact learned model running on Metis receives a short history of robot and task telemetry and selects the best next recovery primitive.
The system will choose from a deliberately bounded set of safe, interpretable actions:
1. **Retry from a new approach angle**
2. **Move to a re-observation pose**
3. **Adjust gripper width**
4. **Push the object into a more reachable position**
5. **Retreat and re-plan**
6. **Escalate to manual review**
The recovery model does not directly command motor torques and does not replace motion planning. It selects among pre-defined, simulator-validated recovery actions. ROS 2 control nodes execute the selected action, while a host-side safety gate rejects invalid transitions.
The core question is simple:
> Can Metis-accelerated recovery selection improve task completion rate and reduce repeated failures compared with fixed retry and rule-based baselines?
---
## Why simulation first?
I want this project to be reproducible, measurable, and honest about failure modes.
MuJoCo makes it possible to create many difficult robotic failure scenarios that would be slow, expensive, or unsafe to reproduce repeatedly on physical hardware:
- missed grasps;
- object slip after grasp;
- inaccurate object-pose estimates;
- random object mass and friction;
- changed gripper width;
- delayed actuator response;
- unexpected obstacles;
- partial occlusion or noisy state estimates;
- poor approach angles;
- collision risk during recovery.
The simulator also provides something extremely valuable for training: ground-truth contacts, object poses, task state, and outcome labels.
That means RecoveryBench can generate its own labeled dataset without manually annotating images or videos.
Each simulated episode becomes a ROS 2-recorded example containing telemetry, failure type, recovery decision, and final outcome. The resulting benchmark can be replayed deterministically to compare different recovery policies.
---
## How it works
```text
MuJoCo robot task
|
| ROS 2 topics:
| joint states, end-effector pose, gripper state,
| object-pose estimate, contact events, task progress
v
Telemetry window builder
|
v
Metis recovery-policy model
|
| predicted recovery action + confidence
v
Host safety gate and recovery manager
|
| validated ROS 2 recovery command
v
Motion planner / controller
|
v
MuJoCo robot executes recovery action
|
+--> success, repeated failure, timeout, or manual-review state
```
The learned model will consume a fixed-size temporal feature window, for example:
- joint position and velocity;
- end-effector pose and motion error;
- gripper width and closure state;
- estimated object pose relative to the gripper;
- contact or collision indicators;
- task phase;
- recent action history.
The initial model will be intentionally small and deployment-friendly: a compact MLP or temporal convolution model exported to ONNX. This keeps the first Metis deployment focused, measurable, and realistic.
---
## The role of Axelera Metis
Metis is the inference engine for the recovery-selection model.
The host CPU will run MuJoCo, ROS 2 nodes, rollout generation, the motion planner, safety checks, logging, and dashboard rendering. Metis will execute the learned inference workload that maps a telemetry history to a recovery recommendation.
This split is deliberate:
- **Metis:** fast, local inference for recovery classification/ranking;
- **Host:** simulation, deterministic safety policy, planning, ROS 2 orchestration, and reporting.
The project will measure:
- Metis inference latency;
- end-to-end ROS 2 decision latency;
- recovery success rate;
- overall task completion rate;
- number of repeated failed attempts;
- collision / invalid-action count;
- task completion time;
- performance under unseen object and dynamics conditions.
---
## The role of Voyager Wingman
Voyager Wingman will be my AI pipeline engineering partner throughout the build.
I will use it to:
- identify a suitable small temporal model architecture for Metis deployment;
- prepare the ONNX model and validate its fixed input/output contract;
- generate and refine the Voyager deployment pipeline;
- integrate Metis inference with a Python ROS 2 node;
- diagnose model-conversion, runtime, latency, memory, and structured-output issues;
- profile inference and end-to-end performance;
- iterate on model shape, quantization, and inference settings;
- document the full prompt journey, generated artifacts, corrections, and benchmark results.
The project is not about asking Wingman for one large script. It is about showing the complete engineering loop:
> simulation data → model training → Metis deployment → ROS 2 integration → closed-loop evaluation → optimization.
---
## Dataset and evaluation
RecoveryBench will generate a synthetic MuJoCo dataset from randomized pick-and-place rollouts.
The first MVP will use:
- **one robot arm and gripper;**
- **three object families;**
- **three primary failure modes;**
- **five executable recovery primitives;**
- **one pick-and-place task family.**
### Failure modes
The first version will focus on:
1. **Missed grasp** — the gripper closes without securely capturing the object.
2. **Object slip** — the object is initially grasped but becomes unstable or falls during transport.
3. **Blocked or poor approach** — the robot cannot safely complete the planned path because of an obstacle or poor approach geometry.
### Domain randomization
Simulation episodes will vary:
- object shape, size, mass, and friction;
- object placement and orientation;
- gripper alignment and width;
- observation noise;
- control delay;
- obstacle location;
- contact properties;
- initial arm configuration.
Labels are created automatically from simulation outcome and contact/task state:
- failure type;
- candidate recovery action;
- whether recovery succeeded;
- time cost;
- collision occurrence;
- final task outcome.
Training, validation, and held-out test episodes will use separate random seeds. The held-out evaluation will also include partially unseen object shapes and dynamics ranges, so the model is not judged only on near-duplicate scenarios.
---
## The experiment
RecoveryBench will compare three policies under the same seeded simulation scenarios:
### 1. Fixed retry baseline
The robot repeats the original attempt after failure.
### 2. Rule-based baseline
A transparent rule set maps obvious failures to a fixed recovery action.
### 3. Metis-informed recovery policy
The learned model chooses a recovery primitive from recent telemetry and task context.
The comparison will answer:
- Does the learned policy complete more tasks?
- Does it avoid repeating the same failed action?
- Does it reduce unnecessary retries?
- Does it choose safer recoveries when uncertainty is high?
- Can it do this with low enough inference latency for a closed ROS 2 control loop?
The final dashboard will show the live MuJoCo scene, current task state, detected failure, selected recovery action, model confidence, ROS 2 messages, inference latency, and cumulative benchmark results.
---
## Four-week plan
### Week 1 — Build the benchmark and data generator
- Set up the MuJoCo manipulation scene and ROS 2 interfaces.
- Implement the baseline pick-and-place controller.
- Define failure modes and recovery primitives.
- Record initial rollouts as ROS 2 bags.
- Build automatic labeling from MuJoCo contacts and task outcomes.
- Start the first synthetic dataset generation run.
**Deliverable:** reproducible MuJoCo + ROS 2 scenario with visible failures and fixed-retry baseline.
### Week 2 — Train and validate the recovery model
- Generate randomized training, validation, and held-out test rollouts.
- Train a compact recovery-selection model.
- Compare against fixed-retry and rule-based baselines offline.
- Export the selected model to ONNX.
- Define the model input/output contract and confidence thresholds.
**Deliverable:** trained model with offline benchmark results and reproducible dataset-generation scripts.
### Week 3 — Deploy on Metis with Voyager Wingman
- Use Voyager Wingman to deploy and profile the recovery model on Metis.
- Integrate inference into a ROS 2 recovery-policy node.
- Add the host safety gate and recovery-action executor.
- Run the first closed-loop Metis-powered simulations.
- Diagnose and optimize latency, model deployment, and ROS 2 integration.
**Deliverable:** live MuJoCo task where a Metis inference result changes the robot's recovery action.
### Week 4 — Evaluate, document, and publish
- Run held-out benchmark scenarios across multiple random seeds.
- Compare all three policies.
- Record live demonstrations of different failure-and-recovery cases.
- Publish source code, MuJoCo scenes, ROS 2 package, model artifacts where permitted, dataset generator, benchmark protocol, results, and full Wingman prompt journey.
**Deliverable:** a working, reproducible Metis + ROS 2 + MuJoCo recovery system with measured results.
---
## Final demo
The final demo will show a robot arm attempting the same task under several controlled failures:
- an object is slightly misaligned and the first grasp misses;
- an object slips after a seemingly successful grasp;
- a planned approach is blocked by an introduced obstacle.
For each case, viewers will see:
1. the baseline behavior;
2. the failure event;
3. the telemetry state entering the Metis model;
4. the selected recovery primitive and confidence;
5. the ROS 2 recovery command;
6. the resulting task outcome.
The core result will be easy to understand:
> Instead of blindly repeating a failed motion, the robot learns to choose a more appropriate next move.
---
## What I am not claiming
This is a simulation-first research and engineering prototype.
RecoveryBench will not claim that:
- it is a certified robot safety system;
- it can recover from every manipulation failure;
- simulation results automatically transfer to a real robot;
- the learned model directly controls robot motors;
- the first version supports arbitrary robots, tasks, or recovery strategies.
The prototype proves a narrower and more useful point:
> A compact learned recovery selector can be trained from automatically generated simulation data, deployed on Metis, integrated with ROS 2, and evaluated transparently in a closed-loop robotic task.
---
## Deliverables
The final project will include:
- a working ROS 2 + MuJoCo recovery demonstration;
- a Metis-deployed recovery-policy model;
- Voyager pipeline artifacts and deployment instructions;
- synthetic dataset-generation scripts;
- ROS 2 bag recordings for benchmark replay;
- fixed-retry and rule-based baseline implementations;
- held-out evaluation protocol and results;
- live demo video;
- source code repository;
- complete Voyager Wingman prompt history, iterations, failures, fixes, and performance observations.
---
# The Prompt
> **Wingman, help me build RecoveryBench: a ROS 2 and MuJoCo simulation-first robotic manipulation benchmark where an Axelera Metis accelerator runs a learned recovery-policy model.**
>
> The robot performs a pick-and-place task. When it misses a grasp, drops an object, encounters a blocked approach, or enters another defined task-failure state, the system must select one safe recovery primitive rather than blindly retrying the same motion.
>
> Start by asking me to confirm the robot model, MuJoCo version, ROS 2 distribution, and the exact observation topics available. Do not assume a camera pipeline is required; the first version uses fixed-size temporal telemetry features.
>
> Build this in stages:
>
> 1. **Model and interface design:** Recommend a small Metis-friendly model architecture for selecting one recovery action from a fixed-size temporal window of robot telemetry. Inputs may include joint positions and velocities, end-effector pose, gripper state, estimated object pose, contact indicators, task phase, and recent action history. Outputs must be a recovery-action class and calibrated confidence.
>
> 2. **Dataset contract:** Define an ONNX-friendly fixed input tensor shape, feature ordering, normalization requirements, class vocabulary, and structured output contract. The classes are: retry_new_approach, reobserve, adjust_gripper, push_to_reachable_pose, retreat_replan, and manual_review.
>
> 3. **Metis deployment:** Generate the simplest Voyager pipeline required to run this model on the Metis PCIe accelerator. Help me export, validate, quantize if appropriate, compile, and profile the model. Keep inference on Metis wherever supported.
>
> 4. **ROS 2 integration:** Generate a Python ROS 2 node design that subscribes to a fixed telemetry-window topic or synchronized telemetry topics, invokes the Metis pipeline, validates output, and publishes a recovery recommendation with confidence and timestamp.
>
> 5. **Safety boundary:** The Metis model must not directly command actuators. Generate a separate host-side validation layer that checks confidence, task state, and allowed state transitions before publishing the final recovery command.
>
> 6. **Performance and debugging:** Help me measure Metis inference latency, end-to-end ROS 2 decision latency, throughput, output validity, and failure behavior. When deployment or runtime errors occur, diagnose them step by step and preserve a reproducible record of every correction.
>
> 7. **Optimization:** Once the pipeline works, help me optimize model shape, quantization, runtime configuration, and ROS 2 integration while preserving output agreement with the host reference model.
>
> The final system must support a live closed-loop MuJoCo demonstration and a reproducible benchmark comparing fixed retry, rule-based recovery, and the Metis-informed recovery policy.
