The Recipe That Can See — A Visual Stovetop Copilot

Can we give an ordinary stovetop useful visual intelligence without replacing it, sending its video to the cloud, or turning the kitchen into a robotics lab?

I like using small Linux computers, cameras and ESP32 boards to make ordinary things around me a little more useful, and the kitchen feels full of opportunities for that kind of intelligence.

Cooking asks us to notice many things at once: the colour and texture of food, the sound of oil or boiling water, what has already gone into the pan, where a tool was put down, and whether something has been left on the heat for too long. It is easy to joke that someone has no talent for cooking, but often the difficult part is simply keeping track of everything while the kitchen gets hot, noisy and messy.

There are many ways a local system could help. A camera beneath the range hood has a natural view of the stovetop. Another angle could help with occlusion. A thermal camera could add temperature information; a microphone could distinguish frying from boiling; simple environmental sensors could watch for smoke or gas. These sensors do not need a robot arm or a computer sitting in the grease. They can stay in the kitchen while a local edge machine processes their streams elsewhere over the home network.

That is the larger direction I find exciting: give the existing kitchen better senses, keep the data local, and let people remain in control. But it also contains far more than one month of work. The first useful piece can be much simpler.

A recipe knows the order of its instructions, but it has no idea what is happening in the pan.

“Cook for three minutes” sounds precise, but three minutes on two different stoves can produce very different food. The recipe still moves on at the same time. The cook has to keep looking back and forth between the instructions and the pan, deciding whether “golden”, “reduced” or “done” has actually happened.

That is the idea behind The Recipe That Can See: a local visual copilot that watches an ordinary stovetop and lets the recipe follow the food, instead of asking the food to follow a timer.

What it does

A camera mounted beneath the range hood looks straight down at the cooking area. The video is processed locally on the Axelera Edge 130p. The system observes the active pan, recognises a small number of useful cooking states and passes them to a recipe engine running on the host.

If the eggs have just gone into the pan, the recipe waits. When they have visibly set, it can say, “Ready for the tomatoes.” If one stage takes longer because the heat is lower, it does not rush ahead just because a countdown expired. If everything is progressing normally, there is no reason for it to keep talking.

This is meant for people who can follow a recipe but find the constant checking, timing and context-switching tiring. It is not trying to replace cooking skill. It simply gives the recipe the one thing it has always been missing: a view of the stove.

It is also a retrofit. I do not want to build another expensive smart oven. The camera can be added to an existing range hood, while the computer stays away from heat and grease and receives the feed over the local network. No special cookware is required, and the kitchen video stays inside the home.

The first version

For the one-month build, I will start with one camera, one pan and one recipe chosen for clear visual changes: a simple tomato-and-egg dish. The prototype will follow a handful of checkpoints such as eggs entering the pan, eggs setting, tomatoes being added and the mixture reaching the next usable state.

The main demonstration will be the same recipe cooked twice. In one run I will use higher heat; in the other I will slow the process down. A normal timer would give both cooks the same instruction at the same moment. The visual recipe should wait for what is actually happening in each pan.

The live view will show the state currently seen by Metis, its confidence, the recipe step and the moment a prompt is emitted. I will also show the measured latency and throughput, but the result that matters is easy to understand without a benchmark table: did it speak at the right moment, and did it avoid speaking at the wrong one?

I would also like to connect an ESP32-S3 environmental sensor node so a basic kitchen warning can enter the same local event stream. That is a useful extension of the idea, but it will not be allowed to hold up the visual cooking demo or be presented as a replacement for a proper household alarm.

Metis and Wingman

The camera feed will run through a small vision pipeline on Metis. Depending on what works best with the real camera view, this may be a detector followed by a pan-state classifier, or a simpler fixed-view classifier. The host application will keep the recipe state, smooth uncertain frame-by-frame predictions and decide when a visual change is stable enough to become an instruction.

Voyager Wingman is how I want to get from that description to a working Axelera pipeline. I will use it to bring up the live camera, choose a suitable supported model, deploy my custom cooking-state weights, connect the inference results to the Linux application and tune the pipeline on the Edge 130p. The starting request is simple: help me turn “camera sees the pan” into a local stream of useful recipe events. The exact prompts will evolve with the data and hardware, and I will publish that process with the final project.

Edge inference matters here because this is continuous video from inside a home. Uploading it to a remote service just to decide whether the next ingredient should go into the pan makes little sense. Local processing keeps the camera private, continues working without an internet connection and gives the system a reliable response while cooking. Metis also leaves room to add another view or another perception model later without changing that local-first design.

How I will build it

I will first set up the camera view, record several complete cooking sessions and find the few state changes that remain recognisable across different lighting, utensils and cooking speeds. My existing Rock 5A and camera modules mean I can start that work before the challenge system arrives.

I will then train the smallest useful model and work with Wingman to run it on Metis. Once the live predictions are stable, I will connect them to the recipe logic and test the prompts during complete cooking runs.

Then I will add the ESP32-S3 sensor input if the main loop is working, refine the interface and repeat the fast-versus-slow comparison until the behaviour is clear on video.

Finally, I will record the demonstration and publish the code, Voyager configuration, model preparation, Wingman iterations and build notes needed to reproduce it.

The boundary is straightforward: the challenge version follows one known recipe from one known camera view. It will not judge arbitrary food, estimate internal temperature or operate the stove. It watches, keeps track and prompts the human cook.

Why me

I have a computer science background, and embedded systems are one of my main hobbies. One of my larger DIY builds was a six-layer RK3368 board laid out from the Rockchip reference design. I brought out a Raspberry Pi-compatible 40-pin GPIO header and added an AP6255 wireless module, temperature and humidity sensing, ambient light, barometric pressure, an IMU and a silicon microphone. The point was not only to make another Linux board, but to give it enough senses to understand something about the room around it.

On the system side, I built NAND A/B images for rollback and added a U-Boot recovery path that falls back to an NFS root over Gigabit Ethernet after a failed local boot. That made the board useful as a recoverable sandbox for agent experiments. Scheduled Linux services could use temperature, sound and light context for low-power wake-up and connect the agent to the smart-home network for local environmental control.

This kitchen project follows the same pattern: a local Linux system observes the real world through simple sensors, keeps its decisions on the local network and interacts through clear boundaries. I already have a Rock 5A, camera modules, several other Rockchip and ESP32 boards and different environmental sensors, so I can prototype the camera, networking and sensor sides immediately. Metis is the new part, and Wingman can help me turn the vision idea into a working Axelera pipeline.

What I want to show at the end of the month is not a futuristic automated kitchen. It is one ordinary pan, one ordinary recipe, and a small but meaningful change:

the recipe can finally look before it tells you what to do next.