LabMemory

At the TU/e Electrical Engineering lab, we already have a camera watching the experimental setup. The idea is that if something goes wrong during an experiment, or if we want to understand what happened afterwards, we can look back at the footage.

In practice, almost nobody uses it.

The problem is not the camera. It is the amount of footage. An experiment can run for several hours, and if you want to find one specific event, you have to manually scroll through hours of video to find a few seconds that matter. The camera records everything, but it doesn't tell you what happened or where to look.

This is where the idea for LabMemory comes from.

The goal is to use an edge-AI system to turn the camera footage into a timeline of what happened during the experiment. Instead of having four hours of video, we want to have something like:

14:32:11 — Sample A placed on measurement stage
14:32:18 — Probe connected to Sample A
14:35:42 — Sample A removed from measurement stage
14:35:51 — Sample A returned to storage tray

The researcher can then go directly to the relevant event instead of searching through the entire recording.

What we want to build

For the hackathon, we want to focus on one specific experimental setup rather than trying to understand an entire laboratory.

The camera will observe a controlled experiment containing a small number of relevant objects and locations. For example, we might have a sample tray, a measurement stage, a probe and an instrument.

The system will detect these objects and track them as they move around the setup. From their positions over time, we can infer simple events such as a sample being moved from the tray to the measurement stage or being removed from the setup.

The important part is that we are not trying to build a general-purpose system that understands every possible human action. We want to show that relatively simple visual information can be turned into useful experimental information.

How it works

The camera provides a continuous video stream of the experiment.

The Axelera Metis PCIe card on the Dell XE5 will handle the computer-vision part of the system. It will detect the objects that are relevant to the experiment and provide their positions and confidence scores.

The host CPU will then take these detections and track the objects over time.

For example, if the system sees Sample A in the tray, then sees it moving, and then sees it consistently inside the measurement area, it can infer that Sample A was moved from the tray to the measurement area.

These events are then added to the experiment timeline.

The overall pipeline is therefore:

Camera → Metis → Object detection → Tracking → Event detection → Experiment timeline

Why Metis

The camera is intended to run continuously, so we need to process the video in real time rather than analysing it manually afterwards.

Using Metis for the vision pipeline gives us a way to process the footage locally at the edge. We don't need to continuously upload laboratory video to a cloud service just to determine what is happening.

It also makes the project a good fit for the hardware. The Metis card handles the computationally intensive visual perception, while the host handles the higher-level logic and experiment timeline.

The first experiment

The first version will be deliberately constrained.

We will choose one physical experiment and define the objects and areas that matter. We will then record several examples of the experiment and use those recordings to develop and test the system.

For example, the setup could contain:

• a sample tray
• one or more samples
• a measurement stage
• a probe
• an instrument

The system would initially be able to recognize events such as:

• a sample entering the workspace
• a sample leaving the tray
• a sample entering the measurement area
• a probe entering the measurement area
• a sample being removed
• a sample returning to the tray

This gives us a manageable problem while still demonstrating the larger idea.

Object detection

The first technical step is getting reliable object detection running on Metis.

We will train or adapt a model for the objects in our particular setup rather than trying to solve generic laboratory object detection.

The output should be available as structured data to the rest of the application. For example:

timestamp: 14:32:28

Sample A
confidence: 0.96
position: ...

Probe
confidence: 0.91
position: ...

The system should also be able to process recorded footage first, so that we can develop and evaluate it without having to run the physical experiment every time.

Once that works, we can move to the live camera.

Tracking and event detection

Object detection alone isn't enough. We need to know how objects change over time.

If Sample A is detected in the tray in one frame and in the measurement area a few seconds later, we need to know that it is the same sample.

The tracking layer will therefore assign persistent IDs to objects and keep track of their locations.

On top of that, we can define simple rules for events.

For example:

If Sample A was previously in the tray and is now in the measurement area, the system generates:

“Sample A moved from Tray to Measurement Area.”

This approach keeps the first version relatively simple and interpretable. We can focus on making a small number of events reliable rather than trying to recognize arbitrary actions.

The experiment timeline

The main output of the system is the timeline.

The interface will show the camera feed alongside the events that have been detected.

For example:

14:32 — Sample A detected in tray
14:32 — Sample A moved to measurement stage
14:33 — Probe connected
14:35 — Measurement completed
14:35 — Sample A removed

Each event should also be connected to the original video, so that clicking an event takes the researcher directly to that point in the recording.

This is what makes the system useful. We aren't trying to replace the video. We are making the video searchable.

Implementation plan

We will start by setting up the physical experiment and defining exactly which objects, locations and events we want to detect.

Then we will build the object-detection pipeline and get it running on Metis. At this stage we will measure detection accuracy, FPS and latency.

Once the detections are reliable, we will implement tracking and the event-detection logic on the host.

Finally, we will connect everything to a simple interface and run the complete system with the live camera.

The development flow will be:

Physical setup → Dataset → Object detection → Metis deployment → Tracking → Event detection → Timeline → Live demonstration

How we will evaluate it

There are a few different things we want to measure.

First, whether the system can reliably detect the objects in the setup.

Second, whether it can maintain the identity of objects as they move around.

Third, and most importantly, whether it can correctly reconstruct the sequence of events.

We will repeat the same experiment several times and compare the generated timelines with what actually happened.

We will also measure the technical performance of the system, including inference FPS, latency and stability during longer recordings.

The most practical test will be comparing the time required to find an event in a long recording manually versus using the generated timeline.

If finding something in several hours of footage takes minutes of manual searching, while LabMemory can take us directly to the relevant event, then the system is doing what we intended.

What success looks like

The minimum successful demonstration is quite simple:

Someone performs a predefined experiment in front of the camera.

Metis detects the relevant objects.

The system tracks them as they move around the setup.

The event detector recognizes what happened.

Those events appear automatically in a timeline.

Then we can ask something like:

“Where did Sample A enter the measurement stage?”

and immediately jump to the relevant point in the recording.

That would already demonstrate the core idea.

Where this could go

The hackathon version would only understand one controlled experimental setup, but the longer-term idea is much broader.

A lot of experimental information currently exists in separate places. There are measurement files, instrument logs, analysis scripts, notes and sometimes video recordings. The physical part of the experiment is often the least structured.

LabMemory could eventually provide another layer of information: a machine-generated record of what physically happened during the experiment.

That could make experiments easier to reproduce and debug, and could also make it much easier to understand why a particular measurement looks the way it does.

The interesting thing is that the first part of this system already exists: the camera is already watching.

We just aren't making much use of what it sees.

LabMemory is about turning that footage into something researchers can actually use.

​