Cascaded Pipeline

Hi FreezerJohn,

how do you access the results now? Can you show me parts of your Python code so that I understand better what your requirements are?

Hi Sascha,

sure! So from my understanding in the pipeline above the tracker tracks the results from the first stage, as configured with bbox_task_name: keypoint_detections

In code, I access the frame_result.meta which contains entries of type CocoBodyKeypointsMeta and tracking_meta as expected.

My problem is: as soon as there is more than one person in a frame, I do not know which tracker result belongs to which keypoint detection. There seems to be no link in the meta data and indices are random.
Both bounding boxes are quite similar of course, so with a IoU calculation I can link them - but that feels like an unnecessary extra step on my side.

#!/usr/bin/env python3
"""
Minimal script to demonstrate the tracker / keypoint correlation issue.
Uses public image: https://www.ultralytics.com/images/bus.jpg
"""

import os, sys, urllib.request
from pathlib import Path

if not os.environ.get('AXELERA_FRAMEWORK'):
    sys.exit("Please activate voyager-sdk venv")

import numpy as np
from axelera.app.stream import create_inference_stream

IMAGE_URL = "https://www.ultralytics.com/images/bus.jpg"
IMAGE_PATH = "/tmp/bus.jpg"

if not Path(IMAGE_PATH).exists():
    urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH)

stream = create_inference_stream(network="yolo11lpose-coco-tracker", sources=[IMAGE_PATH])

for frame_result in stream:
    meta = frame_result.meta
    tracker = meta['tracking']
    kp_det = meta['keypoint_detections']

    print("keypoint_detections.boxes (indexed 0,1,2...):")
    for i, box in enumerate(kp_det.boxes):
        print(f"  Detection[{i}]: {box}")

    print("\ntracker.boxes (keyed by track_id):")
    for track_id, box in tracker.boxes.items():
        print(f"  Track[{track_id}]: {box}")

    print("\nPROBLEM: Indices don't match, must use IoU to correlate:")
    for det_idx in range(len(kp_det.boxes)):
        for track_id, track_box in tracker.boxes.items():
            if np.allclose(track_box, kp_det.boxes[det_idx], atol=1):
                print(f"  Detection[{det_idx}] -> Track[{track_id}]")
                break
    break

stream.stop()

Output:

Init ByteTrack!
keypoint_detections.boxes (indexed 0,1,2...):
  Detection[0]: [221 407 345 860]
  Detection[1]: [668 394 809 882]
  Detection[2]: [ 49 398 247 904]
  Detection[3]: [  0 549  78 873]

tracker.boxes (keyed by track_id):
  Track[4]: [  0 549  78 873]
  Track[3]: [ 49 398 247 904]
  Track[2]: [668 394 809 882]
  Track[1]: [221 407 345 860]

PROBLEM: Indices don't match, must use IoU to correlate:
  Detection[0] -> Track[1]
  Detection[1] -> Track[2]
  Detection[2] -> Track[3]
  Detection[3] -> Track[4]