Demos / track-lab

Evaluation protocol · experiment E5

Measuring a tracker when the truth is wrong

In the field nobody hands you clean labels. An operator marks a target when they notice it, seconds after it appeared, and never marks the one they missed. Every metric computed against those labels is biased. The question is not how to get perfect labels but which numbers still tell you, reliably, that tracker B is better than tracker A.

The experiment

Fix a tracker and a scenario. Sweep its true quality in six steps by moving the detection threshold. Evaluate every step twice: against the clean labels the simulator knows, and against labels corrupted the way real ones are. Then ask, for each metric, whether the ranking of the six steps under corrupted labels agrees with the ranking under clean labels: Spearman's rank correlation, one number per metric per corruption. A metric with ρ near 1 still orders trackers correctly; a metric near 0 has become noise; a negative one is actively misleading.

Four corruptions, each with a real-world reason:

  • drop 30 %: a third of the targets were never labelled at all. The operator missed them, or the logging did.
  • late by 2 s: every label starts two seconds after the target really appeared. The operator locks on once the target is obvious.
  • offset 10 m: labelled positions are shifted by 10 m of noise. Hand-drawn boxes, a mis-registered map.
  • sparse, every 5th scan: labels exist for one scan in five. Annotation at 2 Hz on a 10 Hz sensor.

What follows for a protocol

The measured answer is not the one intuition gives. The point-level metrics survive late, dropped and offset labels: MOTA, IDF1 and HOTA keep a perfect ranking under all three, because the corruption shifts every setting by a similar amount and the order is preserved. Sparse labels are the exception, and they break MOTA alone: with labels on one scan in five, every track row at an unlabelled scan counts as a false positive, so the settings that track best are punished most and the ranking reverses. IDF1 and HOTA, which match identities over whole tracks, hold.

Time to confirm is stable under every corruption. False tracks per hour is not, under any of them, and the reason is instructive: across this sweep the clean false-track rate barely moves, so its ranking is noise before any corruption is applied, and a correct track on a dropped target is counted as false. A track-level number is only as robust as the spread it is asked to resolve.

The protocol that comes out of this is plain. Report HOTA or IDF1, not MOTA, whenever labels are sparse. Report time to confirm as the operator-facing number. Report false tracks per hour only with its standard error and only where the sweep actually moves it. And never compare two trackers on labels of different completeness: a lead time measured against a late label is a lead time measured from the wrong zero.

Manufacturing early labels: reverse tracking

time operator lock target appears no label, weak signal track backwards from the lock: large to small is the easy direction every step leaves a label the operator never made label exists here late, but reliable
A late label is still a reliable anchor in time and position. Track the target backwards from it, from large and clear toward small and faint, and the trail becomes labels for exactly the regime no human annotates: the sub-threshold seconds before anyone noticed. Each recorded operation yields these automatically, and they are the training and evaluation set for early detection.

This turns the biggest weakness of field data, late labels, into a labelling machine. It needs three things: a detector for the operator's lock marker in the recording (a fixed-shape rectangle, an easy problem), a tracker run in reverse time (the same tracker; time is a parameter), and a stopping rule for when the backward track has lost the target (the same sequential test, run on the reversed sequence). The reference tracker on these pages already has the last two.

track-lab · tracklab.gt holds the corruptions and the rank-stability test; the tables here are experiment E5 from docs/data/e5.json.