Benchmark evidence taxonomy

GazeForge separates three questions that are often conflated in eye-tracking validation:

1. Who produced the reference labels?
2. Was the analysed sampling rate native or derived?
3. What strength of validation claim can the reference support?

BenchmarkDatasetCard records these as annotation_origin, sampling_origin, and reference_strength. The package rejects internally inconsistent combinations, such as declaring vendor-algorithm labels to be an expert-human reference.

Annotation origin

Supported values are:

• expert-manual: expert human annotation;
• human-manual: human annotation where expert status is not required for the claim;
• human-assisted: human annotation created with algorithmic assistance and human correction;
• vendor-algorithm: labels generated by tracker/vendor software;
• research-algorithm: labels generated by a research algorithm;
• derived: labels derived from another reference;
• synthetic: simulated labels;
• mixed or unknown where a single origin cannot be defended.

Sampling origin

• native: the evaluated rate is the acquisition rate;
• resampled: the evaluated data were resampled from another acquisition rate;
• mixed: multiple origins are combined;
• synthetic: simulated data;
• unknown: origin cannot be established.

Native sampling alone does not make labels ground truth. A native 60 Hz recording with vendor-generated fixations is an algorithm-concordance benchmark, not human validation.

Reference strength

• expert-human-reference: expert-corrected/manual human reference at the analysed rate;
• human-reference: human annotation without a stronger expert claim;
• derived-human-reference: a lower-rate or otherwise transformed view of human labels;
• algorithmic-concordance: comparison against vendor/research algorithm outputs;
• synthetic-smoke-only: implementation/smoke testing only;
• unknown: no stronger interpretation is justified.

Current external catalog

Lund2013

The public Lund2013 benchmark contains paired expert MN/RA sample labels. Native-rate analyses are human-reference validation; GazeForge's 500-to-60-Hz tranche is explicitly marked derived-human-reference because the 60 Hz labels are created from expert 500 Hz annotations using auditable label-purity windows.

VISUS

VISUS is catalogued as a candidate native-60-Hz human dynamic-AOI benchmark: 25 participants, 11 video stimuli, and two human AOI annotators. GazeForge does not redistribute the historical raw benchmark because a current authoritative distribution location and reuse terms still need to be verified.

Hollywood2 manual events

The Agtzidis–Startsev–Dorr annotations provide expert-corrected fixation, saccade, smooth-pursuit, and noise labels for 16 observers at 500 Hz. This is a strong cross-dataset human reference, but it does not close the native-low-rate event-validation gap.

Reporting rule

A benchmark report should describe the strongest claim supported by its dataset card, not the claim a researcher would prefer to make. In particular:

• derived 60 Hz human labels are not equivalent to native 60 Hz human labels;
• vendor event labels are not expert ground truth;
• synthetic labels remain smoke-test evidence;
• human–human disagreement should be reported alongside model–human performance whenever paired annotations are available.