GazeBase multi-detector inferential-robustness protocol

Protocol ID: gazebase-r1s1-fxs-vs-tex-multidetector-v1
Status: frozen before inspection of GazeBase alternative-detector endpoint outputs
Frozen: 2026-09-11
Machine-readable protocol: docs/case_studies/gazebase_multidetector_protocol.json
Protocol fingerprint: 3f64122f62cbc9762b0bd0e0b95c7fef6ff40c4700215ee4b90f005af77003b1

The fingerprint is SHA-256 over the GazeAudit-style canonical JSON representation of the JSON file’s protocol object: keys sorted, compact separators, UTF-8, and no non-finite JSON numbers.

1. Purpose

This document freezes the first public real-data multi-detector inferential-robustness case study for GazeAudit before alternative detector outputs are inspected.

The scientific question is not which detector best reproduces a preferred result. It is:

Does the scientific contrast in fixation duration between a sustained fixation task and a reading task survive a predeclared set of widely used eye-movement detection algorithms?

The case study is intended as a high-robustness demonstration if, and only if, the predeclared recovery rule is met. The label is not assumed in advance and must not be rescued by removing detectors, changing thresholds, changing the endpoint, or changing the participant cohort after results are seen.

2. Dataset and external reference

The study uses GazeBase, described by Griffith, Lohr, Abdulin, and Komogortsev (2021), Scientific Data, 8, 184, DOI 10.1038/s41597-021-00959-y. The data repository is 10.6084/m9.figshare.12912257 and is distributed under CC BY 4.0.

GazeBase contains monocular left-eye recordings collected with an EyeLink 1000 at 1000 Hz. The published acquisition geometry is 1680 × 1050 pixels on a 47.4 × 29.7 cm display at a viewing distance of 55 cm. Calibration and validation were performed before each task.

The primary case study is restricted to:

• Round: 1
• Session: 1
• Task A: fixation task (FXS)
• Task B: reading task (TEX)

The GazeBase paper reports 322 participants completing the Round 1 battery. The analytic cohort is nevertheless defined by the explicit reference-only eligibility rules below rather than hard-coded as 322.

GazeBase distributes sample classifications generated by the EyeLink real-time parser for relevant tasks. Those labels classify fixation, saccade, and blink samples. The published dataset documentation states that labels are absent for VD1, VD2, and BLG because the EyeLink parser does not label smooth pursuit. FXS and TEX therefore permit an independent parser-based reference.

The EyeLink parser is treated as an external reference, not biological ground truth.

3. Fixed analytic cohort

The cohort is constructed before any alternative-detector endpoint is inspected. A participant enters the fixed cohort only when all of the following hold:

1. Round 1, Session 1 FXS and TEX recordings are both present.
2. Timestamps are finite and strictly increasing within each recording.
3. EyeLink parser labels are available for both recordings.
4. At least one EyeLink-parser fixation event can be recovered in each task.

Cohort inclusion must not depend on pEYES labels, pEYES effect estimates, statistical significance, or whether a participant makes a detector look more or less favorable.

Once this reference-defined cohort is frozen for an execution, alternative detectors are evaluated against that same denominator.

4. Frozen software environment

Publication execution uses these exact interoperability versions:

• pymovements == 0.28.0
• pEYES == 0.2.2
• GazeAudit >= 0.1.0.dev7, with the exact GazeAudit commit recorded in the execution manifest.

The pEYES methodological reference is Nir and Deouell (2026), “Systematic classification differences across eye movement detection algorithms,” Behavior Research Methods, 58, 109, DOI 10.3758/s13428-026-02983-5.

Any future rerun with different pEYES or pymovements versions is a new execution and must not silently replace the frozen publication run.

5. Coordinate and timing normalization

GazeBase positions are loaded through pymovements. Native visual-angle positions are converted to pixels using pymovements deg2pix and the published GazeBase screen geometry.

For the pEYES interface:

• viewer distance = 55.0 cm;
• pixel size = 47.4 / 1680 = 0.028214285714285713 cm;
• timestamps are supplied in milliseconds;
• no data-dependent rescaling is permitted.

This horizontal pixel pitch is predeclared because the current pEYES public detector contract accepts one scalar pixel-size value. The GazeBase horizontal and vertical pixel pitches are nearly equal; this choice must not be changed after detector outputs are inspected.

6. Predeclared detector specification space

All seven pEYES detector families in the frozen specification space are evaluated:

1. ivt
2. ivvt
3. idt
4. idvt
5. engbert
6. nh
7. remodnav

Shared detector arguments are:

• missing value: NaN;
• minimum event duration: 40 ms;
• blink padding: 0 ms.

Algorithm-specific parameters use the defaults resolved by pEYES 0.2.2. Before endpoint aggregation, the complete resolved parameter dictionary for every detector must be serialized into the execution manifest. Thus “defaults” cannot become an unrecorded moving target.

No detector may be dropped because its estimate is inconvenient. No thresholds may be tuned after the scientific endpoint is examined. A scientifically motivated alternative threshold analysis, if later desired, must be explicitly labelled a separate sensitivity analysis and cannot replace this primary specification space.

7. Missingness policy

Primary analysis preserves missing gaze samples as NaN. There is no interpolation.

For the EyeLink reference, blink, saccade, invalid, and unlabeled samples are not fixations. Alternative-detector participant-task summaries that cannot be computed are left missing and are not imputed.

This policy intentionally allows detector failure or incomplete classification to affect the completeness diagnostic rather than hiding it through preprocessing.

8. Primary scientific endpoint

The primary endpoint is the median within-participant fixation-duration contrast, in milliseconds, for FXS − TEX.

8.1 Fixation events

For each participant-task recording, a fixation event is a maximal contiguous run of samples labelled as fixation by the classifier being evaluated.

For a run with first timestamp t_first and last timestamp t_last, event duration is

(t_last - t_first) + median_positive_sample_interval_within_recording

This avoids assigning a one-sample event a duration of zero and does not assume that every recording is perfectly sampled at exactly 1 ms intervals.

8.2 Participant-task summary

For each participant and task:

median fixation-event duration (ms)

8.3 Participant contrast

For each participant with finite summaries for both tasks:

FXS median fixation duration - TEX median fixation duration

8.4 Study effect

For each detector/reference:

median of participant-level FXS - TEX contrasts

No p-value, confidence-interval crossing, or significance status is part of the endpoint-recovery definition.

9. External-reference effect

The reference effect is exactly the endpoint above computed from the distributed EyeLink real-time parser labels over the fixed reference-defined cohort.

The reference is not selected from among the pEYES outputs and must be computed before alternative-detector endpoint inspection.

Because the primary recovery criterion uses relative error, if the EyeLink reference effect is exactly zero, execution must stop before inspecting alternative-detector endpoints. A new versioned protocol would then be required with a scientifically justified zero-effect rule; the software must not invent a denominator or silently switch tolerances.

10. Detector completeness gate

Before conclusion recovery is classified, every predeclared detector must yield finite FXS and TEX participant-task summaries, and therefore a finite paired contrast, for at least 95% of the fixed reference-defined cohort.

The denominator is always the fixed reference cohort.

If any detector fails this 95% gate:

• the case study is labelled incomplete;
• no robust/fragile classification is issued;
• the detector is not silently removed;
• its failure and coverage remain reportable results.

This gate prevents apparent robustness from being manufactured by allowing difficult detectors or participants to disappear from the analysis.

11. Predeclared conclusion-recovery rule

The primary rule is:

ConclusionRule(
    relative_tolerance=0.20,
    absolute_tolerance=None,
    require_sign=True,
    minimum_recovery_fraction=6 / 7,
)

A detector recovers the reference conclusion only when:

1. its absolute relative error versus the EyeLink reference effect is ≤ 20%; and
2. its effect has the same direction as the reference effect.

After the completeness gate has passed, the case study is classified robust only if at least 6 of the 7 predeclared detectors recover the conclusion. Otherwise it is classified fragile within this declared detector specification space.

This is a methodological robustness classification, not a null-hypothesis significance decision.

12. Secondary descriptive outputs

The publication bundle may additionally report, without changing the primary rule:

• paired participant count for each detector;
• participant coverage fraction;
• reference and detector effect estimates;
• absolute and relative errors;
• sign recovery;
• task-specific fixation-event count distributions.

Any agreement metrics added later are secondary unless a new protocol version explicitly promotes them before results are inspected.

13. Execution and provenance requirements

The real-data runner must produce an immutable execution record containing at least:

• this protocol ID and protocol fingerprint;
• dataset DOI/version and selected round/session/tasks;
• exact GazeAudit commit;
• Python, numpy, pandas, pymovements, and pEYES versions;
• complete resolved parameters for all seven detectors;
• fixed cohort participant identifiers or a deterministic fingerprint thereof;
• source-data identity/checksum information where technically available;
• coordinate-conversion settings;
• missingness policy;
• reference endpoint and detector endpoint table;
• detector coverage table;
• completeness-gate result;
• conclusion-recovery table;
• final PublicationAuditBundle scientific and bundle fingerprints.

The execution code must verify that the machine-readable protocol fingerprint is the expected frozen value before detector execution.

14. Anti-HARKing / anti-optimization guardrails

The following actions are forbidden for the primary case study after alternative-detector results are inspected:

• replacing FXS or TEX with a task yielding a preferred contrast;
• changing Round 1 or Session 1 because another subset is more favorable;
• removing a detector because it fails recovery;
• changing detector thresholds to improve agreement;
• changing the 20% tolerance or 6/7 recovery requirement;
• changing the 95% completeness gate;
• redefining the fixation-duration endpoint;
• excluding participants based on alternative-detector behavior;
• introducing interpolation or imputation into the primary analysis;
• using p-values to decide which specifications count.

If a genuine methodological defect is discovered, the protocol must be superseded by a new version before rerunning the affected detector outcomes, and the reason for supersession must be documented. The original protocol remains in repository history.

15. Interpretation boundary

A positive result means only that this specific FXS-versus-TEX fixation-duration conclusion is robust to at least six of seven predeclared pEYES 0.2.2 detector families under this cohort, missingness policy, endpoint, and tolerance.

It does not establish that one detector is universally superior, that the EyeLink parser is ground truth, that all possible eye-movement algorithms would agree, or that detector choice is unimportant for other tasks or endpoints.

16. Planned separation from the fragile real-data demonstration

This protocol is the first real-data case study only. A later fragile-case protocol must be frozen independently before its alternative-detector outputs are examined. A candidate may involve a substantially smaller within-task/session contrast, but the fragile case must not be chosen retrospectively because observed results happened to disagree.

References

• Griffith, H., Lohr, D., Abdulin, E., & Komogortsev, O. (2021). GazeBase, a large-scale, multi-stimulus, longitudinal eye movement dataset. Scientific Data, 8, 184. DOI: 10.1038/s41597-021-00959-y.
• GazeBase data repository. DOI: 10.6084/m9.figshare.12912257. CC BY 4.0.
• Nir, J., & Deouell, L. Y. (2026). Systematic classification differences across eye movement detection algorithms. Behavior Research Methods, 58, 109. DOI: 10.3758/s13428-026-02983-5.