Frozen validation programme · evidence map
996102e0
Evidence map
GazeAudit’s real-data validation programme is deliberately designed to preserve different scientific outcomes rather than reward one preferred answer. Each case was governed by frozen scientific choices, source identity, checksummed artifacts, archive-before-reveal execution, and a post-outcome interpretation boundary.
incomplete, robust_negative, and materially_fragile describe what happened under three different frozen audits. They do not rank the source datasets, methods, or research teams.
See the three frozen outcomes
Five detector specifications reached 322/322 finite participant estimates; two reached 0/322 under the frozen no-interpolation policy.
Open the frozen case →
The hard and uncertainty-propagated effects differed in magnitude, while all 2,000 prespecified measurement-model draws remained below zero.
Open the frozen case →
The sign remained negative at every target rate, but magnitude recovery failed at 125, 100, and 50 Hz under the frozen ±20% tolerance.
Open the frozen case →Compare the audits before comparing the labels
The three statuses come from different scientific questions and different frozen decision rules. The useful comparison is therefore what was held fixed, what was deliberately varied, and what rule converted the resulting evidence into a status.
| Case | Audit axis | Frozen target | Declared variation | Frozen decision rule | Frozen status |
|---|---|---|---|---|---|
| GazeBase | Specification-space completeness | Finite participant estimates for a fixed cohort of 322 participants | Seven predeclared detector specifications under a NaN-preserving, no-interpolation policy | The declared detector space had to satisfy a 95% finite-participant completeness gate before a robustness classification was permitted | incomplete |
| Korthals | AOI measurement uncertainty | Jumping-circle occupancy minus moving-circle occupancy within a 1-dva target-centred AOI | Grouped isotropic Gaussian gaze-error propagation with 2,000 prespecified Monte Carlo draws | The hard effect had to be negative and at least 95% of measurement-error draws had to remain below zero | robust_negative |
| Pedrotti/de Chambrier | Sampling and missingness sensitivity | Participant-balanced long-minus-short numeric gaze-path-rate difference in px/s | Prespecified downsampling plus MCAR and single-block added-missingness families | Recovery required sign agreement and no more than 20% relative magnitude deviation; material fragility was triggered when a frozen perturbation family reached the predeclared ceiling | materially_fragile |
What each status establishes—and what it does not
GazeBase
Establishes: this exact seven-detector audit did not satisfy its predeclared completeness prerequisite, so the remaining computable specifications were not promoted into a robustness label.
Does not establish: that GazeBase is a poor dataset, that the five computable detectors are invalid, or that a differently frozen future protocol could not be executed.
Korthals
Establishes: the negative direction of the frozen paired AOI estimand remained stable under the prespecified measurement-error propagation model.
Does not establish: that the measurement-model interval is a population confidence interval, Bayesian posterior interval, causal-effect interval, or population p-value.
Pedrotti/de Chambrier
Establishes: several frozen sampling or missingness perturbation families failed the declared magnitude-and-direction recovery criterion.
Does not establish: a sign reversal. Every sampling-rate estimate remained negative; the fragility concerns magnitude recovery under the frozen criterion.
Choose a case by methodological question
These routes are organised by the uncertainty you want to inspect, not by the desirability of the observed outcome.
- Can a declared multiverse be called complete when some specifications fail to compute? Start with GazeBase.
- Does an AOI contrast retain its direction after explicitly propagating gaze-position error? Start with Korthals.
- How far can a rate-like endpoint move under lower sampling resolution or controlled missingness while keeping the same sign? Start with Pedrotti/de Chambrier.
Read the programme in three layers
Define the scientific question before outcome inspection.
Fix the estimand, cohort, perturbation space, decision rule, and source identity before the final result is known.
Vary the declared methodological dimension.
Detector choice, measurement-error propagation, sampling resolution, and missingness are different audit axes and should not be collapsed into one generic robustness score.
Apply the frozen rule literally.
Stop when prerequisites fail, retain stable results when the rule supports them, and preserve fragility when the declared criterion is not recovered.
Why three different outcomes matter
A robustness framework is scientifically useful only if it can stop, preserve a stable conclusion, or expose fragility. These cases therefore exercise three distinct states:
- Incomplete — the declared specification space is not complete enough to support a robustness classification.
- Robust — the prespecified scientific direction remains stable under the declared uncertainty model.
- Materially fragile — the endpoint fails the declared magnitude-and-direction recovery criterion under defensible perturbations.
The case-study pages are explanatory views over the frozen authoritative records. They do not replace those records and do not change scientific choices after outcome inspection.