From a passing research run to a justified claim¶
A research result needs several distinct checks. Confusing them is especially costly for functional gaze inference.
| Question | A passing test can show | It cannot by itself show |
|---|---|---|
| Numerical execution | Code completed under a specific finite fixture | Statistical accuracy for new experiments |
| Reproduction | A separate implementation recreated the archived original source outputs | Independent new-world sampling or real-data validity |
| Identifiability | The observed law plus explicitly justified assumptions pins down a parameter or set | That unobserved selection assumptions are true |
| Frequentist calibration | A sufficiently powered simulation at a stated DGP estimates sampling error/coverage | Uniform validity across untested DGPs |
| Product qualification | A frozen distribution and public API satisfy designated contracts | Release permission when science blockers remain open |
Three practical distinctions¶
Whole-curve inclusion is not pointwise inclusion. A functional band must cover the entire generating effect when claiming simultaneous confidence. Coverage interpretation.
Randomized invitation is not completed response. Known audit assignment probabilities do not identify unseen outcomes when return is selective. R93, R94, R95.
A zero-width assumption-conditional set is not observational proof of the assumption. R95 equality of the two completion probabilities can point-identify a binary mean if equality actually holds, but returned outcomes alone do not verify equality. Identification explainer.
Evidence labels on this site¶
We use stable for published package features, candidate for unpublished source interfaces, research-only for experimental analyses, and not qualified for failed or insufficient statistical inference contracts. Archived graphs preserve their original exact source denominators and limitations.
Browse research evidence · Published release status · Reproduction notes