Workspace · Conclusion rule

Development docs · 996102e0

View source

Conclusion Rule Design Center

Use this center only when a categorical recovery decision is scientifically justified.

This builder does not choose a reference effect or threshold for you. A conclusion rule is a researcher-owned scientific decision. The reference effect and tolerances must be justified independently of the robustness estimates being judged. The UI therefore supplies no scientific defaults and requires explicit choices even where the Python dataclass itself has defaults.

A conclusion rule is most straightforward when:

the data-generating effect is known in a simulation or benchmark;
an independently justified external/reference effect exists;
the decision criterion was fixed before robustness-result inspection.

For ordinary real-data analyses without an independently justified reference effect, descriptive robustness reporting is usually the safer route.

rule_name Rule name Required Stable project label for the declared conclusion/recovery rule. Boundary A name identifies the rule but does not justify its scientific thresholds. reference_type Reference-effect type Required State whether the reference is known truth or an independently justified real-data reference. Boundary A reference derived from the same robustness estimates is circular and must not be used to validate those estimates. reference_effect Reference effect Required Finite scientific reference value against which branch-level recovery is evaluated. Boundary GazeAudit does not infer this value from the robustness results. reference_rationale Reference rationale Required Record the independent source and interpretation of the reference effect. Boundary Provenance cannot rescue a circular or scientifically inappropriate reference. relative_tolerance Relative tolerance Optional Optional maximum relative error from the reference effect. Boundary Relative tolerance cannot be evaluated when the reference effect is zero. absolute_tolerance Absolute tolerance Optional Optional maximum absolute error in the endpoint's declared unit. Boundary At least one of absolute or relative tolerance is required; when both are present, both must be satisfied. tolerance_rationale Tolerance rationale Required Explain why the declared error tolerance is scientifically acceptable. Boundary Tolerances chosen after seeing the branch estimates are outcome-informed decision rules. require_sign Require sign recovery Required Explicitly declare whether recovered branches must have the same direction as the non-zero reference effect. Boundary Sign recovery does not establish statistical significance or substantive importance. minimum_recovery_fraction Minimum recovery fraction Required Fraction of represented specifications that must satisfy the branch-level recovery rule for the summary classification. Boundary The fraction must be justified before result inspection and applies only to the supplied represented specification set. decision_timing Decision timing Required Record when the reference and rule were fixed relative to robustness-result inspection. Boundary Post-result or reviewer-requested rules must remain temporally visible rather than being relabelled as prespecified. interpretation_boundary Interpretation boundary Required State what a recovered or fragile classification does not establish. Boundary The summary classification is conditional on the reference, rule, endpoint, and represented specification space.

No conclusion rule generated yet.

Documentation-side declaration

This JSON preserves the scientific rule and timing. It is not a runtime object and does not validate the reference or thresholds.

{
  "schema": "gazeaudit-conclusion-rule-declaration-v1"
}

Python rule skeleton

The generated call supplies every scientific argument explicitly; it never relies on the dataclass's default sign or recovery-fraction values.

# Complete the conclusion rule declaration first.

What the runtime requires

ConclusionRule requires at least one of:

relative_tolerance;
absolute_tolerance.

If both are supplied, both must be satisfied for tolerance recovery.

The tolerances must be finite and non-negative.

minimum_recovery_fraction must be in [0, 1].

Zero reference effects need special handling

When the reference effect is exactly zero:

relative error is undefined;
effect ratio is undefined;
sign recovery is treated as recovered;
a relative tolerance cannot be evaluated;
use an absolute tolerance instead.

Do not work around this by adding an arbitrary epsilon to the reference value.

Sign recovery is a separate criterion

For a non-zero reference effect and require_sign=True:

the branch must satisfy the declared tolerance;
the branch must also have the same sign as the reference.

With require_sign=False, only the tolerance rule governs branch recovery.

This is a scientific rule choice, not a significance test.

Minimum recovery fraction applies to the supplied represented set

summarize_conclusion_recovery() computes the fraction of represented rows with conclusion_recovered=True.

It classifies the supplied set:

robust when recovery fraction is at or above the declared minimum;
fragile when below it.

That classification is conditional on:

the supplied specification table;
the reference effect;
the tolerance rule;
the sign rule;
the declared minimum recovery fraction.

It is not a universal property of the dataset.

Avoid circular references

Bad pattern:

We used the median robustness estimate as the reference effect, then evaluated whether the robustness estimates recovered that reference.

The same evidence is defining and validating the target.

Better sources include:

known simulation truth;
independently generated calibration/benchmark truth;
a preregistered external reference;
another independently justified target whose provenance predates the recovery analysis.

Conclusion rules are not p-value rules

ConclusionRule deliberately evaluates effect recovery using:

absolute error;
relative error;
sign recovery;
across-specification recovery fraction.

It does not use:

p-values;
confidence-interval crossing;
specification selection;
posterior probability.

Do not translate robust into “statistically significant” or fragile into “not significant”.

Reporting example

Methods

Before robustness-result inspection, we defined the recovery target as [reference effect and provenance]. A specification-level endpoint was considered recovered when [absolute/relative tolerance rule] and [sign rule] were satisfied. The overall conclusion rule required at least [fraction] of represented specifications to recover the target. The reference and thresholds were fixed [timing].

Results

[R] of [N] represented specifications satisfied the predeclared recovery rule (recovery fraction = [x]). This [met/did not meet] the declared minimum recovery fraction of [threshold], yielding the rule-based classification [robust/fragile] for the supplied specification set.

Limitation

The classification is conditional on the independently defined reference effect, tolerance rule, sign criterion, represented specification space, and execution completeness. It does not establish statistical significance, external validity, or robustness to unrepresented analytical choices.

API links

conclusion_recovery_table() is used in the package workflow, but this page does not invent a stable API deep link for a symbol absent from the governed source-level inventory.

Continue from here

Design conclusion-recovery rules — scientific rationale, zero-reference behavior, dual tolerance, timing, reporting, and misuse prevention.
Conclusion-rule edge cases — deterministic synthetic recovery cases.
Publication audit guide — bind the final rule and evidence into deterministic publication artifacts.
Robustness Diagnostics Center — use descriptive reporting when a categorical reference-based rule is not justified.