GazeAudit uncertainty × robustness
v0.2.0 · development docs GitHub

Workspace · Robustness interpretation

Development docs · 996102e0
View source

Robustness Diagnostics & Sensitivity Interpretation Center

Use this center after the endpoint, specification space, validity rules, and execution ledger are fixed.

Do not interpret robustness diagnostics before execution completeness is reconstructable. A specification curve or sensitivity table summarises represented endpoint values. It does not erase unresolved valid branches, non-finite endpoints, or technical failures.

The six diagnostic families below are generated from one governed catalog. They keep three questions separate:

  1. What does the output literally describe?
  2. What scientific interpretation can it support?
  3. What stronger claim does it not support?

Showing all 6 diagnostic families.

prerequisite execution ledger + specification declaration

Execution completeness

Question Is the declared valid denominator fully represented before robustness summaries are interpreted?

Key fields
declared count; valid count; successful count; failure/not-run/non-finite states
Reporting rule
Report successful / valid first and preserve every unresolved valid branch state.

Can support

Whether robustness diagnostics have a complete valid execution denominator.

Does not support

Whether the observed endpoint is scientifically important or whether a missing branch would have changed the conclusion.

Open practical route →
ordered branch table specification_curve()

Specification curve

Question How does the common endpoint vary across successfully represented valid specifications?

Key fields
spec_id; declared factor columns; estimate
Reporting rule
Describe the full represented range and branch denominator before discussing factor patterns.

Can support

The observed ordering, range, sign changes, clusters, and branch metadata across represented estimates.

Does not support

Which specification is scientifically best, whether an estimate is statistically significant, or whether missing branches can be ignored.

Open practical route → API pathway → Plot example →
descriptive summary effect_stability()

Effect stability summary

Question How dispersed are the represented endpoint estimates and how often do they fall above, below, or exactly at the declared null?

Key fields
n_specifications; median_estimate; mean_estimate; min_estimate; max_estimate; q025; q975; positive_fraction; negative_fraction; exact_null_fraction; sign_stability
Reporting rule
Call q025/q975 empirical specification quantiles and sign fractions descriptive branch proportions.

Can support

Descriptive magnitude spread and sign concentration across the represented specification estimates.

Does not support

Confidence intervals, posterior probabilities, statistical significance, or universal robustness.

Open practical route → API pathway → Plot example →
descriptive factor table marginal_sensitivity()

Marginal factor sensitivity

Question Which declared factor levels align most strongly with observed endpoint variation?

Key fields
factor; n_levels; marginal_eta2; level_mean_range
Reporting rule
Describe factors as associated with or aligned with variation, not as causing a percentage of uncertainty.

Can support

Which declared factor has the largest descriptive between-level alignment with endpoint variation.

Does not support

Causal attribution, unique variance decomposition, or a set of shares that must sum to one.

Open practical route → Plot example →
descriptive pair table pairwise_interaction_sensitivity()

Pairwise non-additive sensitivity

Question Which declared factor pairs show the strongest descriptive departure from an additive specification pattern?

Key fields
factor_a; factor_b; n_cells; interaction_ratio; max_abs_interaction
Reporting rule
Use non-additive descriptive pattern language and report imbalance/incompleteness where relevant.

Can support

Which factor pair shows the largest descriptive non-additive pattern in the represented branch means.

Does not support

An inferential interaction effect, p-value, causal interaction, or factorial-model substitute.

Open practical route →
controlled perturbation curve spatial_sensitivity_curve() / sampling_sensitivity_curve() / missingness_sensitivity_curve()

Controlled one-dimensional sensitivity

Question How does one common endpoint change as one declared perturbation dimension is varied deliberately?

Key fields
perturbation value; endpoint/diagnostic value; preserved endpoint identity
Reporting rule
State the perturbation values and distinguish controlled representation sensitivity from specification-space choice.

Can support

The endpoint trajectory across the explicitly declared perturbation values.

Does not support

What a different physical tracker would have measured, causal effects of missingness, or general robustness outside the perturbation range.

Open practical route → API pathway → Plot example →

Interpretation order

Read diagnostics in this order.

1. Execution completeness

Before any estimate summary, reconstruct:

  • declared combinations;
  • valid combinations;
  • successful branches;
  • valid technical failures;
  • non-finite endpoints;
  • branches not run.

If the valid denominator is not fully represented, the available diagnostic pattern is incomplete.

Use the Execution Ledger & Recovery Center.

2. Specification curve

Use the curve to inspect:

  • range;
  • sign changes;
  • clusters;
  • gaps;
  • factor combinations attached to extreme estimates.

Do not use ordered position as a quality ranking.

The leftmost and rightmost specifications are simply the lowest and highest represented endpoint estimates.

3. Effect stability

effect_stability() reports:

  • count;
  • mean;
  • median;
  • minimum;
  • maximum;
  • empirical 2.5% and 97.5% specification quantiles;
  • positive, negative, and exact-null fractions;
  • sign stability.

These are descriptive summaries of the represented specifications.

The empirical quantiles are not confidence intervals or credible intervals.

4. Marginal sensitivity

marginal_sensitivity() compares endpoint means across levels of each declared factor.

Its marginal_eta2 is a descriptive between-level sum-of-squares ratio.

It can help locate where variation aligns with declared choices.

It is not:

  • causal variance attribution;
  • a probability;
  • an inferential effect size;
  • guaranteed to sum to one across factors.

5. Pairwise sensitivity

pairwise_interaction_sensitivity() compares represented cell means with an additive factor-level expectation.

Use it to identify non-additive specification patterns.

Do not call interaction_ratio a statistical interaction test.

If a formal interaction is scientifically required, fit an appropriate model under its own assumptions.

6. Controlled sensitivity curves

Sampling, missingness, and spatial-error sensitivity curves differ conceptually from discrete specification spaces.

They ask:

What happens as one declared perturbation dimension changes?

Do not interpret downsampling as a physical simulation of another eye tracker, or added missingness as a causal missing-data experiment unless the design independently supports that claim.

Finite endpoint prerequisite

The robustness summary functions use a common finite-estimate validation path.

If the selected endpoint column:

  • is missing;
  • contains no rows;
  • contains NaN;
  • contains infinity;

the summary should fail rather than silently reinterpret the unavailable value.

That is desirable.

Resolve the execution/end-point contract before asking for robustness summaries.

Read the plot as a visual summary, not new evidence

The Plot gallery contains deterministic code-generated examples.

Two especially relevant plots are:

The plots help visualise an existing analysis table. They do not create a new inferential result.

A four-question diagnostic reading

For every plot or table, ask:

  1. Denominator: Which valid represented branches produced this output?
  2. Quantity: Which exact endpoint field is being summarised?
  3. Pattern: What is literally visible or numerically described?
  4. Boundary: Which stronger interpretation remains unsupported?

If one of those answers is missing, do not compress the result into a label such as “robust”.

Reporting examples

Methods

Robustness was evaluated across the declared valid specification space using a single common endpoint. Execution completeness was reconciled before descriptive specification summaries were calculated. We inspected the ordered specification curve, effect-stability summary, marginal factor sensitivity, and pairwise non-additive sensitivity. These diagnostics were treated descriptively rather than as confidence intervals, posterior probabilities, causal decompositions, or inferential interaction tests.

Results

[S] of [V] valid specifications successfully yielded finite endpoint estimates. Across those represented branches, estimates ranged from [min] to [max], with median [median] and empirical 2.5%–97.5% specification quantiles of [q025] to [q975]. [Positive/negative/exact-null pattern]. Descriptive marginal sensitivity was largest for [factor], while the strongest pairwise non-additive pattern involved [factor A × factor B].

Limitation

These diagnostics characterise the declared and successfully represented analytical space. They do not quantify uncertainty from untested analytical or measurement choices, and the specification quantiles/sign fractions are not sampling-theory or posterior uncertainty statements.

When not to use a robustness diagnostic

Do not interpret these summaries when:

  • branches estimate different endpoints;
  • the valid denominator cannot be reconstructed;
  • non-finite endpoints were silently converted to zero;
  • failed valid branches were deleted;
  • specification factors were chosen after inspecting which values produced attractive outcomes;
  • the scientific question actually concerns sampling uncertainty rather than across-specification variation.

Resolve the upstream contract first.

marginal_sensitivity() and pairwise_interaction_sensitivity() are documented here and in the worked examples, but this center does not invent stable API deep links for symbols absent from the governed source-level API inventory.

Continue from here

Documentation search

Find the right documentation

↑↓ navigateEnter openEsc close