Skip to contents

Validation Before Promotion: Recovery, SBC, PPC, and Transportability

Source: vignettes/advanced-model-validation-0-7.Rmd
advanced-model-validation-0-7.Rmd

A common validation contract

Every advanced model should answer the same questions before promotion:

  • can its parameters be recovered under its own generator?
  • how biased and noisy are the estimates?
  • do intervals cover at their nominal rate?
  • how often does estimation fail?
  • are parameters empirically identifiable?
  • is Bayesian computation calibrated?
  • can replicated data reproduce scientifically important features?
  • what happens under misspecification and preprocessing changes?
  • does measurement transport to held-out devices, sessions, sites, and items?
  • does the process channel add out-of-sample information beyond conventional response/RT information?
spec <- eyeprocess::irt_validation_spec(
  model_id = "joint_gaze_rt",
  replications = 500,
  parameters = c("ability", "speed", "engagement"),
  grouped_validation = c("device", "session", "site")
)
spec
#> eyeprocess IRT validation specification
#>   model: joint_gaze_rt 
#>   replications: 500 
#>   metrics: bias, rmse, coverage, interval_width, convergence 
#>   grouped validation: device, session, site

Recovery table

The canonical recovery table has one row per replicate/parameter and columns replicate, parameter, truth, estimate, and optionally lower, upper, converged, scenario, engine, and failure_type.

summary <- summarize_parameter_recovery(recovery)
audit_bias(summary, threshold = .10)
audit_rmse(summary, threshold = .30)
audit_coverage(summary, minimum = .90)
audit_interval_width(summary)
audit_convergence(recovery, minimum = .95)
audit_identifiability(recovery)
validation_mcse(recovery, "coverage")
recommended_validation_replications(target_mcse = .01, metric = "coverage")
plot(summary)

Prior SBC 

sbc <- run_sbc(
  simulator = function(r) simulate_one_dataset(r),
  fitter = function(dat) fit_bayesian_model(dat),
  posterior_draws = function(fit) as.matrix(fit$draws),
  replications = 250
)

audit_sbc(sbc)
plot(sbc, parameter = "ability_sd")

SBC is an inference-algorithm validation. It does not establish empirical model fit or construct validity.

Posterior SBC

Posterior SBC addresses a different question: whether inference is calibrated in the region relevant conditional on the observed data. Because the correct conditional self-consistency experiment is model-specific, eyeprocess requires an explicit callback rather than pretending ordinary SBC is posterior SBC.

contract <- posterior_sbc_contract(function(replicate, observed_data) {
  # Model-specific implementation following the posterior-SBC construction.
  # Must return the simulated truth and posterior draws from the corresponding
  # conditional self-consistency experiment.
  list(truth = truth, draws = draws)
})

psbc <- run_posterior_sbc(observed_data, contract, replications = 100)
audit_sbc(psbc)

Posterior predictive checks 

posterior_predictive_discrepancies(
  observed = observed_fixations,
  replicated = replicated_fixations,
  discrepancies = list(
    mean = mean,
    sd = sd,
    zero_rate = function(x) mean(x == 0),
    p95 = function(x) unname(quantile(x, .95))
  )
)

Choose discrepancies that could falsify the scientific use of the model: tail fixation counts, omission rate, transition entropy, pupil peak timing, response accuracy by item difficulty, and other substantively meaningful summaries.

Misspecification 

stress_test_latent_distribution(runner)
stress_test_local_dependence(runner)
stress_test_speededness(runner)
stress_test_missingness(runner)
stress_test_preprocessing(runner, variants = c(
  "default", "strict_validity", "alternate_fixation_detector", "alternate_pupil_filter"
))

The runner owns data generation and fitting; the framework records scenario, replicate, results, and classified failures.

External and grouped validation 

external_validate_irt(train, external, fitter, predictor, scorer)
leave_device_out_validation(data, "device", fitter, predictor, scorer)
leave_session_out_validation(data, "session", fitter, predictor, scorer)
leave_site_out_validation(data, "site", fitter, predictor, scorer)
leave_item_out_validation(data, "item_id", fitter, predictor, scorer)

Then quantify transportability rather than reporting only a pooled score:

audit_measurement_transportability(
  held_out_results,
  metric = "rmse",
  higher_is_better = FALSE,
  max_range = .15
)

Incremental information and negative controls

A process channel should survive a stricter test than in-sample significance.

inc <- audit_channel_incremental_information(
  data,
  fold = "participant_fold",
  baseline_fitter = fit_response_rt,
  process_fitter = fit_response_rt_gaze,
  predictor = predict_trait,
  scorer = trait_rmse,
  higher_is_better = FALSE
)
plot(inc)

negative_control_process_test(
  data,
  process_columns = c("fixation_count", "evidence_dwell"),
  within = c("person_id", "item_id"),
  evaluator = full_crossvalidated_score,
  permutations = 250,
  higher_is_better = TRUE
)

This guards against adding gaze merely because a high-dimensional channel can improve training fit.

Process-dependent discrimination 

pd <- process_dependent_discrimination_audit(
  data,
  response = "correct",
  theta = "theta",
  process = "rt_ms",
  person = "person_id",
  item = "item_id"
)
plot(pd)

The diagnostic asks whether effective response discrimination varies with the person-by-item process residual. It should be described as an association unless a design identifies a causal mechanism.

Evidence grade 

grade_model_evidence(
  recovery = recovery,
  spec = spec,
  external_validation = held_out_results,
  sbc = sbc,
  ppc = ppc,
  semantic_roundtrip = roundtrip
)

The grade is a governance summary of supplied evidence, not a substitute for construct validity or independent replication.