Skip to contents

gp3ml

gp3ml is a governance-first R package for leakage-resistant and group-aware predictive modelling and validation using Gazepoint-derived research data.

The package is designed for explicitly observed, non-sensitive outcomes and declared scientific purposes. It is not a generic automatic machine- learning wrapper and does not perform autonomous model selection.

Installation

Until a public CRAN package page exists, install the current source from GitHub:

install.packages("remotes")
remotes::install_github("stefanosbalaskas/gp3ml")

Scope

The package provides:

  • task, purpose, outcome, predictor, and identifier-role governance;
  • feature-provenance manifests and leakage auditing;
  • deterministic group-aware holdout splitting;
  • repeated grouped resampling with materialized folds;
  • fold-balance, coverage, exclusion, and outcome diagnostics;
  • fold-local preprocessing contracts;
  • governed statistical and optional machine-learning engines;
  • discrimination, error, calibration, and uncertainty assessment;
  • external-validation reports;
  • model cards and reproducibility reports.

Supported predictive-generalization targets are:

  • new trials among known participants;
  • new participants;
  • new stimuli;
  • simultaneous new-participant and new-stimulus generalization.

Scientific safeguards

Participant overlap is treated as a failure when the declared target requires generalization to new participants. Stimulus overlap is treated as a failure when the declared target requires generalization to unseen stimuli. Preprocessing must be estimated inside the relevant analysis partition or resampling fold. Package examples and tests use deterministic synthetic data. ## Leakage-audit workflow

audit_gazepoint_ml_leakage() audits already-defined analysis and assessment partitions before predictive evaluation. It checks:

  • compatibility with the declared generalization target;
  • participant, participant-trial, and stimulus overlap;
  • outcome and declared identifiers included as predictors;
  • declared target-derived and post-outcome predictors;
  • exact row overlap and repeated predictor profiles;
  • duplicated rows and missing grouping identifiers; and
  • identifier-like predictor names requiring manual review.

The returned audit has an overall pass, review, or fail status, a complete check table, a non-passing issue table, and partition summaries. Audit tables can be exported using write_gazepoint_ml_leakage_audit_csv().

analysis <- data.frame(
  participant_id = c("P01", "P02"),
  trial_id = c("T01", "T02"),
  stimulus_id = c("S01", "S02"),
  outcome = c(0, 1),
  fixation_duration = c(210, 245)
)

assessment <- data.frame(
  participant_id = c("P03", "P04"),
  trial_id = c("T03", "T04"),
  stimulus_id = c("S03", "S04"),
  outcome = c(1, 0),
  fixation_duration = c(275, 230)
)

audit <- audit_gazepoint_ml_leakage(
  analysis = analysis,
  assessment = assessment,
  outcome = "outcome",
  predictors = "fixation_duration",
  participant_id = "participant_id",
  trial_id = "trial_id",
  stimulus_id = "stimulus_id",
  generalization_target = "new_participants"
)

audit

The audit identifies structural risks visible in the supplied partitions and declared variable roles. It does not establish that preprocessing or feature selection was estimated within resampling folds; those safeguards require separate provenance and resampling infrastructure.

Feature-provenance workflow

create_gazepoint_feature_manifest() records the declared origin, transformation, availability, role, and preprocessing scope of each intended predictor.

validate_gazepoint_feature_manifest() checks whether the manifest contains sufficient provenance metadata and identifies predictors declared as outcome-derived, post-outcome, unavailable at prediction time, identifiers, or incompatible with required fold-local preprocessing.

manifest <- create_gazepoint_feature_manifest(
  features = c("fixation_duration", "pupil_change"),
  scientific_source = c(
    "Gazepoint fixation export",
    "Gazepoint all-gaze export"
  ),
  source_table = c("fixations", "all_gaze"),
  transformation = c(
    "Trial-level mean",
    "Baseline-adjusted change"
  ),
  availability_stage = "during_exposure",
  prediction_time_available = TRUE,
  preprocessing_scope = c("none", "resampling_fold"),
  fold_local_required = c(FALSE, TRUE)
)

validation <- validate_gazepoint_feature_manifest(manifest)
validation

Validation returns an overall pass, review, or fail status, together with complete check and issue tables. The manifest or its validation tables can be exported using write_gazepoint_feature_manifest_csv().

The manifest records declared provenance. It does not independently verify that preprocessing was executed within the stated partition or resampling scope.

Group-aware holdout splitting

split_gazepoint_ml_data() creates deterministic analysis and assessment partitions that preserve the grouping unit implied by an explicit predictive-generalization target. A passing feature-provenance manifest is required before splitting, and the resulting partitions are checked using the leakage audit.

split_data <- expand.grid(
  participant_id = sprintf("P%02d", 1:8),
  trial_id = sprintf("T%02d", 1:4),
  KEEP.OUT.ATTRS = FALSE,
  stringsAsFactors = FALSE
)

split_data$outcome <- as.integer(
  seq_len(nrow(split_data)) %% 2L
)

split_data$fixation_duration <-
  200 + seq_len(nrow(split_data))

split_data$pupil_change <-
  seq_len(nrow(split_data)) / 100

split <- split_gazepoint_ml_data(
  data = split_data,
  outcome = "outcome",
  predictors = c(
    "fixation_duration",
    "pupil_change"
  ),
  feature_manifest = manifest,
  generalization_target = "new_participants",
  participant_id = "participant_id",
  trial_id = "trial_id",
  assessment_prop = 0.25,
  seed = 17
)

split
split$validation

Supported generalization targets are:

  • new_trials_known_participants;
  • new_participants;
  • new_stimuli; and
  • new_participants_and_new_stimuli.

For simultaneous participant and stimulus generalization, rows belonging to only one held-out dimension are placed in the excluded partition. This preserves strict separation between the analysis and assessment participant–stimulus blocks while retaining complete source-row accounting.

validate_gazepoint_ml_split() checks partition structure, source-row accounting, provenance status, and the embedded leakage audit. Split assignments, summaries, validation checks, and materialized partitions can be exported using write_gazepoint_ml_split_csv().

The splitter creates one deterministic holdout split. It does not perform preprocessing, automated feature selection, cross-validation, nested resampling, or model fitting.

Group-aware resampling

create_gazepoint_group_folds() creates deterministic repeated grouped V-fold plans for the same explicit generalization targets supported by the holdout splitter. A passing feature-provenance manifest is required, and every analysis-assessment pair is checked using the leakage audit.

folds <- create_gazepoint_group_folds(
  data = split_data,
  outcome = "outcome",
  predictors = c(
    "fixation_duration",
    "pupil_change"
  ),
  feature_manifest = manifest,
  generalization_target = "new_participants",
  participant_id = "participant_id",
  trial_id = "trial_id",
  v = 4,
  repeats = 2,
  seed = 17
)

folds
folds$validation
folds$audit

Supported targets are new trials among known participants, new participants, new stimuli, and simultaneous new-participant and new-stimulus generalization. For simultaneous participant and stimulus generalization, v may contain separate participant and stimulus fold counts. Crossed assessment blocks preserve strict separation in both dimensions, while cross-block rows are explicitly assigned to the excluded partition.

validate_gazepoint_group_folds() checks fold counts, source-row accounting, assessment coverage, materialized partitions, feature provenance, and embedded leakage audits. audit_gazepoint_group_folds() aggregates the fold-level audits. Assignments, summaries, validation tables, audit tables, and optionally materialized partitions can be exported using write_gazepoint_group_folds_csv().

The fold planner does not perform preprocessing, automated feature selection, tuning, nested resampling, or model fitting.

Resampling diagnostics

diagnose_gazepoint_group_folds() evaluates the balance, coverage, outcome representation, and exclusions of an existing grouped resampling plan. Diagnostics remain downstream of fold construction and do not modify assignments or fit models.

fold_diagnostics <- diagnose_gazepoint_group_folds(folds)

fold_diagnostics
fold_diagnostics$repeat_metrics
head(fold_diagnostics$outcome_balance)
head(fold_diagnostics$assessment_coverage)

validate_gazepoint_fold_diagnostics() reports pass, review, or fail findings for fold counts, partition accounting, non-empty partitions, assessment coverage, fold-size imbalance, group presence, outcome-level presence, embedded leakage audits, and source-plan status.

For crossed new-participant and new-stimulus designs, exclusion_summary reports the rows intentionally excluded because they belong to only one held dimension rather than the strict crossed assessment block.

All diagnostic tables can be exported for review and reporting:

write_gazepoint_fold_diagnostics_csv(
  fold_diagnostics,
  directory = "resampling-diagnostics",
  prefix = "gazepoint_folds"
)

Diagnostics do not perform preprocessing, automated feature selection, hyperparameter tuning, nested resampling, or model fitting.

Governed modelling and reporting

The governed modelling core provides related interfaces for:

  • declaring tasks and validating intended uses;
  • validating outcome, predictor, and identifier roles;
  • fitting and applying fold-local preprocessors;
  • fitting reviewable model engines and integrating externally fitted black-box models under explicit governance;
  • calculating classification and regression performance metrics;
  • assessing and applying calibration;
  • producing explicitly labelled bootstrap metric uncertainty;
  • evaluating genuinely external datasets;
  • generating model cards and reproducibility reports.

These interfaces do not convert row-level metrics into participant-level estimates merely because a split was participant-grouped. Version 0.2.0 adds governed tuning and nested grouped resampling while retaining explicit human-reviewed selection. Autonomous winner selection remains outside the package’s scope.

Prohibited uses

The package does not support person identification, health inference, protected-attribute prediction, or direct inference of emotion, stress, mental state, cognition, comprehension, personality, deception, or intent.

See:

Citation

Use the Zenodo concept DOI to cite all versions of gp3ml. For the archived 0.2.0 release, use the version-specific DOI.

Release status

Version 0.2.0 is the second formal source release of gp3ml.

It extends the governance-first 0.1.0 foundation with repository-aware evaluation across materialized grouped folds, governed model comparison and tuning, nested grouped resampling, target-aligned uncertainty, external validation and transportability reporting, release model cards, and nine deterministic synthetic workflow articles.

Model selection remains explicitly reviewable and human-governed; the package does not perform autonomous winner selection or relax its prohibited-use boundaries.

Analytical roadmap

gp3ml now extends its materialized grouped-fold infrastructure with a complete, reviewable evaluation and reporting layer:

  • fold-local evaluation across gazepoint_group_folds;
  • explicit deterministic candidate grids and human-reviewed selection;
  • nested participant/stimulus-aware resampling;
  • observation, participant, stimulus, two-way, fold, and repeat uncertainty;
  • independent external-dataset declarations and transportability reports;
  • release-ready model cards recording selection, uncertainty, limitations, and external-validation status.

Nine deterministic synthetic articles demonstrate predefined recording-quality review, assigned-condition discrimination, observed non-sensitive endpoints, all supported generalization targets, contaminated provenance, nested resampling, and external validation. These workflows retain the package’s prohibited-use boundaries and do not perform autonomous model selection.

Project resources