Sequence Data Validation and Preparation
Source:vignettes/sequence-data-validation-and-preparation.Rmd
sequence-data-validation-and-preparation.RmdWhy preparation is explicit
Ordered categorical data can contain missing states, duplicated
positions, unsorted rows, consecutive repeats, zero durations, unknown
states, and inconsistent metadata. Silent repair can change the
analytical object. gp3sequences therefore separates
non-modifying audit and validation from policy-driven preparation.
A deliberately problematic synthetic input
The example includes an unsorted sequence, a duplicated position, a missing state, a consecutive repeat, a zero duration, an unexpected state, and an unused factor level. Participant and group metadata remain constant within each sequence.
problem_data <- data.frame(
sequence_id = c("s2", "s1", "s1", "s1", "s1", "s2", "s2", "s2", "s2"),
sequence_order = c(2, 2, 1, 2, 3, 1, 2, 3, 4),
state = factor(
c("search", "search", "home", "search", NA,
"home", "home", "product", "other"),
levels = c("home", "search", "product", "checkout", "other", "unused")
),
duration = c(120, 100, 90, 110, 80, 100, 0, 150, 130),
participant_id = c("p2", "p1", "p1", "p1", "p1", "p2", "p2", "p2", "p2"),
group = c("interface_b", "interface_a", "interface_a", "interface_a",
"interface_a", "interface_b", "interface_b", "interface_b",
"interface_b"),
stringsAsFactors = FALSE
)
expected_states <- c("home", "search", "product", "checkout")
problem_data
#> sequence_id sequence_order state duration participant_id group
#> 1 s2 2 search 120 p2 interface_b
#> 2 s1 2 search 100 p1 interface_a
#> 3 s1 1 home 90 p1 interface_a
#> 4 s1 2 search 110 p1 interface_a
#> 5 s1 3 <NA> 80 p1 interface_a
#> 6 s2 1 home 100 p2 interface_b
#> 7 s2 2 home 0 p2 interface_b
#> 8 s2 3 product 150 p2 interface_b
#> 9 s2 4 other 130 p2 interface_bAudit without modification
audit_sequence_data() reports one row per issue using
stable issue codes and severity values. It does not repair the data.
audit <- audit_sequence_data(
problem_data,
sequence_id_col = "sequence_id",
order_col = "sequence_order",
state_col = "state",
duration_col = "duration",
metadata_cols = c("participant_id", "group"),
expected_states = expected_states
)
audit
#> sequence_id row column issue_code severity
#> 1 s1 2 sequence_order duplicated_position error
#> 2 s1 4 sequence_order duplicated_position error
#> 3 s1 5 state missing_state error
#> 4 s2 1 sequence_order duplicated_position error
#> 5 s2 7 sequence_order duplicated_position error
#> 6 s1 2 sequence_order unordered_rows review
#> 7 s1 4 state consecutive_repeated_state review
#> 8 s2 1 sequence_order unordered_rows review
#> 9 s2 7 duration zero_duration review
#> 10 s2 9 state unknown_state review
#> 11 <NA> NA state unused_state_levels info
#> value message
#> 1 2 The sequence contains a duplicated position.
#> 2 2 The sequence contains a duplicated position.
#> 3 <NA> The row has no usable state value.
#> 4 2 The sequence contains a duplicated position.
#> 5 2 The sequence contains a duplicated position.
#> 6 2 | 1 | 2 | 3 Rows are not ordered within the sequence.
#> 7 search The state repeats consecutively within the sequence.
#> 8 2 | 1 | 2 | 3 | 4 Rows are not ordered within the sequence.
#> 9 0 The duration value is zero.
#> 10 other The state is absent from `expected_states`.
#> 11 checkout | unused The factor contains unused state levels.
#> action
#> 1 Correct the position or apply an explicit first/last policy.
#> 2 Correct the position or apply an explicit first/last policy.
#> 3 Supply a state or apply an explicit missing-state policy.
#> 4 Correct the position or apply an explicit first/last policy.
#> 5 Correct the position or apply an explicit first/last policy.
#> 6 Sort deterministically by sequence and order.
#> 7 Preserve or collapse repeats through an explicit policy.
#> 8 Sort deterministically by sequence and order.
#> 9 Preserve, drop, or reject zero duration explicitly.
#> 10 Preserve, drop, or reject unknown states explicitly.
#> 11 Preserve or drop unused levels explicitly.
as.data.frame(table(audit$severity), stringsAsFactors = FALSE)
#> Var1 Freq
#> 1 error 5
#> 2 info 1
#> 3 review 5
as.data.frame(table(audit$issue_code), stringsAsFactors = FALSE)
#> Var1 Freq
#> 1 consecutive_repeated_state 1
#> 2 duplicated_position 4
#> 3 missing_state 1
#> 4 unknown_state 1
#> 5 unordered_rows 2
#> 6 unused_state_levels 1
#> 7 zero_duration 1Compact validation contract
A review-level issue does not automatically invalidate an input. Error-level issues must be resolved through source correction or an explicit supported policy.
validation <- validate_sequence_data(
problem_data,
sequence_id_col = "sequence_id",
order_col = "sequence_order",
state_col = "state",
duration_col = "duration",
metadata_cols = c("participant_id", "group"),
expected_states = expected_states
)
validation[c("valid", "status", "n_errors", "n_reviews", "n_info")]
#> $valid
#> [1] FALSE
#>
#> $status
#> [1] "fail"
#>
#> $n_errors
#> [1] 5
#>
#> $n_reviews
#> [1] 5
#>
#> $n_info
#> [1] 1
validation$mapping
#> role source_column
#> 1 sequence_id sequence_id
#> 2 sequence_order sequence_order
#> 3 state state
#> 4 duration duration
#> 5 metadata:participant_id participant_id
#> 6 metadata:group groupApply explicit preparation policies
This example deliberately chooses to:
- drop rows with missing states;
- retain the first row at duplicated positions;
- collapse consecutive repeated states;
- drop zero-duration rows;
- drop states absent from the declared state set;
- drop unused factor levels.
These are analytical choices, not universal defaults.
prepared <- prepare_sequence_data(
problem_data,
sequence_id_col = "sequence_id",
order_col = "sequence_order",
state_col = "state",
duration_col = "duration",
metadata_cols = c("participant_id", "group"),
expected_states = expected_states,
missing_state_policy = "drop",
duplicate_position_policy = "first",
repeated_state_policy = "collapse",
zero_duration_policy = "drop",
unknown_state_policy = "drop",
unused_state_levels = "drop"
)
prepared$status
#> [1] "pass"
prepared$decisions
#> step policy affected_rows
#> 1 missing_states drop 1
#> 2 unknown_states drop 1
#> 3 zero_durations drop 1
#> 4 duplicated_positions first 1
#> 5 row_order deterministic_sort 3
#> 6 consecutive_repeats collapse 0
#> 7 unused_state_levels drop 3
#> 8 column_mapping canonicalise 5
#> details
#> 1 Rows with missing or blank states were removed.
#> 2 States absent from `expected_states` were removed.
#> 3 Rows with zero duration were removed.
#> 4 Duplicated positions were resolved by retaining the first occurrence.
#> 5 Rows were ordered by sequence identifier, sequence order, and original row.
#> 6 The first row was retained for each repeated run; available durations were summed.
#> 7 Unused factor levels were removed.
#> 8 Mapped columns were standardised while unmapped columns were preserved.
prepared$data
#> sequence_id sequence_order state original_row duration participant_id
#> 1 s1 1 home 3 90 p1
#> 2 s1 2 search 2 100 p1
#> 3 s2 1 home 6 100 p2
#> 4 s2 2 search 1 120 p2
#> 5 s2 3 product 8 150 p2
#> group
#> 1 interface_a
#> 2 interface_a
#> 3 interface_b
#> 4 interface_b
#> 5 interface_b
prepared$audit
#> stage sequence_id row column issue_code severity
#> 1 input s1 2 sequence_order duplicated_position error
#> 2 input s1 4 sequence_order duplicated_position error
#> 3 input s1 5 state missing_state error
#> 4 input s2 1 sequence_order duplicated_position error
#> 5 input s2 7 sequence_order duplicated_position error
#> 6 input s1 2 sequence_order unordered_rows review
#> 7 input s1 4 state consecutive_repeated_state review
#> 8 input s2 1 sequence_order unordered_rows review
#> 9 input s2 7 duration zero_duration review
#> 10 input s2 9 state unknown_state review
#> 11 input <NA> NA state unused_state_levels info
#> value message
#> 1 2 The sequence contains a duplicated position.
#> 2 2 The sequence contains a duplicated position.
#> 3 <NA> The row has no usable state value.
#> 4 2 The sequence contains a duplicated position.
#> 5 2 The sequence contains a duplicated position.
#> 6 2 | 1 | 2 | 3 Rows are not ordered within the sequence.
#> 7 search The state repeats consecutively within the sequence.
#> 8 2 | 1 | 2 | 3 | 4 Rows are not ordered within the sequence.
#> 9 0 The duration value is zero.
#> 10 other The state is absent from `expected_states`.
#> 11 checkout | unused The factor contains unused state levels.
#> action
#> 1 Correct the position or apply an explicit first/last policy.
#> 2 Correct the position or apply an explicit first/last policy.
#> 3 Supply a state or apply an explicit missing-state policy.
#> 4 Correct the position or apply an explicit first/last policy.
#> 5 Correct the position or apply an explicit first/last policy.
#> 6 Sort deterministically by sequence and order.
#> 7 Preserve or collapse repeats through an explicit policy.
#> 8 Sort deterministically by sequence and order.
#> 9 Preserve, drop, or reject zero duration explicitly.
#> 10 Preserve, drop, or reject unknown states explicitly.
#> 11 Preserve or drop unused levels explicitly.Revalidate the canonical result
The prepared table uses stable canonical columns while preserving unmapped metadata and original-row provenance.
revalidation <- validate_sequence_data(
prepared$data,
sequence_id_col = "sequence_id",
order_col = "sequence_order",
state_col = "state",
duration_col = "duration",
metadata_cols = c("participant_id", "group"),
expected_states = expected_states
)
revalidation[c("valid", "status", "n_errors", "n_reviews", "n_info")]
#> $valid
#> [1] TRUE
#>
#> $status
#> [1] "pass"
#>
#> $n_errors
#> [1] 0
#>
#> $n_reviews
#> [1] 0
#>
#> $n_info
#> [1] 0
prepared$mapping
#> role source_column
#> 1 sequence_id sequence_id
#> 2 sequence_order sequence_order
#> 3 state state
#> 4 duration duration
#> 5 metadata:participant_id participant_id
#> 6 metadata:group group
prepared$state_levels
#> [1] "home" "search" "product"Errors that require source correction
Some conditions are intentionally not repaired automatically. Examples include missing sequence identifiers, missing or non-finite order values, negative or non-finite durations, absent mapped columns, duplicated column names, invalid column types, and metadata that varies within a sequence. These conditions require correction or an explicit redefinition of the sequence unit.
Reporting recommendations
A reproducible report should record the input mapping, expected states, every preparation policy, the audit table, the decision log, original and prepared row counts, and the final state levels. These records describe data handling; they do not validate a substantive interpretation of the resulting sequence patterns.