Statistics review of the E2 analysis code (Phase 6)

Reviewer: stats-reviewer subagent, 15 Oct 2026, run before any E2 result was read (the grid was still running; the reviewer was told not to read results/). Scope: src/driftwatch/metrics/stats.py, grid_tables.py, scripts/make_tables.py, scripts/make_figures.py, tests/test_stats.py, tests/test_make_tables.py, against the registered plan (prereg-v1). Verdict of the reviewer: the statistical machinery is faithful to the registration (contrasts, units, pairing, block lengths, margin, family size), but the table code was not ready to produce citable numbers; fixes for findings 1 to 4, the role and verdict columns of finding 10 and the diagnostics of finding 5 were to land and be re-reviewed before the first E2 table is read. No change to the registered bootstrap scheme was proposed without an addendum.

Findings and what was done

#	Severity	Finding (short)	Action	Where
1	blocker (RQ1 secondary row)	The “E0 window-level” comparator was built from f1_random_pooled, not from f1_random_split (the registered E0 comparator, CLAIMS E-19)	Fixed. Secondary gap = f1_random_split minus time-aware AUT per window and seed. A test builds a small E1-style run and checks the table equals (f1_random_split - aut) averaged over windows	grid_tables.rq1_table; tests/test_make_tables.py::test_the_e0_window_level_gap_is_the_prior_matched_comparator
2	major	Label counts of an arm stored as two seed blocks were merged with the last block winning, so a “0-4” row carried 5-9 numbers	Fixed. Counts are kept per block and aggregated over exactly the blocks inside the row’s seed set, weighted by cells	Collected.counts_for; test_units_signs_and_counts_follow_the_registered_definitions
3	major	The unit label of the delay slope said “negative = loses” while the value was AUT lost (positive)	Fixed. Unit text is “AUT lost per slot of delay (positive = AUT falls)”; the raw slope is estimate; a sign test	secondary_table; same test
4	major	A missing arm dropped a contrast silently; planned-but-unstarted cells were absent from the ledger table; Holm used the number of rows present; contrasts_table ran its own Holm	Fixed. One row per registered contrast, model and seed set, status = not_computed with the reason when an arm is missing; the ledger table lists planned cells from the grid spec (ids from sim/cell_ids.py, pinned equal to GridSpec.cells); Holm over a fixed family (datasets x models x 4), missing contrasts at p = 1, only the 10-seed primary rows adjusted; no Holm inside contrasts_table	contrasts_table, add_holm, collect(planned=...); test_a_partial_grid_still_produces_tables, test_holm_family_has_fixed_size...
5	major	Moving blocks without wrap weight edge windows less; the BODMAS window stage has only 16 equally likely block choices; the block is clipped silently when n_windows < block length; the last block is truncated	Scheme kept as registered (a change needs an addendum). Diagnostics added: block_len, block_len_effective, window_stage_coarse, boot_mean_minus_point. The report must say that BODMAS and Chow-stride intervals are coarse and not centred on the point estimate. A circular or stationary block bootstrap is a candidate for an addendum only before results are read	stats.effective_block, window_stage_is_coarse, Estimate.boot_mean
6	major	Wilcoxon on per-seed means ignores window dependence and treats seeds as replicates; with 5 seeds the Holm value is identically 1.0; the CSV puts it next to detected	Fixed. Columns renamed seed_level_wilcoxon_p and seed_level_wilcoxon_p_holm; the Holm value is computed only for the 10-seed primary rows; detected/verdict is the only decision column; chapter 3 states what the p-value measures	grid_tables, report/ch03_analysis.md section 3.6
7	major	On BODMAS d = 3 equals never, so C-D2 equals C-P numerically and the delay “rate” is one refit	Flagged, not removed (the contrasts are registered). delay_axis_note column on BODMAS contrast, slope and heat-map rows; chapter 7 must report C-D2 and C-P as one comparison on BODMAS and must not describe a BODMAS “rate” of delay	Collected.delay_axis_note
8	minor	a_aut_std had a different definition than in E0 (std over windows of seed-averaged AUT)	Fixed. a_aut_std follows E0 (std over windows per seed, then the mean over seeds); the seed-averaged version is a_aut_std_seed_averaged	aaut_table
9	minor	Absolute counts missing at both ends of budget steps; none on the heat map or contrast figure	Partly fixed. labels_lo and labels_hi on budget rows; counts on the heat-map axis and in the contrast-figure labels. Not added: labels queried before the last refit (stated as a limit of the registered unit)	secondary_table, scripts/make_figures.py
10	minor	detected written for 0-4 and Chow rows without a role; both colours for detected and within margin; wrong-direction intervals grey; 6-decimal rounding can print ci_low = 0 with detected	Fixed. role column (primary, seed_robustness, chow_secondary, descriptive); verdict three-way plus reversed, equivalent, not_computed; figure colours by verdict; CSV at 8 decimals. The 0-4 rows are nested in the 10-seed set and are not a replication	verdict(), scripts/make_figures.py
11	minor	Unit used the allowance n_budget, not queries	Fixed. Units use mean queries per slot (equal to the allowance under random selection)	_queried
12	minor	Seed stage draws seeds independently per window position, though overlapping windows share C3 draws	Documented, not changed. The scheme is the registered one; the net effect is conservative. Chapter 3 says so	report/ch03_analysis.md
13	minor	A contrast could mix cells from different code versions	Fixed. collect refuses a ledger with cells completed at several code versions unless allow_mixed_versions (a reviewed rerun)	collect
14	minor	Test gaps	Fixed. Tests for RQ1 secondary, counts across blocks, unit sign, not_computed, the fixed Holm family, mixed versions, effective block and coarseness, verdicts, planned ids	tests/test_make_tables.py, tests/test_stats.py

Additional change prompted by the second leakage review

stable_code_version retries a transient unknown answer from git (a first grid run was stopped once by one unreadable git rev-parse; the cells finished before that, at commit c26aa5f, are unaffected and the run was resumed from the ledger).

Still open for the researcher

• Whether to propose an addendum for a circular or stationary block bootstrap before E2 numbers are read. Recommendation: no; keep the registered scheme and report its coarseness.

• Whether BODMAS C-D2 and C-P should be presented as one row. Recommendation: yes, in prose and in the table note.

Re-review

The reviewer’s blocker and major items are fixed with tests (81 tests in test_grid.py, test_stats.py, test_make_tables.py pass at the commit that carries this file). A second pass of the stats-reviewer on the final tables and the E2 CLAIMS rows is part of the closeout once results exist.