Leakage review of the E3 code (conformal layer, selectors, E3 runner)

Reviewer: leakage-reviewer subagent, read-only, 16 Oct 2026, on branch phase-08-conformal-e3 (code version of the E3 runs: f38858e). A first pass on the same code was lost with a context reset; this is the second pass, and it confirmed the two issues that the first pass had raised. Dispositions are Claude Code’s proposals; the researcher decides the open items.

Verdict

No temporal leakage, no label-delay violation and no calibration contamination in the non-trap arms. Selectors receive only the opaque slot and label-free scores; the true labels leave the sealed slot only through the queue release and the metric code; delay is computed only in sim/queue.py; the Mondrian p-value (1 + #{s_i >= s}) / (n_c + 1) and the credibility p[pred] are correct; centroids and scalers are fitted on the current pool, which excludes hold-out and calibration rows. One design defect (finding 1) and one ADR deviation (finding 2) affect how registered secondary results are read.

Findings and dispositions

#	Severity	Finding	Disposition
1	major (a confound, not a leak)	The trap arm never retrains. With reserved share 0 every released row is routed to the calibration buffer and, with calibration_in_pool: false, none to the training pool (runner_e3.py:178-192), so n_new_since_fit stays 0 and refit is never true. The trap arm’s A-AUT difference to random therefore mixes “contaminated calibration” with “no retraining”. No test asserts the trap arm’s retrain count.	Open, researcher’s decision. The trap cells ran as registered and are kept. Report: the A-AUT of the trap arm is not read as the cost of contamination (CLAIMS E-45, chapter 7 section 7.5); only its coverage and credibility diagnostics are read. Options: (a) an addendum v3 that lets trap rows also enter the pool, then rerun the two trap cells (one per dataset; comparable cells took about 1,200 s on LAMDA); (b) add a matched policy: never comparator for credibility at share 0.20; (c) leave as is and keep the caveat. A test of the trap arm’s retrain count should be added with any of them.
2	minor (possible major reading)	The initial hold-out is cut by timestamp inside the window, not at a slot boundary (runner_e3.py:131, recalibrate.py:68-75), while ADR-0004 item 8 says “cut at slot boundaries”. On BODMAS benign timestamps are mostly midnight, so a cut can split a day and make the hold-out edge class-skewed. No future-to-train leakage (C1 against the evaluation slots still holds). On LAMDA the timestamp is probably month-level, so the cut is slot-aligned by accident [verify].	Recorded as a deviation. ADR-0009 rule 1 (“newest 20% of rows”) was written without reconciling ADR-0004 item 8. The runs are kept; chapter 8 lists it as a threat to validity for the initial calibration counts. A fix (split at a slot boundary chosen to hold out about 20%) would change every E3 cell, so it needs an ADR and a rerun; not proposed unless the researcher wants it.
3	minor	Equal-labels coverage was thin: the table check compared per-condition means, hard-coded seeds 0 to 4, skipped share 0 and the centroid score, and overwrote the keys of the three reserved shares.	Fixed in metrics/e3_tables.py: every arm (selector, share, score) is now compared per seed, window and slot with every other arm of the same model, delay and budget. It passes on the real E3 results and the tables are byte-identical.
4	minor	The E3 contamination, delay and spy tests are all slow-marked, so the fast gate never runs them.	Mitigated: the phase closeout runs pytest -m slow for tests/test_runner_e3.py and tests/test_make_e3.py. A fast synthetic variant is future hardening.
5	minor	Duplicate feature vectors between calibration rows and the training pool are not counted (only evaluation slot against pool is). An identical calibration and pool row scores too low and inflates p-values.	Open, listed in chapter 8 as a threat to validity; logging the overlap of buffer and pool hashes is future hardening.

What was checked and found sound

Sealed labels (selectors see OpaqueSlot, proba, cred only); delay (t+1+d and the drop after the window end only in queue.py; slot scored before its query); calibration routing in non-trap arms (origin check in the buffer, test patches the router and fails); reserved slice and selection are disjoint and sum to the budget; fit scope; Mondrian and credibility formulas.