Gate G2 evidence (Phase 5)

Gate: G2, end of week 8 (Sun 6 Dec 2026). Question: does the simulator with label delay and budget run end to end on at least one dataset, so that the delay and budget grid may start? (Stop rule: if not, cut to one dataset and one model.) Prepared: 13 Oct 2026, by Claude Code (engineering lead), from the Phase 5 code and tests, reports/tables/*.csv, docs/CLAIMS.md (S-01 to S-06, E-13 to E-21) and ADR-0007. Reviewed by the leakage-reviewer (first pass: 13 findings, no leak; fixes applied; the second pass did not complete, see Process) and by the stats-reviewer (12 findings; applied). The gate decision is the researcher’s (CLAUDE.md). This file states which criteria the evidence meets, which it does not, and what is open. Nothing here reads a detector result as a finding about decay recovery: the diagnostics count changed decisions and do not compute AUT.

Summary

Criteria 1, 2, 4 to 7 and 9 are met on both datasets. Criterion 3 is partly met: no leak was found, but the second leakage-review pass over the fixes did not complete (usage limit) and three items are carried. Criterion 8 (the pool ADR accepted) is open: ADR-0007 is PROPOSED and only the researcher accepts it. The stop rule (cut to one dataset and one model) is not triggered.

Two things need the researcher before the grid is run, and neither is a defect of the simulator:

  1. Which comparator carries RQ1. E0’s window-level comparator and E1’s per-slot matched comparator give different gaps on LAMDA (0.14 to 0.15 against 0.03 to 0.04) for the same fitted model, so the registered 0.05 band is passed or failed depending on the definition. G1’s criterion 1 was marked met on E0’s number. See “Open decision A”.

  2. Parallel runs do not fit the approved memory cap. Two BODMAS runs in two threads crossed 1.5 GiB; the single-run E1 BODMAS peak is 1.47 GiB (98% of the cap). The grid has to run one run at a time, or the cap has to change. See “Open decision B”.

Criteria

#

Criterion (Phase 5 prompt, PHASES.md G2)

Result

Met

1

Simulator with delay and budget runs end to end on at least one dataset

Periodic retraining every 3 slots, d = 0, random selection at 0.5% and 5%, on LAMDA (16 windows, 3 seeds, LR and LightGBM) and BODMAS (6 windows, 3 seeds); never-retrain baselines and E1 on both (CLAIMS E-13, E-14, E-16, E-17). 288 LAMDA refits per arm, longest 9.9 s

Yes, both

2

test_queue_delay passes for d in {0, 1, 2, 3}, including property tests; nothing lost or duplicated

21 tests; 150 generated scenarios; queued = released + ignored; release exactly at t + 1 + d (S-01)

Yes

3

Spy test passes; leakage-reviewer reports nothing unresolved on queue, pool, runner

Spy test passes at component and runner level, with the E1 comparator on and off (S-02). First review: no temporal leak; findings 1, 3, 4, 5, 6, 7, 8, 9 fixed with tests; 2 is this ADR; carried, not fixed: 11 (no feature-vector hash through queue and pool, so the E0 “seen before” mask is still built from the initial window only), 12 (calibration share is logged and not yet consumed), 13 (code_version ignores untracked files)

Partly: no leak found; second review pass of the fixes not completed; three carried items listed

4

Same config and seed reproduce identical per-slot predictions; an interrupted run resumes cleanly

Tests (S-05) and a real-data check (S-06): a BODMAS arm killed after 3 of 18 units, resumed, same predictions sha256 as the uninterrupted run made at another commit

Yes

5

Regression fixture reproduces the qualitative ordering

Never worst, more labels help, longer delay hurts; three deliberate breakages are each caught (S-03, S-04)

Yes

6

E1 figure exists and regenerates from stored predictions with no model fits

reports/figures/e1_random_vs_time.png; the script imports no model code (tested) and reads stored predictions plus the stored slot and window tables; regenerated three times during review

Yes

7

Fit-time and memory rules

Longest initial fit 8.6 s, longest refit 9.9 s (60 s rule). Memory: one run at a time fits (BODMAS E1 peak 1.47 GiB, LAMDA 0.82 GiB); parallel runs do not (E-20)

Yes, with Open decision B

8

Pool ADR accepted

ADR-0007 is PROPOSED. New-label weight 1 and no truncation were the researcher’s choices (11 Oct 2026); budget base, retrain anchor and the other rules are proposed and await acceptance

Open (researcher)

9

reports/G2.md exists

This file

Yes

Open decision A: the RQ1 comparator

The same fitted comparator model scored two ways (CLAIMS E-19; E1’s window table reproduces E-02 and E-03 exactly):

E0 window-level comparator

E1 per-slot comparator

Test rows

all held-out rows of the pooled window, including the training-era slots

held-out rows of the evaluation slots only

C3

one malware cap over the pool

per slot, checked

Aggregation

F1 pooled over rows, minus AUT

per-slot F1, then AUT, per window

LAMDA gap (LR; LightGBM)

0.152; 0.137

0.029; 0.040

BODMAS gap (LR; LightGBM)

0.028; -0.000

0.035; 0.006

Steps between the two LAMDA LR numbers (E-19): pooled F1 over all held-out rows 0.831; over evaluation-slot rows only, 0.727; mean of per-slot F1, 0.708; time-aware AUT, 0.679. The part due to the training-era rows and the part due to the single global C3 draw are not separated, because the comparator’s predictions were not stored in E0.

What the E1 comparator does not yet show cleanly (E-18): its slots reach a lower ratio than the time-aware slots on LAMDA (0.075 against 0.084) because the random draw removes proportionally more malware than benign from slots near the target. On the cells where both series reached the target the per-slot gap is larger (LAMDA 0.062 LR, 0.042 LightGBM). So the E1 gap is probably biased downward and the E0 gap upward; neither is a “leakage-only” quantity.

What can be said: the gap depends strongly on the definition, the pre-registered band is passed under one definition and not under the other on LAMDA, and both were seen before the tag. What cannot be said: that the gap is “mostly temporal leakage”, or that it is large or small. The choice of the primary comparator is the researcher’s. It should be written into the PREREG RQ1 row and an ADR as made after seeing both, with the definitional reason (a comparator matched per slot is the one that matches “same data, same C3 ratio”), and both numbers reported. A stratified within-slot hold-out (each slot loses the same share of malware and benign) would remove the ratio depletion; that would be a new comparator, an ADR and a rerun. G1’s criterion 1 (“gap of at least 0.05 on one dataset”) is comparator-dependent: met on E0’s definition, not on E1’s mean gap.

Open decision B: memory and parallelism

The memory watchdog measures the process, so concurrent runs add up. Two BODMAS arms in two threads were failed by the watchdog (E-20); one at a time they fit. The BODMAS E1 run came within 27 MiB of the cap (1.47 GiB of 1.5 GiB). Choices for the researcher: run the grid sequentially (default --workers 1), shrink BODMAS runs (the shrink order in PREREG: delay 2, budget 0.5%, windows, seeds), or raise the cap with the knowledge that the machine has 7.25 GB. I have not changed the cap.

What the label-influence diagnostic does and does not say

At 0.5% and 5% of the slot, after the first refit, retraining changes the 0.5 decision on 0.88% and 2.11% of LAMDA rows for LR and 0.71% and 0.99% for LightGBM; on BODMAS 0.22% and 0.45% (LR) and 0.19% and 0.22% (LightGBM) (E-13, E-14). Among malware rows on LAMDA at 5%: 7.7% (LR) and 4.4% (LightGBM) (E-15). These are counts of decision changes. There is no noise-floor arm, no split into fixed and introduced errors, no AUT. ADR-0007 therefore records the design as chosen (weight 1, no truncation), reports the counts, and does not use the word “modest”. Whether weight 1 is enough to make the budget axis informative is answered by the grid, not by this table.

Other caveats the researcher should weigh

  1. BODMAS delay axis. With nE = 6 and period 3 there is one useful refit; at d = 3 periodic equals never (tested), at d = 2 only slot-0 labels have arrived. Delay contrasts on BODMAS are limited by ADR-0005’s geometry.

  2. The BODMAS AUT alarm from G1 is still open. E1 on BODMAS inherits it; its per-slot gaps (0.006 to 0.035) are small against an AUT of 0.91 to 0.98.

  3. Windows overlap. LAMDA’s 16 windows are roughly four independent 12-month blocks. The E1 ranges and sign counts (E-17) are descriptive; no interval is reported.

  4. Seeds are not replicates for LR. Five seeds are five C3 draws and comparator splits.

  5. Figure positions are not time since training. One calendar month sits at the same position in every window of a stride-aligned run; the sawtooth at positions 4, 7, 10 on LAMDA is month composition, not drift. The figure carries a note.

  6. The pool mixes C3-subsampled arrived rows and raw initial rows (two priors); a property of the design (ADR-0007).

  7. Per-slot C3 draw keys differ between the time-aware series (seed, slot) and the E1 comparator (seed, window, slot), which adds window-to-window noise to the gap but not bias.

  8. Duplicates. LAMDA has 6.3% evaluated rows repeating a training-window vector (E-10). The pool does not yet carry feature-vector hashes, so repeats between arrived labels and later slots are not counted (carried item).

  9. Reruns. Diagnostic runs were rerun after leakage-review code changes (ADR-0007 run log, E-21); the shares are identical.

  10. Parallel threads and the shared initial-model cache are tested only on synthetic data. A cache key bound to training-row content was added after the review.

Process

  • Components were written test first (queue, selectors, policies, pool, config, store, runner); three deliberate breakages are each caught (S-04).

  • The leakage review found no temporal leak and one latent cache-key hazard, fixed with a test; the stats review found that the comparator definition moves the RQ1 verdict (above), that “same C3 ratio” had been asserted rather than shown (now columns in the table), and that two sentences overstated the diagnostics (corrected).

  • The second leakage-review pass over the fixes (commit 0117287 and later) was started and stopped by an API usage limit before it reported. The fixes are covered by tests and by deliberate-breakage checks (S-04), but they have had no independent second look. Rerun the leakage-reviewer on b3ff456..HEAD before relying on criterion 3.

  • Evidence runs used clean commit 2bd12d6 (--require-clean); the tables come from scripts/make_e1_figure.py and scripts/label_influence.py at commit 980ab15, committed at e170e7b.

What the researcher still decides

  • Accept, amend or reject ADR-0007 (hard rule 7 spirit: before any grid run).

  • Open decision A: the primary RQ1 comparator and the 0.05 band, before prereg-v1.

  • Open decision B: sequential grid, shrink, or a different cap.

  • Whether BODMAS’s nE = 6 stays (it caps the delay axis there) or a longer window is worth an ADR.

  • Gate G2 itself.