Simulated hostile review: “How Long Does a Malware Detector Last? Label Delay, Analyst Budget and Conformal Monitoring in One Time-Aware Protocol”¶
Reviewer stance: hostile but fair, security-ML workshop. I read report/ch00 to ch09, README.md, docs/PROTOCOL.md, docs/PREREG.md, the two addenda, ADR-0007 to ADR-0010, docs/CLAIMS.md (all rows through E-65 and the L/D rows I needed), docs/ETHICS.md, the two data cards, docs/related_work_addendum.md, reports/tables/*.csv (spot checks), and the code that matters for leakage: src/driftwatch/sim/queue.py, sim/runner.py (the simulation loop and the random-split comparator), sim/monitoring.py, monitors/base.py, monitors/p_uniformity.py, metrics/stats.py. I did not run anything. I did not read the slicer (743 lines), runner_e3.py, selectors.py, the conformal package, the other three monitors or the tests in detail; statements about them rest on the report, the ADRs and the ledger. I could not verify any external citation (all are abstract-level by the authors’ own account, and several carry 2026 identifiers I cannot check).
Overall impression in one line: an unusually honest and well-instrumented study whose headline experiments are, by design, close to null, whose title question is never answered, and whose one genuinely interesting arm (monitors) is evaluated in a regime where the monitors are degenerate.
1. SOUNDNESS¶
1.1 MAJOR. The title question (“how long does a detector last”) is never measured. Search ch07 to ch09 for a lifetime quantity (months or slots until F1 or AUT falls below a stated floor, a retrain interval that holds F1 above a target, a half-life). There is none; “lifetime” appears only in the contribution sentence and in a limitation (ch08 section 8.3 item 8: “The effect of a stricter or looser threshold on the lifetime was not measured”). What is measured is A-AUT differences of 0.001 to 0.02 between policies. A reader finishing the report cannot say how many months a LAMDA LightGBM stays above, say, F1 0.7. The abstract’s four findings are about RQ1 to RQ4 contrasts, not about lifetime. Either define and report a lifetime metric (the per-slot F1 curves in reports/tables/e2_per_slot.csv would allow it) or retitle.
1.2 MAJOR. The manipulated intervention is nearly inert, and the equivalence margin is as large as the whole effect, so “equivalent” verdicts say nothing. ADR-0007 fixes the pool as “initial window plus arrived labels, weight 1, no truncation”; CLAIMS E-13 to E-15 show retraining changes 0.2% to 2.1% of decisions. The authors knew this before the tag (PREREG “What was already known”). Consequence: the retraining effect itself is tiny, and the registered margin of 0.01 A-AUT (P-10, an “[estimate]”) is of the same size. In E4 on LAMDA the entire headroom is: LightGBM never 0.7534 versus periodic(3) d=0 0.7598 (difference 0.0064); LR 0.6546 versus 0.6655 (0.0109) [E-50]. On BODMAS the same difference is 0.0000 and 0.0004 [E-51]. A monitor cannot beat periodic by more than that, and every monitor-minus-periodic interval lying “inside plus or minus 0.01” (“Equivalent”, 9 of 16 on LAMDA, 12 of 16 on BODMAS in the ch07 section 7.6 table) is a statement that is also true of “periodic versus never retraining” at this budget. Quote: ch07 7.6 “Monitor-triggered retraining matched periodic retraining within the margin in most rows” - matching to within a margin that equals the effect of retraining at all is not informative. The authors concede the E3 version of this (“that margin is as large as most of the delay-0 advantages, so these verdicts carry little information”) but not for E4, where it carries the headline. The conclusion “no monitor beat periodic” is therefore close to guaranteed by the design, not discovered by the experiment. The label-influence diagnostic has no noise-floor arm (ADR-0007 admits this), so “0.2% to 2.1%” cannot even be called small or large.
1.3 MAJOR. F1 at a fixed 0.5 threshold under uncontrolled class priors confounds drift with calibration and prevalence; the selection “advantage” is plausibly a prior effect. Evidence in the authors’ own ledger: LAMDA training priors across windows run 0.080 to 0.730 (E-08); 29 of 57 evaluated LAMDA months (51%) are below the 10% target ratio (E-08); the same detector scores F1 0.49 and 0.91 in adjacent months at slot ratios 0.052 and 0.100 (E-08). From reports/tables/e0_per_slot.csv (LAMDA, rolling, 10% target) I find per-slot F1 correlates 0.47 (LightGBM) to 0.55 (LR) with the achieved malware ratio of the slot. So a large part of the “decay” and of the 0.05 to 0.07 SD over windows is prevalence, not concept drift. The threshold is fixed (ADR-0006) while retraining changes the pool prior (arrived rows are C3-subsampled at about 10% malware; initial rows are raw). In E3, selection raises the arrived malware share from 0.085 (random) to 0.19 to 0.40 on LAMDA [E-47]; with a fixed 0.5 threshold, a prior shift alone moves F1. The report concedes it does “not separate” informativeness from class prior, and that the one arm with random-like class mix (centroid-score credibility) is below random [E-46, E-47]. That concession is the finding: RQ3’s positive result is most simply read as “labelling more malware raises F1 at a fixed threshold”. A class-stratified random baseline (match the malware share of the selected labels) or a threshold-free AUT (PR-AUC) is the obvious missing control, and it is cheap because predictions are stored.
1.4 MAJOR. The evaluation of the monitors is not meaningful as designed. (a) Saturation: alarm rates on the never arm are 0.56 to 0.80 for KS, MMD and p-uniformity and 0.997 for the domain classifier on LAMDA, 0.81 to 1.00 on BODMAS [E-63, E-56]; with a 500-versus-500 test at alpha 0.01 between months, any real covariate shift rejects. A monitor that alarms at every slot with min_gap 2 is “periodic(2)”. (b) The ground truth for “drop events” is a fall of F1 by 0.05 below the trailing 3-slot mean; given 1.3, a large share of these are prevalence artefacts of C3 under-target slots, which no covariate monitor can or should detect. Precision then sits at the base rate (0.44 to 0.54 against always-alarm 0.44 to 0.49) by construction. The statement “detected drops in F1 no better than alarming at random would” is therefore not evidence about monitors; it is evidence that the label is noise. (c) The “conformal” monitor (monitors/p_uniformity.py) is a label-free two-sample KS on the confidence score 1 - max(P, 1-P), Mondrian by predicted class. It is not a conformal credibility monitor of the Transcend/Transcendent kind (which use label-conditional p-values), so “conformal drift monitoring” in the title is carried by one of four monitors, and that one is a confidence-shift test. (d) CLAUDE.md rule 10 (“do not tune a monitor until it wins”) is used to freeze alpha = 0.01 and n = 500; that is a good rule against p-hacking, but freezing a configuration that the authors’ own synthetic check and Known limits (“Large slots”, addendum v3) predicted would saturate turns a pre-registration into a guarantee of a degenerate comparison. The report does state all of this (ch07 7.6 “What E4 does not show”), but then the abstract and ch09 still headline “No drift monitor beat retraining every third slot” without the qualifier that the monitors were operating as “retrain every ~2 slots”.
1.5 MAJOR. Interval claims rest on about four independent blocks, are described as “conservative” in some places and “too narrow” in others, and carry no multiplicity control. PREREG, ch03 section 3.4 and README say the two-stage bootstrap “counts within-window variation twice” and is “conservative”; ch03 itself then says the coverage test only checked “above 0.90”, and ch07 section 7.5 says of the E3 intervals “the intervals are likely too narrow, not conservative”. Both cannot be the working assumption. Technical points: (i) 16 LAMDA windows from one 62-month run with block length 4 gives four blocks; a percentile bootstrap on four blocks has poor coverage whatever is done at stage two. (ii) metrics/stats.py resample_means draws block starts uniformly from 0..n_w-b (non-circular), so end windows are under-represented. (iii) The decision rule is “95% interval excludes zero” applied unadjusted to 16 (E2), 16 (E3), 32 and 32 (E4) contrasts; several “detected” calls have a lower end indistinguishable from zero (E-41: LR uncertainty 1% [0.0000, 0.0035]; E-29: LightGBM C-B [0.0001, 0.0020]; E-60: [-0.0008, -0.0001] and [-0.0008, -0.0000]). (iv) The Holm-adjusted Wilcoxon values printed in the E2 and E3 tables (0.0312 in every row) are the floor for 10 seeds and 16 contrasts, i.e. the statement “all ten seed means share a sign given the realised windows”, not a test; the authors say so, yet they appear in every row of the ledger. (v) For a deterministic model (LR) “seeds” are only C3 and selection draws; the seed stage cannot represent model uncertainty. Net: read the LAMDA “detected” verdicts as “positive in this one 2018 to 2023 series”, which is what the authors mostly say; but “detected” is the registered word and it appears in the abstract-level summaries.
1.6 MAJOR/MINOR. RQ1 is a post hoc choice and an under-engaged result; BODMAS cannot carry its half of the study. The primary comparator was chosen after both gap numbers were seen (ADR-0008, ch07 7.2 “The primary comparator was therefore chosen after its number … were known”), and the choice moves LAMDA from 0.14 to 0.15 (above the band) to 0.03 to 0.04 (below it). The report is candid, and the conclusion drawn (“depends on how the comparator is built”) is defensible. What is missing: any engagement with why this study finds a small gap when TESSERACT famously found a large one (ch01 and ch05 cite it and move on); the random-split model is trained on only as many rows as the time-aware window, a handicapped comparator; the 5% sensitivity run is not discussed. On BODMAS, time-aware LightGBM A-AUT is 0.977 and the gap is 0.006 [E-35]; the nearest-centroid canary returns “NO_GAP, inconclusive” [E-09, E-22]; and BODMAS contributes 6 windows of 6 two-week slots (12 weeks, 20 distinct slots over about nine months). The protocol itself flags AUT > 0.90 as a leakage alarm and the report leaves it “open” (“no explanation beyond that is claimed here”). Also BODMAS benign timestamps are daily batch dates of unclear meaning (data card: “appear to be a different kind of date from the malware first-seen times”) so C2 on BODMAS is only nominal; a time-correlated crawl artefact would look like “no drift”. Honest framing: BODMAS shows a short, drift-poor series and cannot speak about lifetime.
1.7 MINOR. Registration hygiene is good but not clean. Disclosed in the documents: RQ1 registered after E0/E1 were read; addendum v2 written after the BODMAS E2 grid was read; E3 LAMDA seeds 5 to 9 added on 20 Oct after the five-seed rows were read. The last is a textbook optional-stopping pattern, made worse by CLAIMS E-42’s own remark that the five-seed set “could not pass Holm (smallest p 0.0625)”: the extension is what made a Holm-significant row possible. The bootstrap verdicts did not change (stated), so this is MINOR, but it should not be summarised as “pre-registered … run to completion” (abstract).
1.8 MINOR. Delay is coarse and unanchored. d = 0 already means “usable at the next slot”, i.e. one month on LAMDA and two weeks on BODMAS; analyst turnaround of days is not representable, and the range 0 to 3 slots is “not calibrated to an observed delay distribution” (ch08 8.3 item 10). “A delay of three slots costs about as much” as retraining (abstract) is a statement about a three-month shift on LAMDA at 218 labels per month, not about a realistic queue. Together with the perfect-analyst assumption (item 11), “label delay” is a simulated parameter with no external anchor.
1.9 MINOR. Leakage review: I found no temporal or label leak in what I read, which is a real strength; a few residuals. Positive: queue.py is the only module with d; SealedSlot exposes labels only to LabelQueue._take and score; the loop in runner.py releases, retrains, scores on the public view, selects, then queries, in that order; C1 is asserted against the pool each slot; SlotView has no label field; the p-uniformity monitor is label-free by construction and its calibration reference is features only. Residuals: (i) ADR-0009 rule 1 says “no timestamp is split across the boundary” for the E3 hold-out but ch08 8.3 item 6 says the hold-out “can cut inside a slot (a deviation from ADR-0004 item 8)”: the ADR and the report disagree. (ii) ADR-0010: the reference slot after a refit can contain about 0.8% rows that are training rows of the model in force, so the MMD score logit is partly in-sample (accepted, not corrected). (iii) The C3 subsampling itself uses labels to build the slot the selector sees (standard in TESSERACT, but it means the stream’s prevalence is oracle-set at exactly 10% when the data allow). (iv) LAMDA labels come from VirusTotal counts at an undisclosed scan date (D-16): all “delay” experiments start from labels that may embed future knowledge. The authors state (iv); nothing in the study can bound it.
1.10 MINOR. Negative and flat results are reported honestly, and I want that on record. The “does not show” lists in ch07/ch08 are the best part of the report: trap-arm A-AUT is explicitly disowned as evidence about contamination; the BODMAS E3 interaction is flagged as equal to the delay-0 advantage by construction; the Wilcoxon floor (P-18) is added after the fact. My complaints are about what the design could ever have shown, not about hiding.
1.11 MINOR. Conclusions that exceed numbers (quotes). (a) Abstract: “retraining on bought labels helps by hundredths of A-AUT and a delay of three slots costs about as much” - LAMDA only; on BODMAS it is thousandths and by construction one refit. (b) ch09 section 9.2 “Selection by uncertainty or credibility beat random selection”: true for A-AUT at a fixed threshold with a class-mix change, LAMDA mostly; the bullet says so in the next sentence, but the bold lead-in is stronger than the evidence. (c) ch07 7.6: “the monitors alarmed in most slots and found drops no better than chance would” - the baselines are exploratory and the drop label is questionable (1.4). (d) ch07 7.3’s slope-grows-with-budget remark is correctly marked exploratory.
2. NOVELTY AND POSITIONING¶
2.1 MAJOR. What is new is a combination and a discipline, not a finding. The report says so (“a statement of what was measured, not of priority”), and I credit it for not writing “first”. But examine what survives: (i) delay as a parameter: Botacin and Gomes (cited, treated at abstract level); (ii) budgeted active selection for Android drift: Chen, Ding, Wagner; Minnei et al.; DRMD; Aurora’s caution; (iii) conformal drift detection: Transcend, Transcendent, CADE; (iv) A-AUT: Chow et al. What the study adds is putting them in one grid with pre-registration, and the result that (under these settings) the effects are small and monitors do not beat periodic retraining. That is a legitimate negative-results contribution, but the paper has to carry it as such; presently the introduction frames “how long a detector lasts” as the question (see 1.1).
2.2 MAJOR. The related work is characterised from abstracts only, and the positioning sentence leans on that. ch05 and ch04 repeat “No full paper was read”. Examples that an examiner will press: Chen et al. is “the closest neighbour on budgeted retraining; the full paper would have to be read for the budgets used and for how labels were assumed to arrive” - this is precisely the delay/budget comparison on which the novelty rests; the table in 5.7 fills cells with “not in the abstract”; the conclusion “No neighbour read at abstract level combines…” (docs/related_work_addendum.md) is “not found in a limited search”. That is honestly worded, but a submission cannot rest its positioning on abstracts. Also no baseline from the literature is reproduced: the study does not run HCC, Transcendent, or any of the cited selective/ active methods (stated: “not re-implemented”), so its “credibility” and “p-uniformity” arms cannot be compared with published numbers, and “conformal drift monitoring” is represented by the weaker of the two ideas (see 1.4c).
2.3 MINOR. Chapter 4 includes material that does not bear on the study (4.5 on LLMs for PyPI packages, JAR files, Ghidra), padding that reads as filler; the authors themselves note it can be cut to a page. The adversarial-attack material (4.4) is survey-level and appropriately limited.
2.4 MINOR. The “two platforms” claim is thinner than it sounds. RQ5 (agreement across platforms) was not run, BODMAS has one refit per window, and the BODMAS E3 interaction is the delay-0 advantage. Effectively the dynamic results (delay, selection, monitors) are LAMDA results with a BODMAS control that barely moves.
3. REPRODUCIBILITY¶
3.1 Strengths. One YAML per run, config hash covers config + code version + lock file; uv.lock is committed and CI runs uv sync --frozen; per-row predictions are stored; tables regenerate from stored predictions with no fits (E-39: sha256-identical on all 9 tables and 4 figures); a ledger records every cell, code version and failure; the claims audit has a row (file, commit) for every number I checked (E-28 to E-65 agree with ch07 and with reports/tables/e4_contrasts.csv in the spot checks, e.g. the KS/LR/d=0 row -0.0089 [-0.0225, -0.0002]). A clean-clone reproduction of Table 1 on BODMAS is documented [E-24]. This is above the norm.
3.2 MAJOR. A stranger can rebuild the tables only if they already have the predictions; they cannot cheaply regenerate E2 to E4. results/ is git-ignored and “not committed”; the stored predictions (23.2 million rows for LAMDA E0 alone, E-01) are not archived anywhere the report names (no DOI, release or data deposit). Regenerating them costs 17.6 h (E2 LAMDA) plus 13.0 h (E4 LAMDA) of cell time on an 8 GB laptop, plus roughly 31 minutes per BODMAS Table 1 run. The “one-command reproduction” is Table 1 (E0 on BODMAS) only; the only automated end-to-end check on CI is on the synthetic fixture (reproduce table1 --smoke). The committed CSV tables are therefore the evidence of record, and the claim “every number … regenerate[s] from stored predictions” holds only on the author’s disk. Recommendation: deposit predictions (compressed parquet) with hashes.
3.3 MAJOR. The RQ1 comparator is not stable across rebuilds. README and E-24: “the random-split comparator columns within 0.006 (their draw is keyed by the data manifest hash and differs after a rebuild)”. The BODMAS LightGBM per-slot gap that underpins “no cell reaches the band” is 0.0057 [E-35]; a rebuild changes the comparator by up to 0.0056. A random draw keyed by a manifest hash makes a “reproduced” run differ in the number being reported. The comparator seed should depend on (seed, window) only.
3.4 MINOR. Version and platform brittleness. The run record covers package versions, but the results were produced on one Windows 11 laptop (“numbers from another machine may differ in the last digits, and that is reported, not corrected”); CI is Linux/Windows but only on fast and synthetic tests. Results at several code versions: results audit shows E2 at c26aa5f (32 src files changed since), E3 at f38858e (23), E4 at 1d302ea; the claim that later code does not change them rests on E-49 (8 of 8 E4 periodic cells reproduce E3 predictions) and the regression tests. That is decent evidence but it is not a rerun.
3.5 MINOR. Data acquisition is fragile. BODMAS comes from a Google Drive folder via gdown with a documented quota failure mode (exit code 3); dataset hashes are in data/manifest.json. LAMDA is pinned to a Hugging Face commit. Good practice, but both are third-party mutable hosts.
3.6 MINOR. Draft state. Every chapter header says “draft written by Claude Code for the researcher to rewrite and defend”; bracketed “[researcher: verify …]” markers remain throughout ch03 to ch09 (e.g. ch07 7.2 on the open BODMAS leakage alarm, ch07 7.3 on practical significance). A reviewer cannot tell which claims the author has independently verified; the report asks the reader to trust the audit scripts instead.
4. ETHICS AND LICENCES¶
4.1 Dual use: low, and handled sensibly. Feature files only; no binaries or decompilation; no evasion tooling; ch04 section 4.4 stays at survey level. A mild residual: results about “how often to retrain and how late labels hurt” describe defender dynamics, not attack methods. I see nothing here that I would ask them to withhold.
4.2 MAJOR/MINOR. Licence statements are inconsistent for BODMAS. docs/ETHICS.md table: “BODMAS | BSD-2-Clause [verified in build guide §1]”. The BODMAS data card says: “The repository code is BSD-2-Clause … No separate licence is stated for the dataset files; the build guide lists BSD-2-Clause [verify]”. The README and ch08 8.4 correctly say the dataset licence “is not stated by its authors and is an open question”. ETHICS.md should be corrected, and the study should say what it does with feature files whose licence is unstated (it downloads and uses them but does not redistribute; the synthetic fixture is the only committed data). docs/ETHICS.md also refers to docs/licence_audit.md (“Phase 12”), which does not exist in the repository. EMBER2024 and the AndroZoo exclusions are described correctly. I could not verify the LAMDA MIT licence beyond the data card’s own citation of the dataset card.
4.3 MINOR. Committed source documents. docs/source/ holds the “Build Guide”, “Implementation Plan” and “Phase Prompts” (PDFs and Markdown) and the repository tracks agent configuration (.claude/, hooks). If the repository is made public these are process documents of unclear licence and not research outputs; decide deliberately. ETHICS says nothing on them.
4.4 Label provenance. Well disclosed for LAMDA (VirusTotal counts: 0 benign, >= 4 malware, 1 to 3 discarded; AV-centric bias; scan date unpublished so labels can embed future knowledge [D-15, D-16]) and flagged [verify] for BODMAS [D-17]. Consequences are acknowledged (the “easier, high-consensus malware” effect), but they are not carried into the headline: detector decay on a dataset that discarded ambiguous apps and has labels from a later scan is a lower bound on difficulty, and this belongs in the abstract.
4.5 MAJOR. Disclosure of tool assistance is unresolved and the authorship statement is awkward. ch00: “[Researcher: the faculty’s policy on disclosing AI assistance is not known … Replace this paragraph with the wording the policy requires.]” while the same paragraph says “The report and the code were produced with Claude Code (Anthropic), working as the engineering lead”. The substance of the disclosure is there, but a placeholder in the submitted front matter, with the question of the institution’s policy still open, is a compliance defect. Related: the front matter cites “reviews by independent subagents (leakage, statistics, a simulated hostile review)” as part of the assurance; reviewers that are instances of the same model family that wrote the code and text are not independent in the sense an examiner will read, and this should be worded as self-review. The “researcher decides every gate” statement is plausible (ADRs show approvals), but the text should say what the researcher wrote.
The three weakest points an examiner would press¶
The experiment cannot show what the title and abstract imply (1.1, 1.2). No lifetime metric is reported, and the key manipulated variable (retraining on 1% to 5% of labels, weight 1, no truncation) changes 0.2% to 2.1% of decisions while the equivalence margin (0.01) equals the whole retraining effect at the E4 budget. “Equivalent” and “no monitor beat periodic” are then nearly foregone conclusions.
Confounded metric and a degenerate monitor evaluation (1.3, 1.4). F1 at a fixed 0.5 threshold, with LAMDA slot F1 correlated about 0.5 with slot prevalence and half the months below the target ratio, mixes drift, prior and calibration; selection’s “advantage” is not separated from a class-prior change; monitors alarm in 56% to 100% of slots and are scored against prevalence-driven F1 “drops”, so neither the RQ3 nor the RQ4 reading is a clean statement about informativeness or about drift detection.
Inference on very few independent units, plus an unfinished, unverified-by-the-author text (1.5, 3.3, 3.6, 4.5). About four LAMDA blocks, a BODMAS series of 12 weeks that shows almost no drift, unadjusted interval decisions with several borderline calls, an RQ1 comparator chosen after seeing the numbers and not reproducible to better than the size of the BODMAS gap, a draft marked for rewriting and an open AI-disclosure placeholder.
Verdict: weak reject¶
The process is exemplary for a student project: guardrails, a sealed label queue I could not break by reading, a ledger, a claims audit, candid “does not show” sections, and a refusal to claim priority. That earns the paper a serious reading and would make a good artifact or negative-results note. As a workshop submission it fails on substance: the title question is not measured, the two interventions that matter (retraining and drift-triggered retraining) are too weak under the chosen pool design for the registered margin to discriminate anything, the main positive results (selection advantage, delay cost) are confounded with class prior or hold on one Android series with about four independent blocks, the conformal-monitor arm is a degenerate test whose “negative result” is mainly about its own settings, and the report is still an unreviewed draft with an unresolved disclosure and licence inconsistencies. Concrete route to weak accept: report a lifetime metric from the stored F1 curves; add a class-matched random baseline and a threshold-free AUT (both computable from stored predictions); recompute the monitor comparison at a stated, pre-registered alarm-rate level (or with a larger effect-size threshold) and score against a prevalence-controlled drop definition; restate “conservative” intervals honestly with a block count in the table; fix the manifest-keyed comparator seed; resolve the BODMAS licence text and AI-disclosure paragraph; deposit predictions.
Responses to the simulated review¶
Written 22 Oct 2026 by Claude Code. The review above was produced by a fresh subagent that had not seen the drafting history; it is a model’s review of a model’s draft, so it is self-review and not independent review. I checked the review’s own factual claims before acting on them: the correlation of per-slot F1 with the slot’s malware ratio is confirmed (0.47 to 0.55 on LAMDA; CLAIMS E-66), the three tracked process documents under docs/source/, the stale BODMAS licence row in docs/ETHICS.md, and the manifest-keyed comparator draw (partly: the draw is keyed by seed and window identifier, and the window identifier contains the data source’s configuration digest).
“Changed” means the text or a file was changed in this commit. “Acknowledged” means the point stands and is now stated as a limit. “Researcher” means it needs a decision or an input that is not mine to make. A response never changes a result: no experiment was run and no registered analysis was altered.
1. Soundness¶
Point |
Response |
Where |
|---|---|---|
1.1 Lifetime not measured |
Changed (decided by the researcher on 22 Oct 2026: exploratory lifetime reading plus retitle). “Lifetime” is defined as A-AUT with the per-slot F1 curves, and an exploratory count of slots until the mean F1 first falls 0.05 is read from the stored curves and labelled as unregistered (new section 7.7; CLAIMS E-68). The title no longer promises a lifetime in months. |
ch00 title and abstract; ch01 section 1.2; ch07 section 7.7; ch09 sections 9.3 and 9.5 |
1.2 Margin as large as the headroom |
Changed. The headroom (never against periodic at the E4 budget) is stated next to the E4 reading, with the numbers, and “equivalent” is no longer allowed to carry the conclusion. The domain-classifier arm’s extra refits are stated. |
ch07 section 7.6 (“What E4 does not show”); ch08 section 8.1 (RQ4); ch00 abstract |
1.3 Class prior confounds F1 |
Changed and Acknowledged. The correlation is added to the ledger as an exploratory row and stated as a threat, with its consequences for RQ3 and for the monitors’ drop events. The class-matched random baseline and a threshold-free score are listed as future work; they would be new, unregistered analyses of stored predictions. |
CLAIMS E-66; ch08 section 8.3 item 14; ch09 section 9.5 |
1.4(a) Saturation, (b) drop labels |
Acknowledged. Both stated; the claim that monitors “detect drops no better than chance” is kept only with its exploratory label and the prevalence caveat. |
ch08 item 14 and item 13; ch07 section 7.6 |
1.4(c) The “conformal” monitor |
Changed. Stated plainly that p-uniformity is a label-free test of a confidence score and not a Transcend-style credibility monitor; the label-conditional quantity is the E3 selector. |
ch08 item 15; ch09 section 9.5 |
1.4(d) Frozen settings |
Acknowledged. The registration’s own known limit predicted saturation; freezing was the honest choice and also made a comparison of saturated monitors. The abstract no longer presents the monitor result as general. |
ch08 item 13; ch00 abstract |
1.5 Intervals |
Changed. The contradiction between “conservative” and “too narrow” is resolved: the seed stage is wider by design, the window stage is probably too narrow on about four blocks, the block starts do not wrap, and the net direction is unknown; no coverage is claimed. Unadjusted decisions and the Wilcoxon floor are already stated (chapter 3, section 3.8; chapter 7). A circular or stationary bootstrap stays future work. |
ch03 section 3.4 |
1.6 RQ1 |
Changed and Acknowledged. The report now says it does not explain the small gap, lists the untested differences, and says the BODMAS gap is at the reproducibility of the comparator. The post hoc choice of comparator was already disclosed. |
ch08 section 8.1 (RQ1) |
1.7 Optional stopping |
Changed. Disclosed in the abstract and in a threat item; the verdicts did not change. |
ch00; ch08 item 16 |
1.8 Delay unanchored |
Changed. The abstract says three slots is three months on LAMDA; items 10 and 11 of chapter 8 already state the simulated delay and the perfect analyst. |
ch00; ch08 items 10, 11 |
1.9 Leakage residuals |
Changed. (i) is a wording mismatch, not a disagreement: the hold-out never splits a timestamp but can fall inside a slot; both are now stated. (ii) to (iv) were already stated. |
ch08 item 6 |
1.10 Credit |
Noted. |
|
1.11 Over-claims |
Changed. The abstract is scoped to LAMDA where it was; the chapter 9 lead-in for selection no longer says “beat”. |
ch00; ch09 section 9.2 |
2. Novelty and positioning¶
Point |
Response |
|---|---|
2.1 A combination, not a finding |
Changed. The contribution statement now includes “a set of small or null effects under stated settings” (ch00, ch09 section 9.3). |
2.2 Abstract-level related work |
Acknowledged and Researcher. Stated in chapters 4 and 5 and unchanged: no full paper was read, and the positioning is worded as what the abstracts say. Reading Chen et al. in full is the researcher’s, and it is the one paper the novelty rests on. No literature baseline was re-implemented (stated since chapter 1). |
2.3 Chapter 4.5 is filler |
Changed (researcher’s decision, 22 Oct 2026). Section 4.5 was cut to one paragraph. |
2.4 Two platforms |
Changed. Added as item 19 of chapter 8. |
3. Reproducibility¶
Point |
Response |
|---|---|
3.1 Strengths |
Noted. |
3.2 Predictions not archived |
Changed (researcher’s decision, 22 Oct 2026: archive privately). The predictions of E0 to E4 (577 files, 7.20 GiB) are hashed in |
3.3 Comparator draw |
Changed. The mechanism is described exactly (seed and window identifier, which carries the source digest) and the effect on BODMAS is stated (item 17). The draw was not changed: that would change RQ1’s registered numbers. |
3.4 Version brittleness |
Acknowledged. The results audit ( |
3.5 Data acquisition |
Acknowledged. Documented in the data cards. |
3.6 Draft state |
Researcher. This is the stated plan: the report is a draft for the researcher to rewrite and defend, and the remaining |
4. Ethics and licences¶
Point |
Response |
|---|---|
4.1 Dual use |
Noted; the licence audit adds one sentence for the researcher to decide on ( |
4.2 BODMAS licence text |
Changed. |
4.3 Tracked process documents |
Changed (researcher’s decision, 22 Oct 2026). |
4.4 Label provenance in the headline |
Changed. In the abstract. |
4.5 AI-disclosure and “independent” |
Changed and Researcher. The reviews are described as self-review; the placeholder for the faculty’s disclosure wording stays until the policy is known, and the researcher is asked to say what they wrote and checked. |
Options for the researcher that would address the two largest points¶
Decision of 22 Oct 2026: options 1 and 3 were taken (the lifetime reading is section 7.7 of the report, ledger row E-68, and the title was changed); option 2 was not taken and stays future work. The class-prior confound (point 1.3) therefore remains a stated limit.
These are new analyses of stored predictions, not new experiments, and none is registered; each would be labelled exploratory, given a ledger row, and reported whatever it shows. They are not done, because the Phase 11 rule is to mark a missing number and ask.
A lifetime measure from the stored per-slot F1 curves (slots until F1 first falls a stated share below its first value), for the never-retrained detector on both datasets. Cheap; answers point 1.1 without a retitle.
A class-matched random baseline and a threshold-free score for E3, to separate informative selection from a class-prior effect (points 1.3 and the RQ3 reading). Moderate: the baseline needs the selected labels’ class share per slot and a rerun of the random arm with that share, which is an experiment; a threshold-free score alone is cheap.
Retitle, if neither is done: for example “…Under Delayed Labels: Small Effects and Null Monitors in a Pre-Registered Time-Aware Study”.