Chapter 8: Discussion and limitations (partial draft)

Status: draft written by Claude Code for the researcher to rewrite and defend; completed on 22 Oct 2026 for Tiers 1 and 2. The limits in sections 8.2 and 8.3 were written before the results (docs/PREREG.md, “Known limits”) and are not softened by them. No number is stated here without a claim id (the cell counts are in section 8.5 [E-26, E-40, E-48]). Target length: six pages.

8.1 What was measured

The study applied one time-aware protocol to two feature datasets, LAMDA for Android and BODMAS for Windows PE, and measured how the A-AUT of two detectors changes with label delay, analyst budget, the choice of which samples to label, and the trigger for retraining. It also measured how large the gap is between a randomly split test and the time-aware score. Each experiment was registered before its first result (tags prereg-v1 to prereg-v3) [P-17, E-48]. Below, “detected” means the 95% interval excludes zero in the hypothesised direction and “equivalent” means the whole interval lies within plus or minus 0.01 of A-AUT, as the registration defines them [P-10]; the words “confirmed” and “refuted” are not used. All differences are in A-AUT units.

RQ1, the random-split gap. Under the registered per-slot comparator the gap is 0.03 to 0.04 on LAMDA and 0.006 to 0.035 on BODMAS, below the 0.05 band (the LAMDA logistic-regression interval still contains 0.05); under the window-level comparator it is 0.14 to 0.15 on LAMDA [E-35]. The primary comparator was chosen after both numbers had been seen and is biased downwards on LAMDA [E-18]. What the intervals allow: a gap above the band is established under one definition and not under the other, and nothing in the design separates temporal leakage from the other differences between the comparators. The study does not explain why its gap is small where the TESSERACT papers report that commonly reported results are inflated by spatial and temporal bias [L-22]: the data, features, models and comparator construction all differ, and the random-split comparator is trained on only as many rows as the time-aware window, which handicaps it (ADR-0008); none of this was tested. On BODMAS the gap is about the size of the 0.0056 by which the random-split columns move when the data cache is rebuilt [E-24, E-35], so it is not distinguishable from zero at the reproducibility of that comparator.

RQ2, delay and budget. On LAMDA, periodic retraining at 5% of the slot with no extra delay is above never retraining by 0.017 [0.007, 0.024] for logistic regression and 0.020 [0.012, 0.022] for LightGBM, detected in both; a delay of three slots at the same budget costs 0.011 [0.005, 0.017] and 0.013 [0.008, 0.015], detected in both [E-30]. The budget response is positive, and diminishing returns are detected in one stratum of four [E-30, E-34]. On BODMAS the differences are at most 0.003 [E-29] and the delay axis measures one refit, not a curve [P-12]. The effects are hundredths of A-AUT against standard deviations over windows of 0.05 to 0.07 [E-28], so the data resolve differences of about that size and no finer.

RQ3, selection under delay. On LAMDA, with ten seeds and 16 windows, the advantage of uncertainty or credibility selection over random selection at equal labels is 0.0109 to 0.0177 with no delay and 0.0015 to 0.0095 after three slots, and the registered interaction is positive, with intervals excluding zero, in all eight selector, budget and model rows, 0.0061 to 0.0118 [E-42, E-43]. The advantage was therefore smaller after a delay of three slots in this sample, in both models. It is not separated from the change in class mix that selection brings [E-47], the seeds beyond the first five were added after the five-seed rows were read [E-42], and on BODMAS the interaction equals the delay-0 advantage by construction [E-41]. The study does not show that credibility adds anything over uncertainty.

RQ4, monitors as a retraining trigger. None of the 32 monitor-minus-periodic contrasts has an interval above zero [E-59, E-60]. On LAMDA, logistic regression with the KS monitor is below periodic retraining at both delays (reversed), the only rows that reach past the 0.01 margin; the other rows are equivalent or lie below zero inside the margin [E-59]. At the same number of refits, the timing of the alarms made no difference larger than 0.01 on LAMDA, where timing could be tested [E-61]. The monitors alarmed in most slots and detected drops in F1 no better than alarming at random would [E-63]. The headroom matters for this reading. At this budget, retraining every third slot moves A-AUT on LAMDA at delay 0 from 0.7534 to 0.7598 (LightGBM) and from 0.6546 to 0.6655 (logistic regression), and on BODMAS from 0.9692 to 0.9692 and from 0.9083 to 0.9079 [E-50, E-51]. The registered margin of 0.01 is about as large as the whole difference between never and periodic retraining, so a monitor arm that lies within the margin of periodic is a weak statement of equivalence; the reading rests on the signs and positions of the M-P and M-C intervals, and on none of them being above zero. The domain-classifier arm retrained 4.31 times per window against periodic’s 3.00 on LAMDA [E-50]. What this allows: at a budget of 1% and one frozen alarm level per monitor, no monitor was better than retraining every third slot, and several were slightly worse. It does not show that monitors cannot help at another level or budget.

RQ5, agreement across platforms. Not run (Tier 3). The experiments of RQ1 to RQ4 were run on both platforms, but their geometries differ, so a comparison of directions and magnitudes was not registered and is not made here.

8.2 What the results do not show

These statements are the ones a reader is most likely to attribute to the study, and the registered design does not support them.

• That the random-split gap is caused by temporal leakage. The per-slot comparator reaches a lower malware ratio than the time-aware series on LAMDA [P-14], and the two registered comparators disagree about whether the 0.05 band is met on LAMDA [E-19]. The report states the gap under each definition and does not attribute it.

• A delay curve on BODMAS. With six evaluation slots and retraining every third slot, a BODMAS window has one useful refit, and at a delay of three periodic retraining equals never retraining by construction [P-12]. A BODMAS number about delay is a number about one refit.

• Independent replication. LAMDA’s 16 rolling windows are about four independent twelve-month blocks, and logistic regression is deterministic given its training rows, so seeds are draws of the C3 subsample and of the selection, not replicates of the model [P-12]. The intervals are wide because of the registered two-stage design, which counts within-window variation twice; that is an argument from the design, not a measured coverage (chapter 3, section 3.4), and the window stage is coarse in strata with few windows.

• That labels do not help when a flat result appears. In the earlier diagnostic, retraining changed a small share of decisions at the smallest budgets [P-13]; a flat A-AUT at the smallest budget is the resolution of a few tens of labels per slot, not a finding that labels are useless.

• More than a bounded statement about selection. E3 is reported in section 7.5 with its own limits:

  • two delays only;

  • ten seeds on LAMDA, over about four independent blocks of windows;

  • a BODMAS interaction that equals the delay-0 advantage by construction;

  • a selector advantage that is not separated from the change in class mix of the arrived labels [E-47];

  • a trap arm whose A-AUT is confounded with not retraining [E-45].

  Its numbers are not comparable with E2 absolutes (ADR-0009).

• That drift monitors help, or that they cannot. E4 (section 7.6) found no monitor that beat periodic retraining at a budget of 1% [E-59, E-60]. The limits of that reading:

  • it covers one alarm level per monitor, frozen before the run;

  • the monitors alarmed in most slots [E-63], so where they saturate, the matched-count comparison does not test timing;

  • on BODMAS the contrasts rest on one useful refit per window.

  No monitor was tuned after the results (CLAUDE.md rule 10) [P-17]. E5 and E6 (Tier 3) were not run.

• Anything about an adaptive adversary. The datasets contain natural evolution only. A detector that decays slowly here is not thereby robust to evasion (chapter 4).

• Results on other hardware. Every cell was run on one laptop; numeric differences elsewhere are reported, not corrected (docs/GRID.md).

8.3 Threats to validity

1. Time labels. LAMDA is cut by AndroZoo’s collection date, and BODMAS benign samples carry date-only or placeholder timestamps (docs/data_cards/). Slots are cut by first-seen date, which is not the date a sample was compiled or first seen in the wild.

2. Class prior. Slots below the target malware ratio were kept and flagged (cap and flag); on LAMDA a large share of months are below target [E-08]. A-AUT mixes slots at different ratios.

3. Repeated feature vectors. A share of evaluated rows repeats a vector from the training window [E-10], and bought labels can repeat in later slots; the share is counted per slot and the rows are kept.

4. The pool design. The pool is the initial window plus arrived labels, weight one, never truncated (ADR-0007), which mixes raw and subsampled rows. A different pool rule could change the budget response.

5. Two models. Logistic regression and LightGBM with fixed hyperparameters; no tuning was done and none is claimed to be optimal.

6. E3 calibration set-up. The initial calibration hold-out is the newest 20% of the initial window by timestamp, which never splits a timestamp (ADR-0009) but can fall inside a slot (a deviation from ADR-0004 item 8; on BODMAS it can make the hold-out edge class-skewed). Duplicate feature vectors between calibration rows and the training pool are not counted, which can inflate p-values. The trap arm never retrains, so its A-AUT is not read as the cost of contamination (docs/reviews/leakage-review-e3.md).

7. Evaluation by one metric. AUT of the malware-class F1 hides the cost of benign flagging; FPR and FNR per slot are in the tables but are not part of the registered contrasts.

8. Label noise and the VirusTotal threshold. LAMDA’s labels come from VirusTotal detection counts: no detections is benign, four or more is malware, and one to three are discarded [D-15]. The dataset card calls the labelling AV-centric, and the scan date is not published, so a label may reflect knowledge later than the collection date [D-16]. The study cannot remove this form of label leakage and did not inject label noise; how BODMAS’s labels were produced is not described on its project page [D-17]. The effect of a stricter or looser threshold on the lifetime was not measured.

9. Temporal luck. Results are for one cut of time per dataset: LAMDA’s 16 rolling windows are about four independent twelve-month blocks and BODMAS has six evaluation slots per window [P-12]. A different start date, a longer span or a different slot length could give different numbers; the registered secondary check at Chow’s stride changes the window step only (chapter 7, section 7.3).

10. Simulated delay. Delay is a whole number of slots, the same for every label. Real delays differ by sample and depend on the analysis queue (Botacin and Gomes study exactly that [L-18]); the range of zero to three slots was not calibrated to an observed delay distribution, and neither dataset records when a label became known.

11. A simulated perfect analyst. A queried label is the dataset’s own label, returned after the delay without error. A real analyst makes mistakes, disagrees with a scanner, and may not be able to label a sample at all; the budget counts labels, not analyst hours.

12. Dataset mix. The two datasets differ in platform, feature type, slot length (a month against two weeks), time span, class balance and the number of refit events per window. They are analysed side by side, never pooled, and a difference between them is not attributed to the platform.

13. One alarm level per monitor. E4 froze every monitor setting before the run and tested one level (ADR-0010). A monitor that alarmed in most slots is a result about that setting. The registration’s own known limits had predicted that large slots would make the tests reject often (docs/prereg/addendum-v3.md); freezing the setting kept the comparison honest and also made it a comparison of saturated monitors with periodic retraining.

14. Class prior confounds the metric. F1 at a fixed threshold moves with the share of malware in a slot. Over the evaluation slots, per-slot F1 of the never-retrained detector correlates with the slot’s achieved malware ratio at 0.47 to 0.55 on LAMDA, where about half of the slots are below the target ratio [E-66, E-08] (exploratory, computed from the committed table). Part of what looks like decay is therefore class prior. Retraining also changes the pool’s class mix, and selection raises the malware share of the labels that arrive [E-47], so the selector advantage of RQ3 may be partly a prior effect; a class-matched random baseline or a threshold-free score would separate them, both are computable from stored predictions, and neither was registered or run. The monitors’ drop events inherit the problem: a drop in F1 caused by a slot below the target ratio is something no covariate monitor should detect, which is one reason why monitor precision is near the always-alarm level [E-63].

15. What the “conformal” monitor is. The conformal monitor of E4 (p-uniformity) is a label-free two-sample test of a confidence score, with calibration scores grouped by predicted class [P-17]. It is not a Transcend-style credibility monitor, whose p-values are conditional on the true label and so need labels [L-12, L-14]. The label-conditional conformal quantity of this study is the credibility selector of E3, and its use as a monitor is future work.

16. A seed extension decided after reading. The LAMDA seeds 5 to 9 of E3 were added after the five-seed rows had been read, and the five-seed rows could not pass the Holm adjustment because of the seed count; the bootstrap verdicts did not change [E-42]. This is the pattern called optional stopping, and the ten-seed rows should be read as indicative.

17. The RQ1 comparator is not stable to the last digit. Its random draw is keyed by the seed and the window identifier, and the window identifier contains the digest of the data source’s configuration, which includes the data manifest hash; a rebuilt cache therefore changes the draw and moves the comparator columns by up to 0.0056 [E-24]. Keying the draw by the seed and the time span of the window alone would remove this; it was not changed, because that would change RQ1’s registered numbers.

18. Predictions are not public. The result files under results/ are not committed; the committed tables are the evidence of record, and regenerating E4 on LAMDA alone took 13.0 hours of cell time [E-48]. The stored predictions of E0 to E4 (577 files, 7.20 GiB) are hashed in docs/predictions_manifest.csv, and a private archive of them is held by the researcher for examiners who ask [E-69]. Depositing them in a public service is future work and waits for the researcher’s go.

19. The dynamic results are mostly LAMDA results. BODMAS has one useful refit per window, a short and drift-poor series, and benign timestamps that are date-only or placeholders; its delay, selection and monitor numbers barely move [E-29, E-41, E-60]. A difference between the two datasets is not a finding about platforms.

[researcher: verify each of these against your own reading of the data cards; mark any you consider minor.]

8.4 Ethics and scope

The repository contains no malware, no decompiled code and no evasion tooling, and generates no adversarial examples (docs/ETHICS.md). The datasets keep their licences; BODMAS’s dataset licence is not stated by its authors, so the report says only that the files are used as published and are not redistributed (docs/licence_audit.md). Describing how detectors decay is not the same as describing how to defeat them, and the report keeps to the former.

Dual use. Knowing how fast a detector decays and when retraining pays could in principle help an adversary time an attack. The study measures natural drift on public datasets, reports aggregate effects, offers no method to cause drift, and finds effects that are small; it does not model an adaptive adversary (section 4.4).

8.5 Status of the tiers

Tier 1 (E0 to E2) is complete on both datasets: all 54 E2 cells ran and none failed [E-26]. Tier 2 (E3, E4) is complete on both datasets: 56 E3 cells and 76 E4 cells ran and none failed [E-40, E-48]. Gates G1 to G4 were decided PASS by the researcher from the evidence files (reports/G1.md to reports/G4.md, docs/PHASES.md). Tier 3 (E5, E6) was not run: the go checklist failed on the capstone deadline, which is early December 2026 (reports/G5-tier3-decision.md).

8.6 Future work

• A stratified within-slot random-split comparator, which would remove the ratio depletion of the per-slot comparator (ADR-0008).

• A longer BODMAS evaluation window, so that the delay axis has more than one refit (a new ADR).

• A circular or stationary block bootstrap as a sensitivity check on the registered window stage (an addendum before results are read).

• Drift monitors at lower alarm rates, for example with a larger sample per slot or a stricter significance level, registered in a new addendum before any run. Here E4 tested one frozen level, and most monitors alarmed in most slots [E-63].

• A trap arm whose released labels also train the model, so that contamination is separated from not retraining (an addendum before the rerun).