API reference¶
The public building blocks of the protocol, generated from the docstrings. The command-line interface is described by driftwatch --help and each subcommand’s --help.
Time slicing and the constraints¶
The time slicer: initial training window, then evaluation slots one at a time.
Dataset-agnostic. A SlotSource is anything that can list time slots with their
class counts (without touching features) and load one slot’s rows: an in-memory table
(driftwatch.data.tables) or the dataset caches (driftwatch.data.presets).
Planning is features-free:
plan_slots()applies the size gate (ADR-0004): a slot is usable only if it is not empty, not partial, and keeps at leastmin_malwaremalware after C3 subsampling at the target ratio. Every other slot is skipped with a reason.plan_windows()carves windows from maximal runs of consecutive usable slots. A window (nTtraining slots thennEevaluation slots) never contains a gap or a skipped slot.nE=Noneis the single-run protocol: train on the firstnTslots of a run and evaluate on the rest of it.
Iteration is lazy: slice_windows() yields a Window whose training slots are
loaded (raw rows), and Window.eval_slots() then loads, subsamples and checks one
evaluation slot per step. Each evaluation slot passes C3, C2 and C1 (the constraints module)
before it is yielded, so a slot that violates a constraint cannot leave the iterator.
Selectors, policies and monitors must only ever see EvalSlot.public().
- exception driftwatch.data.slicer.SlicerError[source]¶
The slicer’s own invariants broke (grid, counts, densification), not a C1-C3 check.
- class driftwatch.data.slicer.WindowSpec(nT: int, nE: int | None, stride: int, target_ratio: float = 0.1, min_malware: int = 50, seed: int = 0)[source]¶
One slicing configuration: window shape, target ratio, size gate and seed.
- class driftwatch.data.slicer.RowFilter(name: str, fn: Callable[[DataFrame], Any], columns: tuple[str, ...] = ())[source]¶
A named, label-free rule that marks rows to exclude (True = exclude).
The rule only ever sees the columns it declares, so a filter cannot read labels (or family, VT counts, categories) unless it names them, and a reviewer can see that it does.
- class driftwatch.data.slicer.SlotRef(slot_id: 'str', start: 'np.datetime64', end: 'np.datetime64', index: 'int')[source]¶
- class driftwatch.data.slicer.SlotInfo(ref: SlotRef, n_rows: int, n_malware: int, n_benign: int, n_excluded: int = 0, structural: str = '', n_naive_ts: int = 0)[source]¶
One grid slot as a source reports it (counts exclude rows the source filtered out).
- class driftwatch.data.slicer.SlotRows(ref: 'SlotRef', y: 'np.ndarray', ts: 'np.ndarray', row_id: 'np.ndarray', X: 'Any' = None, fv_hash: 'np.ndarray | None' = None)[source]¶
- class driftwatch.data.slicer.PlannedSlot(ref: 'SlotRef', n_rows: 'int', n_malware: 'int', n_benign: 'int', n_excluded: 'int', n_malware_kept: 'int', structural: 'str', usable: 'bool', skip_reason: 'str', n_naive_ts: 'int' = 0)[source]¶
- class driftwatch.data.slicer.WindowPlan(window_id: 'str', segment: 'int', train: 'tuple[SlotRef, ...]', eval: 'tuple[SlotRef, ...]')[source]¶
- class driftwatch.data.slicer.SkippedRun(first_slot: 'str', last_slot: 'str', n_slots: 'int', reason: 'str')[source]¶
- class driftwatch.data.slicer.Schedule(windows: 'tuple[WindowPlan, ...]', skipped_slots: 'tuple[PlannedSlot, ...]', skipped_runs: 'tuple[SkippedRun, ...]')[source]¶
- driftwatch.data.slicer.plan_slots(source: SlotSource, spec: WindowSpec) list[PlannedSlot][source]¶
Apply the size gate to every grid slot. Reads counts only, never features.
- driftwatch.data.slicer.plan_windows(planned: list[PlannedSlot], spec: WindowSpec, source_digest: str = '') Schedule[source]¶
Carve windows from runs of usable slots;
source_digestkeys results by source.
- driftwatch.data.slicer.seen_before_mask(current_fv: ndarray, pool_fv: ndarray) ndarray[source]¶
True where a row’s feature vector is already in
pool_fv.“Past only” is the caller’s contract: pass the training pool that existed when the row was evaluated, never a set that includes later slots (ADR-0004 item 6).
- class driftwatch.data.slicer.TrainWindow(slots: tuple[SlotRows, ...])[source]¶
The initial training window: raw rows (no C3), kept slot by slot.
Hold-outs for calibration or early stopping must be cut at slot boundaries with
split_at_slot(), never by row position inside a slot (ADR-0004 item 8).- split_at_slot(k: int) tuple[TrainWindow, TrainWindow][source]¶
(older
kslots, newer slots);kmust leave both sides non-empty.
- class driftwatch.data.slicer.OpaqueSlot(dataset: str, slot_id: str, X: Any, handle: ndarray)[source]¶
What a selector, policy or monitor may see of an evaluation slot.
A distinct type from the loader’s
PublicSlot(which keeps the dataset’s own row ids and order, and is for loader-level code only), so a type check can tell them apart. There is deliberately norow_idand no back-reference to the slot.
- class driftwatch.data.slicer.EvalSlot(ref: SlotRef, dataset: str, X: Any, y: ndarray, ts: ndarray, row_id: ndarray, c3: C3Result, perm: ndarray, n_seen_in_initial_train: int | None = None, fv_hash: ndarray | None = None)[source]¶
One evaluation slot after C3 subsampling, already checked against C1, C2 and C3.
- public() OpaqueSlot[source]¶
The label-free view (hard rule 4): features and opaque handles only.
Rows come in a seeded shuffle that depends on
(seed, slot id)and nothing else, and the ids are the handles0..n-1of that order. Neither the order nor the ids say anything about time rank, malware drops, source file or class. The features are a copy, so a consumer cannot alter what is later scored. Only the label queue and the metrics map handles back, withresolve().
- class driftwatch.data.slicer.Window(plan: 'WindowPlan', dataset: 'str', train: 'TrainWindow', spec: 'WindowSpec', _source: 'SlotSource', _planned: 'dict[str, PlannedSlot]', _log: 'SliceLog | None' = None)[source]¶
- driftwatch.data.slicer.slice_windows(source: SlotSource, spec: WindowSpec, slicelog: SliceLog | None = None, *, only: Sequence[int] | None = None) Iterator[Window][source]¶
Plan once, then yield each window with its training slots loaded.
Skipped slots and runs are logged through
loggingand recorded inslicelog.onlyrestricts iteration to those positions of the schedule (other windows are not loaded), which keeps spot checks such as the canary cheap.
- driftwatch.data.slicer.count_windows(source: SlotSource, spec: WindowSpec) int[source]¶
How many windows the spec yields on this source (features-free).
- class driftwatch.data.slicer.SliceLog[source]¶
Collects the per-slot manifest of a slicing run and writes it.
- write(out_dir: Path | str, *, spec: WindowSpec, source: SlotSource, extra: dict[str, Any] | None = None) dict[str, Path][source]¶
Write slot_manifest.csv, window_manifest.csv and slicing_run.json under out_dir.
extra(wall-clock, peak memory, row counts) is stored underrun.
TESSERACT constraints C1-C3 as checks that raise (Build Guide section 3; PROTOCOL section 4).
C1 (temporal training consistency): every training sample is strictly older than every evaluation sample.
C2 (temporal window consistency): an evaluation slot holds malware and benign samples, all from that slot’s own window
[start, end).C3 (realistic ratio): the evaluation slot is brought to the target malware ratio by seeded subsampling of malware only. A slot already below the target keeps all its rows and is flagged
below_target(“cap and flag”, the researcher’s decision of 10 Oct 2026; ADR-0005 records it). The achieved ratio is therefore never above the target.
Every check raises a ConstraintViolation. None of them warns, and none can be
switched off. Never weaken a check to get green: write an ADR and ask the researcher.
The checks take plain arrays. They know nothing about datasets, slots or files, so the slicer and the simulator (which re-checks C1 against the real training pool) share them.
- exception driftwatch.data.constraints.ConstraintViolation[source]¶
A TESSERACT constraint does not hold. Never caught and ignored.
- exception driftwatch.data.constraints.C1Violation[source]¶
A training sample is not strictly older than an evaluation sample.
- exception driftwatch.data.constraints.C2Violation[source]¶
An evaluation slot is not made of both classes from its own window.
- exception driftwatch.data.constraints.C3Violation[source]¶
The malware ratio was not reached by malware-only subsampling, or the ratio is invalid.
- driftwatch.data.constraints.check_c1(train_ts: Any, eval_ts: Any) None[source]¶
Raise unless the newest training timestamp is strictly older than the earliest test one.
An empty side cannot be validated and raises too. Missing timestamps raise.
- driftwatch.data.constraints.check_c2(ts: Any, y: Any, start: Any, end: Any) None[source]¶
Raise unless every row lies in
[start, end)and both classes are present.
- class driftwatch.data.constraints.C3Result(target_ratio: float, n_target_malware: int, n_malware_before: int, n_benign_before: int, n_malware_after: int, n_benign_after: int, achieved_ratio: float, share_malware_discarded: float, status: Literal['at_target', 'below_target'])[source]¶
Counts before and after subsampling, as logged per slot in the manifest.
- driftwatch.data.constraints.target_malware_count(n_benign: int, ratio: float) int[source]¶
Largest malware count whose share of the slot does not exceed
ratio.floor(n_benign * r / (1 - r)), in exact arithmetic: 900 benign at 10% give 100.
- driftwatch.data.constraints.subsample_malware(y: Any, ratio: float, seed: int, slot_id: str) ndarray[source]¶
Row indices to keep: every benign row plus a seeded draw of malware (sorted).
The generator depends only on
(seed, slot_id), so a slot’s draw is identical across windows, runs and grid cells, whatever else was sliced before it. The draw is the head of one seeded permutation, so a lower ratio selects a subset of a higher ratio’s malware (the sensitivity check is nested in the headline ratio). A slot already at or below the target keeps every row.
- driftwatch.data.constraints.check_c3(y: Any, kept: Any, ratio: float) C3Result[source]¶
Raise unless
keptis exactly what malware-only subsampling toratioleaves.Raises when a benign row was dropped,
keptis not a set of valid distinct indices, the ratio ends above target, a reachable slot misses its target count, or malware was dropped from a slot that was already below target.
Metrics¶
AUT, A-AUT and the rolling-window geometry of Chow et al. (arXiv 2506.23814v2).
AUT is the guide’s AUT(f, N) = 1/(N-1) * sum_{k=1..N-1} (f_k + f_{k+1}) / 2: numpy’s
trapezoid over per-slot F1 with time rescaled to [0, 1], so 1.0 means no decay. TESSERACT
arXiv v4 says its published Eq. (4) carried an extra 1/N factor (a typo); this is the form
the results rely on (ADR-0006).
A-AUT is (mean, std) of AUT over rolling train/test windows (Eq. 1). The window geometry
of the paper, for n slots:
nE >= nT(Eq. 2-4): training windows start everynTslots,k = floor((n-nE)/nT).nE < nT(appendix A-E, Eq. 6-8): they advance bynE,k = floor((n-nT)/nE).
Both are stride = min(nT, nE) and k = floor((n-nT-nE)/stride) + 1. The slicer takes
the stride as a parameter: ADR-0005 keeps its own strides (LAMDA 3, BODMAS 2) as the
headline, and chow_stride gives the paper’s value for the secondary check.
- class driftwatch.metrics.aut.AAut(mean, std, n_windows)[source]¶
- mean: float¶
Alias for field number 0
- std: float¶
Alias for field number 1
- n_windows: int¶
Alias for field number 2
- driftwatch.metrics.aut.aut(f1_per_slot: Sequence[float]) float[source]¶
Normalised trapezoid area under per-slot F1 (needs at least two slots).
- driftwatch.metrics.aut.a_aut(aut_per_window: Sequence[float], ddof: int = 1) AAut[source]¶
Mean and standard deviation of AUT over windows (sample std by default).
- driftwatch.metrics.aut.chow_stride(nT: int, nE: int) int[source]¶
How far the paper advances a window:
min(nT, nE).
Label delay and the sealed queue¶
The label queue: the only module that knows about the delay d (hard rule 3).
Rule (PROTOCOL.md section 4): a label queried while evaluation slot t is being streamed
becomes usable from slot t + 1 + d (d = 0 means the next slot). A slot is scored
before any label queried in it exists, so a label can never inform the slot it came from.
Window end (documented rule): evaluation slots are indexed 0..n_slots-1 within a window.
A label whose release slot t + 1 + d is >= n_slots has no slot left to inform, so it
is ignored and counted in LabelQueue.n_ignored. Windows are independent runs;
nothing carries over into the next window.
Sealed labels (hard rule 4): SealedSlot wraps an EvalSlot so that nothing
else in sim/ holds the true labels. They leave it by two doors only: the queue’s
LabelQueue.query() (which hands them back, delayed, through LabelQueue.release())
and SealedSlot.score() (metric code). Selectors, policies and the pool receive
OpaqueSlot views and released rows, never the slot.
- class driftwatch.sim.queue.Scored(y_true: ndarray, row_id: ndarray, scores: SlotScores)[source]¶
One scored slot: the labels metric code saw and the scores computed from them.
- class driftwatch.sim.queue.SealedSlot(slot: EvalSlot)[source]¶
An evaluation slot whose true labels are reachable only by the queue and metric code.
- n_seen_in(pool_fv_hash: ndarray | None) int | None[source]¶
Rows of this slot whose feature vector is already in the training pool.
A count only: the hashes stay inside the slot.
Nonewhen either side carries no hashes (a synthetic source without them).
- public() OpaqueSlot[source]¶
The label-free view (the same object every call, so scoring order is stable).
- class driftwatch.sim.queue.Released(slot_index: int, X: Any, y: ndarray, ts: ndarray, row_id: ndarray, queried_in: ndarray, fv_hash: ndarray | None = None, origin: ndarray | None = None)[source]¶
The labels that become usable at one slot.
XisNonewhen nothing arrived.
- class driftwatch.sim.queue.QueryReceipt(n_queried: 'int', release_slot: 'int | None', n_ignored: 'int')[source]¶
- class driftwatch.sim.queue.LabelQueue(delay: int, n_slots: int)[source]¶
Holds queried labels back and releases each one at slot
t + 1 + delay.- release_slot(query_slot: int) int[source]¶
The slot from which a label queried at
query_slotis usable (the delay rule).
- release(slot_index: int) Released[source]¶
The labels that arrive at
slot_index. Call once per slot, in order from 0.
- query(slot_index: int, slot: SealedSlot, selection: Selection) QueryReceipt[source]¶
Spend budget on
selectionof the slot being streamed; labels arrive later.Only the slot just released may be queried (stream order: it was already scored).
Run configuration¶
Run configuration: one strict YAML file per run, and the hash that identifies its results.
Keys follow plan Appendix C (.claude/rules/configs.md): dataset, slots, window
(nT, nE, stride), ratio, seeds, model, policy, selector, delay, budget_pct, nonconf, pool,
monitors, output. A config either describes the Phase 4 baseline (selector, delay and
budget_pct null, policy: never) or a simulation (Phase 5: all three set; policy
never or periodic). nonconf adds the conformal layer (Phase 8). monitors (Phase 9)
runs drift monitors on the conformal path, and two policies use them: monitor (retrain on
the trigger monitor’s alarm) and matched (the matched-count control, whose retrain count
comes from a finished monitor run named by source). Monitor settings are frozen in code
(addendum v3); the config only names which monitors run.
e0 lists the window variants to run (the key keeps its Phase 4 name so that E0 hashes stay
valid) and, in a simulation, the same variants apply.
E0 configs hash exactly as they did in Phase 4: to_dict emits the simulation keys only
when they are set.
The config hash covers every config value, the code version (git commit, -dirty when
the tree has uncommitted changes) and the lock file, so a changed config, change of code or
change of package pins gives a different hash.
- class driftwatch.sim.config.RunConfig(name: 'str', dataset: 'str', preset: 'str', min_malware: 'int', nT: 'int', nE: 'int', stride: 'int', target_ratio: 'float', sensitivity_ratio: 'float | None', seeds: 'tuple[int, ...]', models: 'tuple[dict[str, Any], ...]', policy: 'str', variants: 'tuple[str, ...]', random_split: 'bool', output: 'str', per_slot_random_split: 'bool' = False, policy_period: 'int | None' = None, selector: 'str | None' = None, calibration_fraction: 'float | None' = None, delay: 'int | None' = None, budget_pct: 'float | None' = None, pool_new_label_weight: 'float | None' = None, pool_truncation: 'str | None' = None, nonconf: 'dict[str, Any] | None' = None, policy_trigger: 'str | None' = None, policy_min_gap: 'int | None' = None, policy_source: 'str | None' = None, monitors: 'tuple[str, ...] | None' = None, monitor_cache: 'str | None' = None)[source]¶
- property policy_label: str¶
The arm’s policy column:
monitor-ks,matched-ks, or the plain name.
- driftwatch.sim.config.lock_sha256(repo_root: Path | str) str[source]¶
SHA-256 of
uv.lockwith line endings normalised (Windows and Linux agree).An installed package has no lock file; the hash then covers the versions of the packages in
PACKAGESand the Python version (prefixed so it can never equal a lock hash). That pins an installed smoke run, not transitive dependencies. A source checkout whose lock file is missing is an error, never a fallback: evidence runs need the real lock.
Detectors¶
The detector interface: fit on one training window, score with a fixed threshold.
fit(X, y, window_id)is the only place labels enter. Scoring takes features only.The decision threshold is fixed at construction (default 0.5, ADR-0006). A threshold tuned on data may use only the final training slot, and is passed in here, never learned here.
A fitted detector carries the id of the window it was fitted on. Passing another window’s id to
predict_probaraises, so a model cannot silently score under the wrong window.Scalers and any other fitted state are built inside
_fitfrom the training rows only.
- exception driftwatch.models.base.NotFittedError[source]¶
A detector was asked to score before it was fitted.
- exception driftwatch.models.base.WindowMismatch[source]¶
A detector fitted on one window was asked to score under another window’s id.
- class driftwatch.models.base.Detector(seed: int, threshold: float = 0.5)[source]¶
- spec() dict[str, Any][source]¶
Name, threshold and hyperparameters: what
make_modelneeds, and what is logged.
- feature_space(X: Any) Any[source]¶
The features as the model sees them (scaled for linear models, raw otherwise).
Used by the centroid nonconformity score; it applies only what was fitted on the training window and reads no label.
- decide(proba: ndarray) ndarray[source]¶
Malware (1) when P(malware) is strictly above the fixed threshold.
- property seed_sensitive: bool¶
False when the fit cannot depend on the seed, so one fit serves every seed.
Conformal layer¶
Class-conditional (Mondrian) conformal p-values and credibility.
For a class c the calibration set keeps the nonconformity scores s_i of the labelled
calibration samples whose true class is c. A new sample x with nonconformity s_c(x)
for class c gets
p_c(x) = (1 + #{i : y_i = c, s_i >= s_c(x)}) / (n_c + 1)
(build guide section 5, PROTOCOL.md section 4). The comparison is inclusive: a tie counts as
“at least as nonconforming”. A class with no calibration sample has n_c = 0 and gets
p = 1: nothing in the calibration set says the sample is unusual. The credibility of a
sample is the p-value of its predicted class; a low credibility says the sample looks unlike
what the model knows for that class.
Validity (P(p <= eps) <= eps for a sample exchangeable with the calibration samples of its
class) holds only when the calibration samples are a fair draw from the current data. That is
why calibration labels come only from the reserved random slice (recalibrate.py).
Pure numpy; no model code and no labels of the evaluated slot.
- class driftwatch.conformal.mondrian.ClassCalibration(scores: tuple[ndarray, ...])[source]¶
Sorted nonconformity scores of the calibration samples, one array per class.
- classmethod from_scores(own_scores: ndarray, labels: ndarray, n_classes: int = 2) ClassCalibration[source]¶
own_scores[i]is the nonconformity of calibration sampleifor its own label.
- driftwatch.conformal.mondrian.pvalues(calibration: ClassCalibration, test_scores: ndarray) ndarray[source]¶
Mondrian p-values, shape
(n, n_classes).test_scores[:, c]is the nonconformitys_c(x)of each test sample for classc.
- driftwatch.conformal.mondrian.credibility(p: ndarray, predicted: ndarray) ndarray[source]¶
The p-value of the predicted class, per sample.
Pluggable nonconformity scores: s_c(x) is how unlike class c a sample looks.
Two scores (build guide section 5; a research variable, compared in E3):
ProbabilityScore: one minus the model’s probability of the class,s_1 = 1 - P(malware | x)ands_0 = P(malware | x). Needs only the detector’s score.CentroidScore: the Euclidean distance from a sample to the centroid of classcin feature space (after an optional transform such as the detector’s scaler). The centroids are fitted on the current training rows only (hard rule 5); the object carries the window id it was fitted for and refuses to score under another.
Scores take features and probabilities only. Nothing here reads a label of an evaluated
slot; fit receives the labels of the training pool, which is where labels belong.
Drift monitors¶
One alarm interface for every drift monitor (Phase 9, docs/prereg/addendum-v3.md).
A monitor answers one question per evaluation slot: does slot s look unlike the reference
the model in force knows? It is called as monitor.update(view, state) -> Alarm.
SlotViewis everything a monitor may see of slots: the public features, the model’s malware probability, and the features of the calibration rows that have already arrived (with their scores under the model in force). It has no label field, and the runner builds it from the label-freeOpaqueSlotview, so a monitor cannot reach a held-back label (hard rule 4; spy test in tests/test_sim_sealed.py).MonitorStateis the per-(window, seed, arm) state: the reference sample and its slot (rule R, ADR-0010), what the monitor fitted at the start of the window, and a cache of policy-independent statistics.Alarmis the decision with its statistic, p-value and threshold.decidedis false when the monitor cannot test (for example too few calibration rows); such a slot never alarms and is counted.
Every random draw (subsamples, permutations, folds) is seeded by (seed, window, slot,
role) only, so the same pair (reference slot, current slot) gives the same statistic in
every arm that meets it.
Monitors are frozen by addendum v3 and never tuned (CLAUDE.md rule 10).
- class driftwatch.monitors.base.Alarm(monitor: 'str', slot_index: 'int', fired: 'bool', decided: 'bool', statistic: 'float', p_value: 'float', threshold: 'float', ref_slot: 'int', n_ref: 'int', n_cur: 'int')[source]¶
- class driftwatch.monitors.base.Monitor(*args, **kwargs)[source]¶
- start(state: MonitorState, train_X: Any, train_y: ndarray, ref_full_X: Any) None[source]¶
Fit what the monitor needs once per window, from the initial training window only.
- class driftwatch.monitors.base.MonitorState(seed: int, window_key: str, ref_X: Any = None, ref_slot: int = -1, fitted: dict[str, ~typing.Any]=<factory>, cache: dict[str, ~typing.Any]=<factory>, n_undecided: int = 0, n_alarms: int = 0)[source]¶
Per-(window, seed, arm) state of one monitor.
- class driftwatch.monitors.base.SlotView(slot_index: int, slot_id: str, X: Any, proba: ndarray | None = None, ref_proba: ndarray | None = None, cal_X: Any | None = None, cal_proba: ndarray | None = None)[source]¶
What a monitor may see of evaluation slot
slot_index: no labels.
- driftwatch.monitors.base.current_sample(view: SlotView, state: MonitorState, n: int = 500) tuple[ndarray, Any][source]¶
Seeded positions and rows of the current slot (same draw in every arm).
- driftwatch.monitors.base.reference_sample(X: Any, seed: int, window_key: str, ref_slot: int, n: int = 500) Any[source]¶
The seeded reference sample of a slot (or of the initial hold-out,
ref_slot = -1).