AI Assurance · Test, Evaluation, Verification & Validation

Would you field this model?

OAK Assure is a TEVV workbench for defense-AI. It grades a classifier against a known truth the way a reliable-AI programme needs before fielding — performance, calibration, conformal coverage, adversarial robustness and input drift — and returns a PASS / FAIL verdict with a per-gate table. Not just numbers: a verdict, with the evidence behind it.

Request a demo What it measures

UNCLASSIFIED · representational // T&E tooling — produces assurance evidence, not an accreditation

What it measures

Five dimensions, five gates, one verdict

Each dimension is scored and gated against a configurable threshold. The result is a per-gate PASS / FAIL — illustrative defaults, set per programme.

Dimension	Metric	Gate (default)
Performance	clean accuracy + per-class precision / recall / F1 + confusion	accuracy ≥ 0.80
Calibration	Expected Calibration Error, Brier score, reliability curve	ECE ≤ 0.10
Conformal coverage	split-conformal prediction-set coverage & average set size	no under-coverage
Adversarial robustness	accuracy on the adversarial split — spoof / decoy / noise	drop ≤ 0.25
Input drift	Population Stability Index, train vs test	mean PSI ≤ 0.25

Validated on real data — not just synthetic

Grades a model trained on real public data

OAK Assure's five-gate battery is proven on a model trained on a real public dataset — not only the suite's synthetic records — the credibility step from demonstrator toward relevant-environment validation.

Real acoustic model

On the DroneAudioDataset — 360 drone + 600 background real clips — Assure grades a drone-vs-background detector across all five gates: accuracy, calibration, conformal coverage, adversarial robustness and drift.

Full battery, real verdict

The real-data model returns a clean PASS (acc 0.85 · ECE 0.048 · coverage 0.90 · robustness-drop 0.02 · drift 0.04), and the harden-and-re-grade path holds the verdict on real audio.

A genuine gate

On a smaller eval the same model sits on the calibration line and Assure correctly FAILs it — the gate catches a marginally-miscalibrated real model, proving the verdict is real, not a rubber stamp.

TEVV Workbench

Load, grade, and read the verdict

Load or generate a dataset, pick a model and target, run the assessment
A PASS / FAIL verdict with the per-gate table and per-perturbation robustness
Reliability & robustness diagnostics; compare all models side by side
Export a self-contained HTML assurance report — evidence behind a reliable-AI / TRL claim
A headless --report mode for CI / scheduled assurance runs

OAK Assure — TEVV workbench: PASS verdict with the per-gate table

Close the loop

Harden a borderline model — and re-grade it

OAK Assure doesn't only grade a model; it can harden it and re-grade, honestly, so a borderline FAIL becomes a defensible PASS.

Calibrate — post-hoc temperature scaling on a held-out split. Monotonic, so clean accuracy and the argmax are unchanged; it only honestens the confidences → lower ECE
Augment — robust training on decoy / noise / spoof-style perturbations of the training split (never the held-out test split) → higher adversarial robustness
A worked example: a naïve model FAILs the calibration gate and is weak to decoys; hardening drives ECE and decoy-robustness back into gate — with clean accuracy, coverage and drift preserved

Reliable-AI, end to end

The other half of the story

OAK Assure is the second half of the reliable-AI loop: OAK Synthetic Environment makes the labelled data — with a known truth and an adversarial split — and this harness grades the model on it. The sim makes the data; the harness grades the AI.

It reuses the EW SUITE's AI libraries (ai_core classifiers, ai_calibration metrics) and reads any JSONL with the SynthEnv record schema. It produces assurance evidence; it is not an accreditation.