Section IV of the paper · every figure read from the artifact's report_all.json

Results explorer

Abstention-aware, safety-oriented metrics. Annex I product-safety cases are excluded throughout: the engine does not attempt that route and flags it on every record, so scoring them would measure a determination it declines to make.

The metrics, in plain terms

Correct-or-referred: correct, or escalated to review — it measures routing, not successful resolution, since the human stage is unevaluated
Silent: under-classified and not gated — the unsafe direction, any tier
Severe silent: the subset whose reference is high-risk or prohibited
Gate rate: share of systems escalated to a human (confirmation or review)
Coverage: share decided, i.e. not INDETERMINATE
Exact: exact-tier agreement over all systems

Reference labels are author-assigned; the draft-guideline labels inherit that draft's non-binding, interpretive status; the real-system set is small and author-selected; the evaluation set is development-influenced, hence the post-freeze primary. The engine has no run-to-run variance; benchmark-sampling uncertainty remains, and is reported as Wilson intervals.

Per set (Table II)

Hardening ablation

Cycle 2's two components — synonym normalisation and the persons-net — cut compliance-washing slip-through and post-freeze silent under-classification. The post-freeze change is modest and not statistically distinguishable from chance: two cases improved, none worsened, exact McNemar p = 0.50, and the intervals overlap heavily. The paper reports it as a direction, not a demonstrated effect.

Silent under-classification across the two hardening cycles on the evaluation and adversarial sets. — Silent under-classification across the two hardening cycles on the evaluation (n = 86) and adversarial compliance-washing sets — the artifact's own figure, generated from report_all.json.

Post-freeze confusion matrix (decided cases, Cycle 2)

Rows are reference tiers, columns the engine's decided tier; INDETERMINATE outcomes are abstentions and sit outside the matrix.

Two-annotator reproducibility check (Section IV-D of the paper)

Two annotators — a data engineer and a data scientist, neither a legal practitioner — independently labelled 57 system descriptions (tier, provision, Article 50 set) from a written guide, blind to the engine and to each other; agreement was computed before adjudication and the engine run only after the agreed labels were fixed.

Measure	Result	Reading
Tier agreement between annotators	56 / 57 · κ = 0.971	wherever both named a provision they named the same one (28 / 28)
Author's tier labels reproduced on the 52 agreed units	50 / 52 (96.2%)	Wilson 95% CI 87–99%; both exceptions the author tiered stricter (high) than the annotators (minimal)
Engine on the same 52 units, author's labels vs agreed labels	identical	16 / 52 silent, 35 / 52 safely handled under either — whose labels are used changes nothing about how the engine scores
Unresolved rows	5	deliberately not adjudicated; four turn on whether Article 50 duties survive an Article 5 prohibition, which the Regulation does not settle

Two limits bound this check, in the paper's own words: the annotators are practitioners, so agreement establishes reproducibility, not legal correctness; and the guide as issued carried an Article 6(3) instruction error that could only have pushed labels towards high, so these are round-1 estimates, exposed on the 19 units both labelled high and to be re-run first under the corrected guide. The study package — both guides, scorer, agreed labels, unresolved rows — ships with the artifact.

Penalty-weighted exposure

Illustrative weights proxying the Act's Art. 99 penalty asymmetry — a silent prohibited miss costs far more than a gated one or an over-tier. The claim of record is exactly the paper's: lower engine exposure than the keyword baseline in all four tested scenarios, not a liability estimate.