Results explorer
Classifier metrics cover only fields that actually received a candidate level. Workflow metrics describe what an operator experiences. The two are never mixed: an unmatched field carries interim handling, and counting that as a prediction of "Restricted" would inflate both.
The metrics, in plain terms
- Coverage
- the share of fields the engine is willing to label
- Agreement
- how often a label it does issue matches the reference
- Selective agreement
- the same, among only the labels no human is asked to re-check
- Gate rate
- the share of fields a human must look at
- Safe-handling
- the share that end up either correct or in front of a human
- Silent error
- a wrong label nobody was asked to review — the one outcome the design exists to prevent
- Interim agreement
- among abstentions, how often §1.4's Restricted default happens to equal the reference — workflow safety, not classification accuracy
Reference security levels are author-assigned throughout. Agreement figures are internal consistency with the authors' reading of policy, not independently validated accuracy. The paper's principal limitation, stated in Section IV-F, is exactly this; the blind multi-expert re-labelling that would address it is specified there as the next study.
Conformance corpus and baselines (Table II, n = 200)
A synthetic corpus built from the rule set's own vocabulary, so it measures conformance to the rules, not independent validity. The baselines never abstain, so their 100% coverage is by construction, not capability.
Ablations (Table III)
Vocabulary held fixed, one design element varied at a time. The impact vector contributes no predictive power — the level is a pure function of it — but it buys a replayable rationale. The safety architecture is what suppresses silent error. The substring row is indistinguishable here and 5/19 on the adversarial probe.
Gate-threshold sensitivity
The gate compares each rule's hand-assigned assurance value to a cut τ = 0.75. Sweeping τ across the whole assurance range answers whether the reported safety depends on those hand-picked values: the gate rate moves from 22.5% to 100%, and silent under-classification stays at zero at every τ on both corpora. The property is structural — the Top Secret, §1.4-interim and PDPL-name-candidate triggers never consult assurance — so the values tune reviewer workload, not safety.
Out-of-sample corpora (Table IV)
Three corpora the rule set was not built on. Every rule that fires agrees with the reference label on every set; coverage is the binding limit. The human-review column is the operational cost of that safety: abstentions plus gated candidates.
On unfamiliar vocabulary the artifact is a triage and audit aid, not an automation win. Abstention keeps it safe but must not be read as capability: a deployment should fix explicit targets for acceptable coverage, gate rate and reviewer workload — and measure analyst minutes per gated field, which the paper has not — before the approach is judged operationally useful.
External test: Apache Fineract, field by field
Sixty column names transcribed verbatim from the published core-table schema, classified name-only with the rule set frozen, evaluated once. Rows where a rule fired are highlighted.
Independent PDPL flag evaluation (Section IV-E)
Two annotators — a data engineer and a data scientist, neither a legal practitioner — labelled 123 fields blind to engine output and to each other. Their labels are practitioner readings of the statute, not legal opinions. Agreement was computed before adjudication; the engine was run only afterwards. Every cell below is produced by the study's own scorer (pdpl_flag_study/score_flags.py) at build time, not transcribed: κ on the agreement set, precision and recall on the scoring set.
The Credit Data limb missed an entire statutory category. The paper reports this rather than repairing it: bare loan and credit hints would recover only 3 of the 18, and fitting a repair to these labels would spend the only independent Credit evidence there is. A redesigned detector using table names, schema relationships and descriptions, evaluated on a fresh independent test set, is named as next work.
Full study package — guide, both blind label sets, adjudication and scorer — in pdpl_flag_study/ in the artifact.