Methods · limitations · data handling

How a RESPAI record is computed.

Evidence is only worth what an outsider can check. This page states the methods, thresholds and uncertainty behind every test we run, the provenance that ties a result to its inputs, and the claims a record does and does not make. Last updated October 15, 2026.

Tests

Every test is a published, widely used method. None are proprietary, and all can be re-computed by an auditor from the same inputs. Inputs are the system's decisions (y_pred), the reference labels (y_true), optional scores (y_score), and optional group labels for fairness tests.

Performance

Accuracy is the share of decisions that match the reference label. AUC is the probability that a randomly chosen positive case receives a higher score than a randomly chosen negative case, computed by rank (Mann–Whitney form), which avoids a dependence on any single cut-off.

Fairness

Fairness tests compare outcomes across the groups supplied by the data owner. We report several definitions side by side because they measure different things and can legitimately disagree; a single score would hide that.

impact_ratio = min_g SR(g) / max_g SR(g) SR = selected ÷ group size selection_rate_diff = max_g SR(g) − min_g SR(g) equal_opportunity_diff = max_g TPR(g) − min_g TPR(g) TPR = selected among true positives

The impact ratio is the selection-rate ratio used in the four-fifths rule of the EEOC Uniform Guidelines (§4D) and in the NYC Local Law 144 bias-audit rule (§5-301), which also requires rates and ratios per sex, race/ethnicity and their intersections when the data allows.

Robustness

Perturbation stability re-runs the model on inputs with small Gaussian noise added (default 5% of each feature's standard deviation, five trials) and reports the share of decisions that stay unchanged. A model whose decisions flip under trivial noise is fragile in a way accuracy alone does not reveal. This test needs model access; where only predictions are supplied it is reported as informational.

Data quality

Completeness is the share of non-empty cells in the evaluation set. Class balance is the minority-label share, which flags evaluation sets too one-sided to support the other tests.

Thresholds

Pass/fail thresholds are stated in every record, next to the value. Defaults are below; organizations set their own per jurisdiction and use case before relying on a PASS or FAIL.

Test	Default	Source of the default
Impact ratio	≥ 0.80	Four-fifths rule, EEOC Uniform Guidelines §4D
Selection-rate difference	≤ 0.10	Convention; reported with the ratio, not instead of it
Equal-opportunity difference	≤ 0.10	Convention
Accuracy · AUC	≥ 0.70	Floor for a usable screen; use-case specific
Perturbation stability	≥ 0.90	Convention
Completeness · class balance	≥ 0.98 · ≥ 0.10	Data-adequacy floors

Uncertainty

Rates (accuracy, per-group selection rates) carry 95% Wilson score intervals, which behave well for small groups and rates near 0 or 1. Group sizes are stated so a reader can judge whether a difference is meaningful. Intervals reflect sampling uncertainty only; they say nothing about label error or about how the population has shifted since the data was collected.

Provenance and signing

A record is useful only if it can be tied to exactly what was tested, and only if it cannot be quietly edited afterwards.

Fingerprints. The evaluation dataset, the model and the test code are each hashed (SHA-256). The hashes, not the data, go into the record.
Hash chain. Each test record's ID is the SHA-256 of its content plus the previous record's ID. Editing any record breaks every record after it.
Signature. The complete payload is signed. The browser sample uses an ECDSA P-256 key generated per page load now; production evaluations sign with a persistent RESPAI key (Ed25519) whose public half is published, so a record can be verified years later without contacting us by design.
Run identity. Every record carries schema_version, a run_id, a timestamp and the engine version, so two evaluations of the same system can be compared.

Verification is independent: anyone holding the record and the public key can re-check the chain and signature. The sample page lets you tamper with a value and watch verification fail.

Control mapping

Each test is mapped to the framework clauses that call for the measurement it produces. We use one word for that relationship on purpose: a test supports a control. Whether an obligation is satisfied depends on evidence beyond any one test, on context, and on judgment that belongs to the organization, its auditor or, under the EU AI Act, a notified body where one is required.

Clause references in records point to ISO/IEC 42001:2023 Annex A, the EU AI Act (Regulation 2024/1689) as consolidated, the NIST AI RMF 1.0 and the NYC DCWP rules implementing Local Law 144. Mappings are reviewed when a framework changes; they have not yet been reviewed by outside counsel, and we say so until they have.

What a record claims

A record states

These defined properties were measured on this dataset and model, on this date, by this method.
Here is the value, its uncertainty, and the threshold it was compared to.
Here are the inputs' fingerprints and a signature over all of it.
These framework clauses call for this measurement.

A record does not state

That the system is "safe", "fair" or "compliant" in general.
That any legal obligation is satisfied.
That RESPAI is an independent auditor under NYC Local Law 144 or a notified body under the EU AI Act. It is neither.
That results hold for data the system sees after the evaluation date.

Data handling

We test with the least data needed, and the record keeps fingerprints rather than the data itself.

What	Handling	Status
The sample page	Runs entirely in your browser. Nothing you paste or upload is sent or stored anywhere.	now
Evaluation data in a design-partner engagement	Processed to compute the record, then deleted; evaluations can run inside the partner's own environment so data never leaves it.	by design
The evidence record	Holds SHA-256 fingerprints of dataset, model and code, the metrics and the signature. Not the rows.	now
Group labels for fairness tests	Used only for the test run; never retained beyond the record's per-group counts and rates.	by design
Training on customer data	Never. RESPAI does not train models on evaluation data.	now
Retention, regions and subprocessors	To be published with the first hosted release. Until then, engagements run under a written data-handling agreement.	by design

"Now" means true in what exists today. "By design" means a stated commitment for the hosted product, not yet a shipped guarantee. Website privacy is covered separately at respai.org/privacy.

Known limitations

A test is only as good as its inputs. Group labels, reference labels and the evaluation sample are supplied by the data owner; we fingerprint them so that a weak input cannot pass as a clean result, but we cannot detect categories that were never recorded.
Group-level fairness metrics do not measure individual fairness, and intersectional analysis depends on having enough data in each intersection.
Robustness is tested with additive noise on structured inputs. It does not cover adversarial attacks, prompt-level behavior of language models, or distribution shift.
The current battery is for classification-style decisions. Ranking, regression and generative outputs need methods not yet in the engine.
Evaluations are point-in-time. A system that drifts after the evaluation date needs a new record; comparing records is the purpose of run IDs.

Changes

Methods are versioned with the engine (respai-evidence-engine/0.1). Changes to a method, a default threshold or a mapping are recorded here with a date, and the engine version in every record says which rules applied when it was produced.

2026-10-15 · First published methods. Schema 1.0: run IDs, Wilson intervals, caveats and the supports-vs-satisfies relation added to records.