What can be checked about SIR.

Paired runs, signed certificates, preserved ledgers, and links to the artefacts behind them. These benchmarks are not a leaderboard and not an intelligence ranking surface.

Failing runs are published, not withheld. A run that fails or cannot reach a conclusion is preserved and published in the same form as one that passes. The run archive carries its own generated counters, so the totals shown there are current rather than transcribed onto this page and left to go stale.

Paired runs

Benchmarks

Each row compares the same prompt set without and with SIR in front of the model. The Outcome column checks whether runtime evidence matches the uncovered rows already published in coverage data. Only paired benchmark dispatches appear here. The routine audit published on every merge has no ungated half, so it produces no row in this table; those are published separately in the run archive.

Snapshot as at 1 November 2026.

Run time (UTC)	Model	SIR	Pack	Coverage	Leaks (baseline / gated)	Outcome	Evidence
2026-10-15 10:32:04Z	grok-4.3xai	2.4.0	eu_ai_act_compliance_pressurebaseline UNGATED_BASELINE_RECORD / gated LIVE_GATING_CHECK	74/100	n/a / 26 of 150	Leaks match published uncovered rows exactlyBlocked benign: 0	CI Base Gate
2026-10-15 09:31:50Z	grok-4.3xai	2.4.0	data_exfiltration_pressurebaseline UNGATED_BASELINE_RECORD / gated LIVE_GATING_CHECK	27/27	n/a / 0 of 50	Complete coverage; no leaked rowsBlocked benign: 0	CI Base Gate
2026-09-23 12:14:40Z	grok-4.3xai	2.3.4	data_exfiltration_pressureLIVE_GATING_CHECK	27/27	27 / 0 of 50	Complete coverage; no leaked rowsBlocked benign: 0	CI Base Gate
2026-09-21 13:51:40Z	gpt-5.4-miniopenai	2.3.4	eu_ai_act_compliance_pressureLIVE_GATING_CHECK	74/100	100 / 26 of 150	Leaks match published uncovered rows exactlyBlocked benign: 0	CI Base Gate
2026-09-21 12:03:18Z	gpt-5.4-miniopenai	2.3.3	eu_ai_act_compliance_pressureLIVE_GATING_CHECK	74/100	100 / 26 of 150	Leaks match published uncovered rows exactlyBlocked benign: 0	CI Base Gate
2026-09-20 09:24:56Z	gpt-5.4-miniopenai	2.3.0	eu_ai_act_compliance_pressureLIVE_GATING_CHECK	74/100	100 / 26 of 150	Leaks match published uncovered rows exactlyBlocked benign: 0	CI Base Gate

The original benchmark URL remains available at /benchmarks/.

Signed proof

Verification surfaces

Human-readable

Latest audit

Current published audit certificate.

Open latest-audit.html

Run archive

Published run history and evidence links.

Open run archive

Tooling

Verifier script

Offline certificate verification code.

Open verifier

Independent verification command for gated run 20261015-092939-820950-gh38712792514-308a8a0f91eb (proof class: LIVE_GATING_CHECK):

git clone https://github.com/SDL-HQ/sir-firewall.git \
  || git -C sir-firewall pull
cd sir-firewall
python3 -m pip install cryptography
python3 tools/verify_evidence.py docs/runs/20261015-092939-820950-gh38712792514-308a8a0f91eb

docs/runs/20261015-092939-820950-gh38712792514-308a8a0f91eb

  OK      signing trust      signing_key_id='sdl-2026-10-06' resolved through key_registry.v1.json, with its status and revocation rules applied
  OK      certificate        signature, payload hash and ledger binding all verify
  OK      signed counters    the signed counters were recomputed from the rows the certificate binds
  OK      archive custody    every file named by the signed manifest is present and unchanged
  OK      evidence contract  the certificate satisfies its applicable evidence contract

VERDICT: ESTABLISHED

This is one archive, checked by our own tool with our own key: a signature that verifies establishes that we signed that payload, not that the payload is true. Where a property cannot be checked at all the verdict is NOT ESTABLISHED rather than a pass, and several published archives come back that way. Limits records what a passing verification does not establish, and the claims register lists every claim with the evidence for it.

Don't take our word for it

Check the evidence yourself

A separate repository of real published archives, each chosen because our own verifiers give it a different answer. Verifiable offline from a clean machine, with a recipe for altering a ledger row so you can watch the chain refuse to re-derive.

Open SDL-HQ/sir-evidence

Reading the archive

Certificates from SIR 2.3.4 onward bind to their ledger, and the command above verifies one of them. Certificates published before 2.3.4 carry no run identifier, so a verifier holding only the certificate cannot resolve which ledger is theirs and returns exit 9, binding not checked. That is the intended fail-closed result for historical evidence, not a fault in the verifier.

The full position is whatever tools/archive_verification_report.py prints from a clone. It walks every certificate published under docs/runs/ and reports, per exit code, how many bind their ledger, how many predate the fields a binding needs, and how many fail signature verification. That is where the figures on this page used to come from, and reading them from the generator rather than from this page is the difference between a number you can check and a number you have to trust. The archives that do not verify, and the reason for each, are listed in the archive errata.

The position as it stood at the SIR 2.3.5 release, broken down per exit code across the 284 certificates published by then, is recorded in the SIR 2.3.5 release notes. It is kept as a dated record rather than as the current figure, which the generator above reports.

Negative conformance

Certificates that must be rejected.

The repository publishes five deliberately invalid certificates together with the rejection each one must produce: a falsified leak count, the same falsification with the payload hash recomputed, a signature copied from a different certificate, an unregistered signing key, and a removed required field.

Four of the five pass contract validation. Only the cryptographic check rejects them. That contrast is the point. A certificate can be structurally valid and still not be authentic, so shape and authenticity are separate questions and both have to be asked.

A verifier that is not ours conforms when it rejects all five inputs, distinguishes a payload-hash failure from a signature failure, rejects an unknown signing key, and leaves required-field validation to the contract validator.

Read the negative conformance suite

Test the work

Bring a bounded AI workflow and a real governance question. A controlled pilot tests whether SIR's gates and signed evidence are useful under operating conditions.

Test SIR in a controlled pilot