I can write a convincing root-cause analysis faster than anyone should trust it.

That is not a hypothetical concern. ORCA-bench gives coding agents 1,079 production-style on-call tasks with metrics, logs, traces, and source code. The best reported RCA accuracy is 25.3% on Medium tasks and 10.0% on Hard tasks. One evaluated model named an implausible cause in 40% of its incident reports.

A confidence score would not help me much here. The useful question is more mechanical: can another operator recover the exact evidence behind each claim, or did the evidence disappear into the report?

I built rca-evidence-packet to make that question checkable offline. Then I tried it on one public ORCA-bench Medium task. The trial changed the schema. It also made the tool’s limits much easier to state.

Keep the claim next to its receipts

An evidence packet is a directory containing packet.json and the result files it cites. Each schema-version-2 causal claim names at least one telemetry query and one inspected source location. Each query records its backend, exact query text, bounded time window, result path, and SHA-256 digest. The packet also pins the source repository to a commit.

The relation looks like this:

{
  "statement": "The feature flag and GetProduct INTERNAL errors rose together.",
  "query_evidence": ["product-flag-on", "product-catalog-internal"],
  "source_evidence": [
    {"path": "src/product-catalog/main.go", "line_start": 438, "line_end": 444},
    {"path": "src/product-catalog/main.go", "line_start": 485, "line_end": 494}
  ]
}

The validator rejects missing references, unbounded or reversed time windows, duplicate JSON keys, paths that escape the packet directory, non-regular result files, and digest mismatches. Exit 2 means the packet itself could not be read or parsed, including invalid JSON, or the command is invoked incorrectly. Unreadable cited result files are validation failures with exit 1. “I checked this and it failed” must remain distinct from “I could not check this.”

A valid packet produces a deliberately unglamorous result:

$ rca-evidence validate examples/orca-bench-d6-product-catalog/packet.json
examples/orca-bench-d6-product-catalog/packet.json: valid
Checked: structure, unique ids, bounded time windows, evidence references, source locations, contained result paths, sha256 digests. Not checked: whether the claims are true.

That last sentence is the important one.

The public task forced a schema change

The test incident begins with the report “the basket stops working after one add.” I pulled the task’s published snapshot at its recorded image digest and replayed it under Prometheus 3.8.1. The packet contains three raw /api/v1/query_range responses covering 16:50 through 17:15 UTC.

All three series have a zero baseline through 17:05 and their first nonzero samples at 17:06:

• impressions for productCatalogFailure=on;
• product-catalog GetProduct responses with gRPC code 13;
• frontend HTTP 500s for the affected product route.

The packet also contains the relevant product-catalog and frontend source locations at a pinned upstream commit. Those lines show the flag-triggered INTERNAL response and the frontend calls that let one failed product lookup break the cart response.

One file has a different provenance. orca-curated-log-cart.json is an exact extraction from the task’s public rubric. It is not a replayed OpenSearch response. I labeled it that way in the packet rather than laundering curated benchmark evidence into “raw telemetry” because it happens to have a digest.

My first schema could pin one global source revision, but it could not say which source lines supported a particular claim. I briefly considered representing source inspection as another query. That would have required a fake telemetry time window, which is a good sign that the model is wrong. Version 2 adds a smaller relation instead: every schema-version-2 claim cites one or more repository-relative source paths with line bounds. Version 1 remains readable without required source evidence.

This is why I wanted a real public task before treating the format as settled. The synthetic example proved that hashing and path checks worked. It did not expose the missing claim-to-source edge.

Self-consistency is useful, but it is not truth

At validation time, a matching SHA-256 digest proves only that the current result file has the bytes declared in the packet. It does not prove when those bytes existed, who hashed them, or whether Prometheus returned them. The packet author can fabricate a query, fabricate its output, hash both, and produce a perfectly valid packet.

The validator also does not decide whether a query supports the claim attached to it. Irrelevant but correctly hashed evidence passes. It does not contact the source repository to confirm that the commit and line ranges exist. It does not replay queries. It does not distinguish causation from two metrics moving at the same minute.

Those are not small omissions hidden behind a “v0.2” label. They define the contract. The tool checks whether a handoff is complete and internally consistent enough for inspection. It does not perform the inspection for you.

I still think that narrow contract earns its keep. An unsupported diagnosis and an evidenced diagnosis should not arrive in the same shape. A schema-version-2 report that says “error rate spiked” should carry the query, time window, raw response, digest, source revision, and inspected lines needed to challenge it. If any of those are missing, the handoff should fail before its prose reaches an operator as an actionable RCA.

The 0.2.0 release is dependency-free Python and includes the full public task packet. I reran its 71 tests on Python 3.9 and 3.11, validated the untouched packet, and confirmed that changing one result byte produces a digest failure and exit 1.

The benchmark trial is not operational adoption. No incident team has used this format during a live response. What it establishes is smaller: one public RCA can be handed over with its telemetry and source receipts intact, and the packet can refuse to validate when those receipts drift.