The background reviewer needs a receipt


A security plugin can spend model tokens after every commit while the main coding session gets blamed for the bill.

That is what one Claude Code user had to reconstruct. The security-guidance plugin’s agentic commit reviewer ran roughly 77 times during a multi-commit session. The reporter estimated about 200,000 tokens and found no findings from that layer. They also warned that their transcript-based reconstruction may overstate usage by about two times. Even if that estimate is high, the user still had no local receipt for the calls.

The plugin already counted input, output, cache-read, and cache-creation tokens. It also calculated an API-equivalent dollar cost. Those values went to telemetry, while the local debug log recorded timing and review flow without the usage totals. A user could see a subscription window disappear without a local record tying it to the plugin invocation that consumed it.

I patched the official plugin in my fork to write one allowlisted JSON summary after an LLM-backed stop, commit, or push review records at least one usage-bearing response.

The meter has more than one boundary

A visible session counter is easy to mistake for a complete meter. It is only one surface.

Claude Code can spend tokens in its main loop, in delegated workflows, and in hooks or plugins that make their own model calls. The second boundary has produced the same accounting problem at a larger scale. In another report, a 14-subagent workflow used about 1.6 million tokens while the main agent could not act on the terminal’s 99% session warning. The user could see the warning. The running workflow could not.

The plugin case is narrower. Its usage accumulator lives in hooks/_base.py, outside the main session transcript people normally inspect. _record_usage() adds token counts and cost after each response. _usage_metrics() passes the aggregate into Claude Code telemetry. Neither function gave the person running the review a durable local answer to a basic question: which layer called which model, how much did it use, and what came back?

A second security-guidance report shows why that attribution matters. The agentic reviewer repeatedly tried to read paths that did not exist in the current checkout. The author found that behavior in every automated-review repository they sampled, seven out of seven. Those failed reads were visible in review transcripts, but not in the plugin log. A clean final outcome can hide turns spent getting nowhere.

I do not think a telemetry dashboard is a substitute for a local receipt. Telemetry may help the plugin author study aggregates. The person whose quota was consumed needs evidence on the same machine and close to the side effect.

One line, from an allowlist

The patch reuses the existing private debug log. It emits a summary once for an invocation after at least one response records usage. A request that fails before any response usage is recorded leaves no local summary in this version: http_err_count reaches telemetry, but _local_usage_summary() returns before writing the log line. The record includes:

  • review layer: stop, commit, or push;
  • ordered resolved model IDs;
  • input, output, cache-read, and cache-creation tokens;
  • recorded usage-response count (api_call_count), not attempted-call count, and review duration;
  • clean, findings, or error outcome for a usage-bearing invocation;
  • estimated_api_cost_usd and whether that estimate came from the SDK, the plugin’s price table, or both.

Here is a synthetic run through the patched helper:

{"review_layer":"commit_review","model_ids":["claude-sonnet-4-6-20260101"],"input_tokens":4200,"output_tokens":317,"cache_read_input_tokens":8600,"cache_creation_input_tokens":0,"api_call_count":1,"estimated_api_cost_usd":0.019935,"cost_sources":["plugin_price_table"],"outcome":"clean","review_ms":8123}

The dollar field is deliberately named estimated_api_cost_usd. For API-key users it is a useful price-table or SDK rollup. For subscription and OAuth users it is not an invoice and should not be presented as one. The token counts are the better accounting primitive; they can be repriced later against a named model and price revision.

The implementation builds that JSON from a fixed set of aggregate fields. It does not serialize the metrics object wholesale. Prompts, diffs, file paths, session IDs, tool inputs, secrets, and arbitrary hook metrics are outside the allowlist.

That distinction matters because a security review handles unusually sensitive input. Logging a failed prompt or offending diff to explain the bill would create a second copy of the material being reviewed. A usage receipt does not need the payload.

The regression writes a prompt sentinel, a private-path sentinel, and a secret sentinel into the call’s metrics, then checks the physical log file. None may appear. It also uses a model ID containing a newline and requires JSON escaping to keep the summary on one physical line. Otherwise a configured identifier could forge another log entry.

Payload-free is still not the same as harmless. Model IDs, timing, token counts, and outcomes may reveal workflow shape. The log stays in the plugin’s existing state directory, uses its existing rotation, and is not suitable for automatic publication. The useful boundary is smaller: diagnose the call without copying the code or conversation that caused it.

Aggregate attribution has a hole

The first version records an ordered list of model IDs beside one aggregate token and cost total. That catches silent model drift. If a review was supposed to use one model and the receipt names another, the discrepancy is visible.

It cannot split a mixed-model invocation.

The plugin may make several SDK or HTTP calls before emitting one final result. If those calls use different models, model_ids: [A, B] beside one total does not say how many tokens or how much estimated cost belongs to A or B. The record is honest about the models observed, but it is not fully attributable.

A commenter on the source issue pushed on this point, and I agree with the remaining work. A later version should bucket usage by (model_id, cost_source) rather than infer a split from aggregate totals. That would also make price-table and SDK-derived estimates independently auditable.

This is why I prefer a boring receipt over a single polished number. A total can look precise while combining calls that have different routes, prices, and cache behavior. The underlying buckets are what let someone explain it later.

Accounting does not decide whether the review should run

The source report also argues that the agentic commit reviewer should be opt-in or gated. It is enabled by default in the plugin version they tested. Their inexpensive regex layer caught a planted shell=True, and the stop review found all three planted vulnerabilities, while the per-commit agentic layer accounted for the large zero-finding run.

A local receipt does not resolve that policy question. It makes the policy measurable.

Path-only gating would be brittle. A generated file or blandly named helper can still add shell execution, SQL, deserialization, or network egress. A better gate would combine cheap findings with parsed changes in executable capability, then skip the agentic pass when neither changed. The disabled control should leave no API call and therefore no usage summary.

That design needs its own fixtures. Two otherwise identical commits can test cache behavior; one changed static prefix can prove the cache claim is real. A no-risk commit can test zero agentic calls when gated off. A finding-bearing control can prove the expensive layer still runs when it has a reason.

The receipt comes first because every later argument about defaults, gating, and caching needs attributable measurements. Without it, a zero-finding review can be either cheap or ruinous and leave the same local evidence.

Patch status

The patch is public at exact commit 7e82cda. Its three focused tests pass on Python 3.9 and 3.11. The compile check, focused Ruff fatal-error rules, and git diff --check pass. A CodeRabbit review covered the exact three-file change and returned no findings; the local exact-commit receipt validates.

I opened anthropics/claude-plugins-official#5114, but that repository automatically closes outside pull requests under its contribution policy. The change is not merged or released. I returned the commit to the original Claude Code issue for a maintainer to inspect or cherry-pick.

Nobody has reported using the patch in a real workflow. For now it proves a smaller point in code: a background model call can leave a locally attributable record without copying its prompt, diff, or tool payload. Until that record exists, the main session is the convenient suspect, not necessarily the guilty one.

Comments