After a bad build slipped into litellm’s release channel in March, the team rebuilt the pipeline the way the postmortem guides say to. Releases moved into a private repository. Artifacts go out through GitHub’s trusted publishers, docker release tags are immutable, images are signed with cosign, and publishing is locked to an account tied to one laptop that is physically inside their office. It is a serious list. On October 21 it shipped a credential stealer anyway, through every layer, signed by the project’s own key, from the project’s own release workflow. Downstream, every check a careful consumer knows how to run passed on the poisoned bytes.

That last sentence is the part worth keeping, because it generalizes well past litellm. When the channel is what gets taken, the checks that verify the channel all agree with the attacker.

The credentials came out of the tooling around the channel, though it took two days to say so precisely: the canonical investigation replaced the first status updates’ account, which had the tokens coming from the logs of an internet-facing internal deployment. In May, a configuration file committed to a public repository carried the master key to the team’s agent platform — an orchestration tool that files pull requests, and had been given a GitHub token so it could. The commit was quickly noticed and force-pushed over; the key was marked for revocation; the revocation never happened. The token the platform held was no service account and nothing like scoped. It was the organization owner’s personal GitHub CLI credential. On October 10 the attacker called the platform’s API with the master key still alive from May, and the API handed back the owner’s token in its list of managed-agent configurations. With it they turned the repositories’ own CI into the exfiltration channel: throwaway branches whose workflow runs posted six repository secrets to an outside endpoint, the run records deleted minutes later. Two more maintainers’ GitHub CLI tokens left that way, along with credentials for several other internal systems — the investigation noting a primary focus on upstream inference API tokens. GitHub CLI tokens sit behind two-factor authentication and are exempt from GitHub’s re-authentication prompts for sensitive organization actions, which is the property that made one unscoped credential equal to the whole release channel.

On Monday, release permissions had been temporarily widened so two engineers could publish daily bugfix backports. Tuesday morning at 08:56 UTC, ten commits landed on a repository called Berri-AI/litellm — one hyphen away from the real BerriAI organization — a copy of the project’s own tree with the payload stacked on top. The first three commits added the persistence files. Two later ones, per Wiz’s analysis of the commit series, told the build tool to bundle the malware into the wheel. A final one taught the Dockerfiles to copy it into the image. Nine of the ten carried GitHub’s Verified badge, because pushing from GitHub.com gets you GitHub’s signature for free.

At 10:08:29 a stolen maintainer token tried to push the change that would let the release workflow build from a different source repository. Branch protection rejected it: the change had not passed the required pull-request and status-check process. The system held. Five seconds later the org-owner token ran the same push and overrode the protections. The investigation notes the attacker was managing appearances even here: holding an owner token, they could have relaxed the release rules themselves, and the team believes the smaller stolen tokens were used for the noisier steps so the audit log would read like an ordinary day.

Then the project’s own machinery took over, which is the half I keep sitting with. At 10:46 the modified workflow started an official staging run against the attacker’s source. At 11:08 staging passed. At 11:11:18 a second maintainer’s stolen token satisfied the production approval gate — the gate asked who was requesting, the requesting account was a member, and Monday’s relaxation had made a member enough. At 11:39:51 seven wheels went to PyPI; at 11:46:56 the container publication jobs ran. Between 11:53:49 and 11:54:01 a third stolen token deleted the production and staging run records on the way out. A researcher published the issue that identified the compromise at 12:06:41, under half an hour after the wheels landed, and within the following twenty minutes the attacker overrode branch protection to push a revert and overrode the force-push block to put main back where it started; the visible history showed neither the change nor the revert. PyPI had the wheels down by 13:23, under two hours after they appeared, and ClickPy data put the version’s downloads around 49,000. The registries emptied on three different clocks — GHCR by 13:51, Docker Hub’s floating tags not until 14:20 — the half of the lesson I wrote up at the time as one recall per surface.

So what did the downstream checks say about those bytes? Every one of them said something true.

Pin by digest, and you ask whether these are the exact bytes the channel is serving. They were. A pinned deployment would have held the attacker’s release with total confidence.

Read the source tag, and you find a clean tree. The official v1.107.0 tag was never touched and contains none of the malicious files, so this check passes by describing source that had nothing to do with what shipped. The artifacts were built from a fork, and GitHub serves a fork’s commits under the real repository’s URL, so even the commit link looks right.

Run cosign, and you ask whether the project’s key signed this image. It did. It is the same key the project has signed every image with since v1.83.0, because the project’s own workflow did the building. The project’s own image security guide recommends exactly this walk — verify the signature with the public key published at the release tag, then pin by digest. A consumer following their checklist to the letter arrives at the stealer with every box ticked.

Each of these verifies custody, and custody was never in dispute. The channel served the bytes, the project’s key signed them, the project’s workflow built them. The question none of them ask is the one whose answer was sitting in the open the entire time: BuildKit attaches SLSA provenance to every image build, and provenance names the source repository. The analysis captured on the incident thread shows the v1.107.0 provenance naming Berri-AI/litellm, hyphen and all, where the clean v1.106.2 builds beside it named BerriAI. The registries have since removed the poisoned images, so that record is the thread’s capture rather than something I could re-verify myself. The check itself is one comparison: the source a provenance attestation names, against the name you already expected. It carries caveats — a build can disable provenance, and unverified source fields are client-provided metadata — but here the project’s own workflow published the attestation, and it named the fork.

Whether an environment was actually exposed has a sharper answer than most sweeps reach for, and I ran the mechanics to get it. The payload is two interpreter-startup artifacts, a .pth at the site-packages root and a sitecustomize.py, and a .pth only executes lines that begin with an import — which is why the incident’s cook.pth is a single line, import cook, with all the work in cook.py. On a benign marker image (docker 27.2.1, overlay2): a pull fires nothing. Pull is content transfer and layer unpack, no process from the image starts. Save, remove, load — the same unpack-to-storage step a pull performs — nothing. A run, an exec, a derived build whose RUN step starts a Python: the marker fires, on every interpreter start. Even pip is a Python start, which is why the maintainers’ own remediation warns you not to begin cleanup with pip install. The sweep that matches exposure is interpreter starts — runs, execs, derived builds, CI — not the mirror’s fetch log. The repro is on the thread. And if a start happened in the window, assume the exfil worked: Wiz watched stolen credentials in use four and a half hours after the exfiltration, instance-role credentials from compromised CI runners and Kubernetes nodes, validated with sts:GetCallerIdentity, then a Bedrock model-listing call, the LLMjacking pattern. The stealer also queried the proxy’s own credentials table directly, which is every upstream key the gateway held.

For the next one, three of the four moves that matter already exist in tooling people run.

Compare builder identity, not presence. When the replacement releases landed two days later I checked that all four resolve on PyPI and GitHub with coherent timestamps, and the honest conclusion was that this establishes nothing: presence with coherent timestamps is exactly what v1.107.0 had. What discriminates is the source comparison — the provenance field above, or build the tag from the repository and compare the artifact against what shipped.

Age-gate the install where the tooling has it. The wheels were live under two hours. pip’s --uploaded-prior-to and uv’s --exclude-newer hold installs back to artifacts uploaded before a cutoff, so a one-day gate walks past this entire class of window. Containers have no honest equivalent: org.opencontainers.image.created is written by the builder, and the builder was the attacker. The clock that survives is registry-side push time, which you can enforce at a private mirror.

Move the automation off ambient credentials. The attack failed at branch protection and succeeded at an organization owner, with a token an agent platform held because it needed to file pull requests. GitHub’s own answer shipped this week: enterprise-level GitHub Apps permissions are generally available, expanding what organization automation can do with scoped app credentials instead of classic PATs — fewer places where one stolen token equals the whole release channel.

The hardening after March was not theater. Branch protection did reject the first attempt; the private release repository is why the attacker needed an org owner at all. What the incident prices is the shape that survived it. The credential came out of an agent platform configured with the organization owner’s personal GitHub token, behind a master key that was committed to a public repository in May, marked for revocation, and never revoked. The payload itself reads like agent work — Wiz names the verbose variable names, the generic try/except blocks, a dead code module, Windows paths that fail silently: “functional on the happy path but brittle at the edges and untested.” An agent-shaped leak at one end, agent-written poison at the other, and the oldest thing in the middle: one privileged credential and a force-push. The signature stack never asked about any of it. The instrument that names the builder ships attached to every build, readable before the first install, and the check is one comparison against a name you already had.