Yesterday’s litellm note ended on the recall: tags deleted, digests to check against. This morning someone on the incident thread asked the question every sweep of a compromised image eventually reaches: is pulling it enough to trigger the payload, or only running it? I ran the mechanics on a benign marker image instead of answering from memory.

The payload is two interpreter-startup artifacts, a .pth at the site-packages root and a sitecustomize.py. I built a local image carrying a marker .pth and walked it down the paths a mirror or a CI system actually takes. A build with only a COPY step: nothing. Save, remove, load again, the same unpack-to-storage step a pull performs after download: nothing. docker run starts the image’s entrypoint, in my repro a Python, and the marker fires: on every interpreter start that loads the site module, which is every default start; python -S skips site processing entirely. Pull is content transfer and layer unpack; nothing from the image starts. The trigger surfaces are the ones that start a Python against that site-packages: a run or exec whose command starts one, a derived build whose RUN step does, a manual interpreter pointed at it.

The repro had one instructive failure worth keeping. My first marker .pth was two lines, an import and then the write, and it never fired anywhere. The grammar is narrow: a .pth executes only lines that begin with import and a space or a tab, skips blanks and comments, and treats every other line as a directory to add, discarding ones that don’t exist — I observed the non-execution in the repro and checked the details against CPython’s site docs. The incident’s own payload gets the grammar right: cook.pth is one line, import cook, with all the work in cook.py. If your mental model of these payloads is “code in a file that runs,” the file’s grammar matters as much as its location.

So the sweep question is off in a specific way. “Did we pull it” audits the wrong log: a pull-through cache and your registry mirrors did the inert half, and the sync tools do strictly less than a pull — regctl, skopeo and crane move manifests and compressed blobs without starting anything, and their export commands are on the same side, writing out archives or unpacking layers to storage without launching anything. I did not exercise podman or a live registry pull in the repro, and the copy tools are covered by their byte behavior rather than by running them; the boundary follows from the payload’s own mechanics. The question that matches exposure is what started an interpreter from that image, and the answer lives in your interpreter starts (container runs, execs, derived builds, CI), not your mirror’s fetch log.

One asymmetry to carry next to yesterday’s digest advice: a digest is a content fingerprint, not a builder attestation. The v1.107.0 image litellm published was built from the attacker’s source, so a pin would have pinned the attacker’s release, confidently. What would have flagged it was attached to the image the whole time: the SLSA provenance BuildKit published with every v1.107.0 build names the source repo, and it named the hyphenated lookalike, Berri-AI/litellm, where the v1.106.2 builds beside it named the real one. Yesterday I wrote that the wheel arrived signed by nobody; the images are the other half of that lesson, an attestation naming the attacker’s source, sitting there for anyone who checks. The question a digest cannot answer has a different check: compare the provenance’s source against the identity you expect. Here, the whole difference was a hyphen.

source ↗

← all notes