LiteLLM Supply Chain Attack: Could Your Pipeline Do This?

On 21 October somebody ran pip install litellm and got a credential stealer. Not a typosquat with a transposed letter. The real package name, the real project, the real Docker tags.

The malware is the least interesting part of the LiteLLM supply chain attack. Nobody broke PyPI and nobody broke GitHub. A configuration file holding a master key sat in a public repository from May, the key was never revoked, and five months later it led to a GitHub token belonging to an organisation owner. An organisation owner can switch off the branch protection that was supposed to make a release reviewable.

So before you spend the morning grepping lockfiles, go and read the list of people who can bypass the rules on your own release repository. That list is longer than you think, and it is the whole attack.

What did the LiteLLM supply chain attack actually ship?

Three files into the Python site-packages directory. CERT-In's advisory of 23 October names them as cook.pth, cook.py and sitecustomize.py, and the choice of filenames is the clever part.

Neither of the two active files needs anybody to import litellm. Python's documentation for the site module is explicit that in a .pth file, "lines starting with import (followed by space or tab) are executed", and that afterwards "an attempt is made to import a module named sitecustomize". Both run during interpreter start-up. Install the wheel once in a build step and every python in that image afterwards is carrying the payload.

The sweep is broad. CERT-In lists browser credentials, SSH and GPG keys, AWS, Azure and Google Cloud credentials, environment variables, CI/CD secrets and tokens for AI tooling, with the sitecustomize.py component establishing "a mechanism for retrieving and executing additional Python code".

The exposure window is published. The project's own incident post puts the malicious wheels on PyPI from 11:39 to 13:23 UTC, the Docker Hub tags v1.107.0, latest, main-latest and main-stable from 11:46 until 14:20, and GHCR clean by 13:51. The clean pin is 1.106.2.

How does a key leaked in May become a release in October?

Slowly, and in public. The attack-chain write-up starts on 27 May, when a configuration file containing the "master key to a deployment of the LiteLLM Agent Platform" was committed to a public repository. It was removed quickly. It was never revoked.

On 10 October that key was used against the platform's own API, where one list call "returned the configuration of every managed agent". One of those configurations held a GitHub CLI OAuth token belonging to an organisation owner. Throwaway workflows then exfiltrated six further secrets, including tokens from two more maintainers.

Eleven days after that, at 10:08:34, "the attacker disables branch protections and then force-pushes with the owner's token". The release that followed was built from a fork called Berri-AI/litellm rather than BerriAI/litellm, at a commit that existed only on the attacker's copy. One hyphen.

The line in that write-up worth pinning above a desk is about neither tokens nor hyphens. Release authorisation had been "relaxed earlier this week to accommodate the temporary push for more frequent releases", so that any two engineers could trigger and approve a release instead of the previous requirement for a special account locked to a physical device. The control existed. It was traded for release cadence, nine days before it was needed.

An internal platform whose API hands back every agent's configuration, credentials included, is the same shape of problem as the one we wrote about when agents edited a wiki unnoticed for three months. In both cases nobody had enumerated what the automation could reach.

Who can bypass the rules on your release repository?

Go and find out, because hardly anybody knows the answer from memory. GitHub's rulesets documentation says that when you create a ruleset "you can allow certain users to bypass the rules", and that bypass can go to "users with a certain role, such as repository administrator, or it can be specific teams or GitHub Apps".

Two traps sit under that. The first is layering. Rulesets and the older branch protection rules "work alongside each other, and all applicable rules are enforced", and "multiple rulesets can apply to the same branch at the same time, while only one branch protection rule applies". A repository halfway through a migration between the two models has an effective policy nobody has read end to end.

The second is where the token came from. GitHub's page on OAuth app access restrictions warns that without them "any OAuth app authorized by an organization member can also access the organization's private resources". A gh CLI token in a log file is not a personal convenience. It is an organisation-scoped credential carrying the reach of whoever created it, and it does not expire on its own.

We have made the cheaper version of this mistake, in the workflow layer rather than the identity layer, which is the ground the actions/checkout change covered.

Does a two-hour window mean you got away with it?

Probably, and "probably" is not something you can write in a report. Three ways two hours reaches you anyway.

• Scheduled builds. A nightly image build that landed between 11:39 and 13:23 UTC pulled it with no human present.
• Floating tags. latest and main-stable were affected, so any deployment resolving a tag at pull time was eligible.
• Caches. Removing a wheel from PyPI does not remove it from your layer cache, your CI cache or your internal mirror. Those keep serving the artefact long after the index has stopped.

The stealer also finished its work in the first second of the first interpreter start. So the question is not whether the file is still on disk. It is what that host could reach: the instance role, the provider keys, the database credentials, the environment block. The blast radius arithmetic is the one we walked through for an instance role leaving the box through the SSM Agent. Rotate on the assumption that it ran, then narrow with logs. Not the other way round.

What does CERT-In expect you to have done by now?

Report it, if it reached you. The directions of 28 April 2022 require covered entities to "mandatorily report cyber incidents as mentioned in Annexure I to CERT-In within 6 hours of noticing such incidents or being brought to notice about such incidents". Annexure I carries "Malicious code attacks such as spreading of virus/worm/Trojan/Bots/Spyware/Ransomware/Cryptominers" at item v and "Unauthorised access of IT systems/data" at item iii.

The phrase doing the work is "or being brought to notice". A published advisory is a notice. Six hours is not counted from 21 October for everyone; it starts when you find 1.107.0 in a manifest, which is why the grep matters more than the calendar does. What meeting that clock actually takes is in our post on the six hours.

Scoping needs the other direction. The same text requires logs of all ICT systems, "maintained securely for a rolling period of 180 days and the same shall be maintained within the Indian jurisdiction". If you cannot list which hosts and which build agents installed that wheel, you cannot scope the rotation, and you will rotate everything in a very bad week.

What can you run on Monday?

Three jobs, about ninety minutes between them.

Find it first. Manifests, then hosts and images:

grep -rIE 'litellm.{0,8}1\.107\.0' \
  --include='*.txt' --include='*.lock' --include='*.toml' \
  --include='*.json' --include='*.cfg' .

find / -xdev \( -name 'cook.pth' -o -name 'cook.py' \) 2>/dev/null

Search on cook.pth rather than on sitecustomize.py. The third file has a legitimate name and a legitimate purpose, so by itself it proves nothing. The pair is the signal.

Then read your own bypass list out loud in a room. The rulesets API returns it, with the caveat that bypass_actors comes back "only if the user making the API request has write access to the ruleset":

gh api repos/OWNER/REPO/rulesets \
  --jq '.[] | "\(.id) \(.name) \(.enforcement)"'

gh api repos/OWNER/REPO/rulesets/RULESET_ID --jq '.bypass_actors'

Last, delete the standing upload token. PyPI's Trusted Publishing exchanges an OIDC identity token for a credential valid "only 15 minutes from time of creation", and the page states the problem it replaces in one sentence: "an attacker who compromises a package's release token can use it until its legitimate user notices and manually revokes it." The same argument applies to your internal index and your container registry.

When is none of this worth doing?

When there is nobody to keep out. One internal service, one repository, two engineers who both hold admin: a ruleset with a bypass list is paperwork, and the honest controls there are a lockfile and a digest. Say that rather than buying a policy everybody routes around by Thursday.

Pinning has a bill too. Pin every image by digest and somebody now owns an upgrade queue, and a pinned base image five months behind on patches is a worse position than a floating tag that stays current. We have watched a team pin hard, feel safe, and sit on a known-vulnerable layer until an auditor found it. If nobody is named for that queue, pin only the things that touch production credentials and leave the rest floating on purpose.

The short version

Grep the manifests today, and treat every credential the affected hosts could reach as compromised until the logs say otherwise, rather than the reverse.

Then do the part that outlives this one package. Pull your bypass actors list, put a human name against every entry, and remove the ones that exist because somebody needed to merge something in a hurry last quarter. Move publishing to OIDC so there is no standing token left to steal. Pin by digest wherever a deployment resolves latest, and name the owner of the upgrade queue in the same change.

And put your CERT-In point of contact in the incident runbook rather than in somebody's memory. Six hours is not long enough to go and find out who it is.

This is engineering guidance, not legal advice. Regulations and their annexures are amended over time; read the current text and take your compliance position from counsel.