On October 21, 2026, two stories unfolded in one morning regarding the same class of data — secrets residing in environment variables. At 11:39 UTC, a poisoned version of LiteLLM 1.107.0 was released on PyPI — a popular gateway to language models that is utilized in many AI agents and parsing pipelines. On the same day, the cloud platform Railway reported that an outsider may have accessed the environment variable storage of clients, with some of them having their decrypted values exposed. If your scraper, bot, or agent runs on Python or in a PaaS, let’s figure out what exactly leaked, how to check yourself, and in what order to change keys — including proxy logins and passwords.
What Happened with LiteLLM 1.107.0
According to Datadog Security Labs, attackers used stolen OAuth tokens from the GitHub CLI of three project engineers. With these tokens, they initiated a private release workflow for LiteLLM, but built it from a fake fork Berri-AI/litellm — which differs from the genuine BerriAI/litellm by just one hyphen. The launch was confirmed by the attackers themselves, traces were deleted, and the name of the fork remained in the build provenance signature.
- 11:39 UTC — version 1.107.0 is published on PyPI;
- 13:20 UTC — PyPI quarantines it;
- 13:53 UTC — infected Docker images are removed.
The window was about two hours, but the package, according to Datadog, has around 23 million downloads per week. Not only seven wheel files for macOS, Linux, and Windows were infected, but also the official images on GHCR and Docker Hub. The floating tags latest, main-latest, and main-stable pointed to the malicious image that day. This means that anyone whose container was rebuilt or restarted pulling the “latest version” could have been affected, even if they did not manually update anything.
This is the second incident involving LiteLLM in 2026: in March, versions 1.82.7 and 1.82.8 were compromised. As noted by Datadog, no connection between the incidents has been established. The last clean version is 1.106.2.
Why Removing the Package Is Not Enough
The main trick of the attack is the Python auto-start mechanism. Along with the package, three files were placed in site-packages: cook.pth, cook.py, and sitecustomize.py. Python executes files with the .pth extension and the sitecustomize module at every interpreter startup. As analyzed by Endor Labs, the cook.pth file contained exactly one line import cook — and that was enough for the secret collector to trigger in any Python process on the machine, even if LiteLLM was not imported anywhere.
- The collector (cook.py) packaged the found data, encrypted it with AES-256 wrapped with an RSA-4096 key, and sent it via a POST request to
www.litellm[.]dev/healthcheck— the domain was intentionally made to resemble the legitimate one. - The backdoor (sitecustomize.py) read the server address from a file on GitHub every hour and executed the code sent from there without signature verification. According to Datadog, the file only contained test values, and no second stage was recorded.
- Persistence:
pip uninstall litellmdoes not remove the files, and every launch of python or pip restarts the collector. Therefore, researchers recommend checking the infected machine with tools not based on Python.
What Exactly the Collector Was Looking For
The list of targets is broad — essentially, everything that the process can see:
- all non-empty environment variables, and on Linux, the environment of other processes;
- files .env, CI configs, Docker and Kubernetes, service account secrets;
- AWS, GCP, Azure cloud keys, including instance metadata service keys;
- SSH, GPG, and TLS keys, Git, npm, and PyPI tokens, the .netrc file;
- saved passwords and cookies from Chromium and Firefox browsers;
- keys and sessions from AI tools: Claude, Codex, Gemini, Cursor, Copilot, and others;
- if the LiteLLM Proxy database is configured — up to 500 rows from tables with model provider keys;
- command shell history.
There is no separate mention of the HTTP_PROXY and HTTPS_PROXY variables in the analyses. However, since the collector retrieved all environment variables, .env files, and shell history, proxy credentials were automatically included: that is typically where they are stored. A line like http://login:password@host:port in HTTPS_PROXY, in the .env of the parser, or in a curl command from history is just as much of a secret as a cloud key.
Railway: The Leak Came Not from Code, but from the Platform
In its disclosure on October 21, Railway stated that it noticed anomalous activity in the systems where clients' environment variables are stored. As of now, an outsider may have accessed encrypted variables, and some may have had their decrypted contents exposed. Deployments were paused. The company does not specify the scale: it refers to a “portion of users” and promises to directly contact those affected, but advises everyone to take precautions:
- change all environment variables in Railway, including API keys;
- change credentials for private registries;
- check service logs for unusual activity.
The company links the incident to a series of attacks on platform services, aimed at obtaining keys from AI inference providers; the providers have already been warned, and they are revoking keys. In an update on October 23, Railway additionally recommends reconnecting GitHub and recreating CLI and CI tokens.
The common denominator of the two stories is simple: environment variables are not a safe. They are a convenient shared list of secrets that any code in the process can read and that any platform where you deploy can store.
How to Check if You Were Affected
- Check for the version in dependencies. Look at requirements.txt, poetry.lock, uv.lock, and CI logs from October 21 for
litellm==1.107.0. Consider transitive dependencies: LiteLLM is often pulled in by agent frameworks, not directly by you. - Search for files in site-packages. The presence of cook.pth or any foreign sitecustomize.py is a direct indicator. Use
find, not a Python script. - Verify images. If you pulled
litellm/litellm:latestormain-stableon October 21, compare the digest with the published indicators. - Look for markers in /tmp:
/tmp/.pc_once_72129118c59f5636c0bband/tmp/remote_code_loader.lock. - Check network logs for requests to
litellm[.]devand paths ending with/codepy. - Railway — check your email and the incident status page, but do not wait for a letter: the recommendation to change keys applies to everyone.
If you find anything, it’s better to rebuild the machine from a clean image than to clean it on-site, and change keys from another, known clean device.
In What Order to Change Secrets
Changing “everything” is the right approach, but in practice, this stretches over days. The order should be chosen based on the potential damage and how quickly the key can be exploited:
- Cloud and payment keys — AWS/GCP/Azure, keys from LLM providers, master key for LiteLLM Proxy. These can quickly lead to financial loss.
- Git, npm, PyPI, image registries, and CI tokens — through these, the attack can reach your users. The history of LiteLLM began with stolen CLI tokens.
- Databases and SSH keys.
- Proxy logins and passwords. An outsider with your credentials consumes your paid traffic and accesses the internet from your IPs — the consequences in the form of bans and CAPTCHAs will fall on your accounts.
- Browser and AI tool sessions — log out everywhere and log back in.
For proxies, there is a simple early signal: traffic consumption that you cannot explain. If the balance in your dashboard is depleting faster than usual or the number of requests increases during hours when your tasks are not running, someone is using your credentials.
How to Structure Proxy Credentials to Minimize Leak Impact
It is impossible to completely protect against a poisoned package, but you can structure things so that a leak affects only one task, not the entire stack.
- Separate proxies for each task and environment. Do not use one login for production parsers, local development, and CI. If a key leaks from CI, you disable one proxy instead of halting everything. In ProxyCove, each purchased proxy is a separate login/password pair, so this separation costs nothing: for price monitoring — residential proxies, for mass technical requests where a “home” IP is not needed — datacenter proxies.
- Do not store credentials in the HTTPS_PROXY string globally. A global variable is visible to every process and every package. It is safer to pass the proxy only to the necessary client or set up a local gateway that knows the password — how to do this with gost, 3proxy, or Squid has been discussed in our article on local gateways for proxies with passwords.
- Store secrets in a secrets manager, not in .env files in the repository or shell history. General rules can be found in our guide on securely storing proxy credentials.
- Maintain a small traffic balance on proxies that operate in risky environments like CI or test stands: the ceiling of potential damage equals the balance.
What to Change in the Build
Most of those affected by LiteLLM suffered not due to “bad” code, but due to build habits. Some of the advice below comes from analyses by Datadog and Endor Labs, while some is standard build hygiene:
- lock versions with hashes in lock files, and images by digest, not by the latest tag;
- do not update dependencies on release day: a short delay before accepting new versions cuts off most attacks with a window of a few hours;
- watch for new .pth files and sitecustomize.py in third-party packages — this is rare for a regular library;
- treat developer CLI tokens as production keys and allow release workflows only from trusted repositories and branches;
- run parsers in isolated containers with minimal secrets: a process that collects prices does not need cloud keys and SSH.
Conclusion
LiteLLM and Railway have shown two sides of the same problem: secrets leak both from below, through dependencies you do not even import, and from above, through the platform you entrusted them to. If on October 21 you installed or rebuilt Python environments with LiteLLM or deployed on Railway — change your keys now, starting with cloud and CI tokens, and don’t forget about proxies. For the future, separate proxy credentials by tasks and environments: this way, the next leak will only require disabling one login instead of halting all parsing.
