Your customers depend on the AI features behind your gateway. If your self-hosted LiteLLM deployment received the compromised 1.107.0 release, the first job is to restore safe operation. Then investigate the affected environment and work out what its identity could reach, what evidence shows was taken, and where those credentials were used.
A compromised gateway may still be answering requests. Service availability and trust are separate checks. With OpalSpan, you can give the recovery and investigation work to Mech, the built-in AI technician, or to your own assistant using OpalSpan Agent CLI. Agents can work on connected computers and servers, uncover and analyze evidence, and leave a useful handback for the people responsible for the gateway and its customers.
The accompanying desktop and mobile films show the same fictional gateway recovery through the actual shared Mech interface. Data, actions and timing are scripted; the films illustrate the job rather than a customer incident or measured response time.
Restore safe operation first
Contain the affected instance and prevent its orchestrator, service manager or build automation from starting it again. Where practical, take a quick snapshot of deployment metadata, relevant logs and process/network evidence before replacement. Preserve UTC timestamps, the collection source and retention boundaries. Do not delay urgent containment or credential response to collect a perfect evidence set.
Do not start Python or pip in a suspect environment, including for a version check. The malicious startup hooks could run without importing LiteLLM. Inspect files and deployment records from a known-clean system with trusted tools; preserve the suspect environment for investigation instead of treating an in-place package downgrade as recovery.
Use your existing CI/CD or deployment process to prepare a trusted replacement from verified artifacts and known-clean configuration. Keep the suspect host, build cache and image out of that path. From a clean machine, revoke or rotate credentials the compromised identity could reach and address active sessions or temporary credentials. Put replacement secrets only into the trusted replacement environment.
Before restoring traffic, check:
- Artifact identity: exact image/index and platform-image digests, package hashes, expected source and current signature policy.
- Gateway behavior: routing to the intended models/providers, authentication, tenant/key restrictions and the intended egress policy.
- Dependent applications: representative AI requests, error handling and the customer workflows that consume the gateway. Record what was actually tested.
- Recovery position: a known-clean fallback, the owner of cutover and credential response, and the preserved evidence location. The compromised instance is not a safe rollback.
Choose today's verified release guidance
Guidance checked on 24 October 2026: the initial incident post recommended 1.106.2. LiteLLM has since published 1.106.3, with an eMudhra root-CA removal and a changed cosign signing key, also backported to specified older branches. Review that release's compatibility and verification instructions before selecting your replacement. The release identifies the new key at commit e82a3c646908ba0a6ee0beddc4310d7a6cac4065 and publishes its SHA-256. Update an approved pinned-key/admission policy through its owner; do not disable signature checks to make a deployment pass. A valid signature alone also needs the expected source and release context: the attacker used the upstream release machinery.
Establish which artifact reached your gateway
The LiteLLM incident notice identifies all seven PyPI 1.107.0 wheels as affected, available on 21 October from 11:39:51 to 13:23 UTC. Docker Hub tags v1.107.0, latest, main-latest and main-stable pointed to the compromised index from 11:46:56 to 14:20 UTC:
sha256:1bbcb1443bc962e45f5aad53fe3d2c30600e0379715fb6c664cdd3ac126c339e
The notice also lists affected GHCR image variants and their indexes; those tags were unavailable by 13:51 UTC. Compare the correct registry, variant, index and platform image. A tag's value today cannot establish what a node pulled then. Mirrors, cached wheels, intermediate images and copied artifacts can bring an affected build into service after the original publication window.
The official Git v1.107.0 source tag did not contain the malicious files. A source checkout and a registry artifact bearing the same version are different evidence. The vendor states its private enterprise registry was unaffected; verify where your own deployment actually obtained its image.
Use MAL-2026-2144 and its alias PYSEC-2026-2 as identifier records. That record also contains the separate March 1.82.7/1.82.8 incident: keep its indicators and upstream story separate from October. Microsoft's TVM-2026-0003 is a vulnerability-management identifier, not a CVE; no verified CVE alias for this malicious release is asserted here.
Turn checks into decisions
Use the Microsoft technical analysis and indicator tables alongside the vendor notice. Gateway request logs are one source; the startup collector could run before or outside gateway request handling.
1. Installation and startup files
Check: compare deployed image/index digests and package hashes with the published artifacts. Read package records, litellm-1.107.0.dist-info, image layers and deployment/install times without invoking the suspect interpreter. Inspect every relevant site-packages root, outside the litellm directory, for cook.pth, cook.py, sitecustomize.py and associated cached bytecode.
Possible finding: an affected artifact or a file matching a published SHA-256. The October indicator records list:
cook.pth:7f191632e7320705d154b85204f1ae546b543acbd7e8861229b76f5042b92bdbcook.py:9dd0a3926f96a37eb86cf1ee2310d9be93139cc4ebef3ef9e2efe9531390be25sitecustomize.py:34bef04fe88ac9d0aae0475630b1738681d15b94f233ac8aa46eacf5968305fa
The linked Microsoft tables include additional Windows-wheel and other variants; these three examples are not an exhaustive hash list.
Interpretation → decision: matching content establishes a much stronger artifact finding than a generic filename. Correlate it with interpreter execution and the workload identity; keep the environment contained and rebuild. Absence after replacement, package removal or cleanup does not clear the original environment.
2. Process ancestry and staging
Check: retained process/EDR records around installation and subsequent interpreter starts. Relate parent/child Python activity, the service or container identity, OpenSSL processes and temporary .pc_e_ staging to the published behavior.
Possible finding: the interpreter starts, cryptographic staging follows, and the same identity makes an outbound connection.
Interpretation → decision: a coherent sequence supports execution/collection evidence. OpenSSL or Python activity on its own is common. Preserve the source event IDs and times, investigate descendants and widen the credential inventory to what that identity could read. Microsoft describes a Unix loader capable of retrieving additional code; the cited analysis did not confirm second-stage execution.
3. Network and temporary markers
Check: container/service logs plus DNS, proxy, firewall and egress records. Correlate the October destinations in the published analysis with workload, time, response and transfer metadata. Inspect preserved /tmp/remote_code_loader.lock and /tmp/.pc_once_72129118c59f5636c0bb evidence without executing its contents or contacting attacker infrastructure.
Possible finding: a collector marker alongside an outbound upload, or a loader lock tied to a surviving process.
Interpretation → decision: Microsoft reports that the collector attempted to write its marker after an HTTP 200 response. That supports an upload finding when corroborated; it does not list exactly which credentials were in the encrypted payload. A lock filename alone is not proof of a running loader. Missing temporary files, short-lived containers, retention gaps or missing DNS logs leave uncertainty. Record the gap and continue urgent credential response.
Build a credential exposure inventory without copying the secrets
Inventory names, IDs, secret references, access permissions and evidence, not raw values. Start with the affected process identity and include:
- Environment variable names, secret-manager references, mounted-secret paths and readable configuration locations; on Linux, account for other process environments readable through procfs.
- Model-provider keys; LiteLLM master and virtual keys; database credentials; cloud instance/task/service-account identities and their permission scopes.
- Source-control, registry and CI/CD tokens; SSH/GPG material; developer-tool and browser sessions where the affected identity could access them.
- Proxy database metadata. The published collector targeted
LiteLLM_CredentialsTableandLiteLLM_ProxyModelTable, particularlycredential_valuesandlitellm_params. Identify affected credential/model records and their owners through a clean administrative path. Do not dump those fields' raw secret values into a chat, Case, report or diagnostic bundle.
For each entry distinguish accessible, evidence of theft, and evidence of use. Permissions or a mount can establish accessibility. Correlated collection/upload evidence can support suspected theft. Provider/cloud audit event IDs, source addresses, usage and identity events can establish or refine subsequent use. With authorized access, check CloudTrail GetCallerIdentity near the exposure and later actions by that role, accounting for your own infrastructure and expected automation. That API call alone does not prove theft.
These checks use the relevant platform's existing tools and audit access; OpalSpan does not imply a native integration with every provider. An empty audit search can mean limited logging or retention. Do not wait for proof of abuse before revoking or rotating reachable credentials.
Trace how the artifact entered your deployment
Follow your own image pull or package resolution back through lock files, CI run IDs, base images, caches and internal mirrors. Record the source URL/registry, exact digest, build identity, deployment owner and time. Check how floating tags, trust/admission rules, secret scopes and release approvals allowed that artifact to reach the workload. Locate copies that could reintroduce it.
Keep those findings separate from the vendor's upstream investigation. LiteLLM's 23 October canonical account says an exposed, unrevoked master key allowed access to the separate LiteLLM Agent Platform API, which returned an organization owner's GitHub token. Additional credentials and release-workflow changes followed. This corrected the initial proxy-log/Claude hypothesis. It explains the upstream publication; it does not establish how your gateway was deployed or what happened on your host.
Give Mech the recovery job, then the investigation
Connect the relevant systems to your OpalSpan workspace and provide the affected gateway, clean replacement/deployment path, incident window, evidence sources and your operational limits. A useful request is:
Restore safe service for our LiteLLM gateway first. Contain gateway-01 and prevent restarts; preserve a quick metadata/log snapshot if it will not delay urgent recovery. Do not run its Python or pip. Use our trusted CI/CD and clean recovery environment, and coordinate urgent revocation/rotation from a clean machine. Keep new secrets away from the suspect environment. Verify routing, authentication and the dependent app before cutover. Then analyze the preserved artifact, process, network and credential-reference evidence for the October 1.107.0 incident. Correlate provider audits where access is available, without copying raw secrets. Trace how the artifact entered our deployment and identify remaining cached copies. Hand back observed, likely and unknown findings with source/time, owner and next action. Ask before expanding beyond these systems or changing our recovery limits.
An external assistant can carry out the same computer/server work using OpalSpan Agent CLI. You direct the outcome and handle meaningful decisions; the agent works through the available evidence and tools.
The handback should make the next shift useful: what is serving safely and how it was checked; which credentials were handled; what the evidence establishes; what remains unknown; and who owns each next action. Preserve original evidence in its controlled location and link the relevant records. That gives your agency and your customer a defensible recovery position while deeper investigation continues.
Start with OpalSpan · Explore Mech and external-agent workflows