The API Token That Cost $2.4 Million

Most security incidents start with something complicated. This one started with copy and paste.

An engineer needed to debug a deployment pipeline late on a Friday. They copied a personal access token into a local script, pushed the script into a private repository, and forgot about it. The repository itself never became public. That part is what makes the story useful. The leak did not happen because someone did something cartoonishly reckless. It happened because a normal team made a normal shortcut inside a system that treated long-lived credentials as acceptable infrastructure.

What followed was brutally efficient. The token had enough scope to reach a CI system. The CI system had enough trust to read deployment secrets. Those secrets opened the door to cloud control-plane access. Within hours, attackers had provisioned a burst of compute, spread workloads across regions, and started mining at a scale large enough to trigger a seven-figure invoice before finance, engineering, and security were even looking at the same dashboard.

The total damage landed at roughly $2.4 million once you combined fraudulent infrastructure spend, containment work, downtime, legal support, and the lost momentum of a leadership team forced into all-hands crisis mode. Expensive, yes. Rare, no. Variants of this are happening everywhere because API tokens have quietly become the master keys of modern companies.

Why tokens are more dangerous than passwords

Most executives still think about compromise in password terms. A password feels personal. A token feels technical. That mental model is outdated.

A password usually authenticates a human. An API token often authenticates a system that already has trust, automation, and permission built around it. The token is not just a login. It is delegated authority without friction. It does not get tired. It does not hit MFA. It does not ask whether this action looks strange.

That changes the blast radius completely.

• A leaked employee password may expose one account.

• A leaked CI token can expose every environment the pipeline can touch.

• A leaked cloud token can create, destroy, snapshot, exfiltrate, and reconfigure at machine speed.

• A leaked Git token can become a map of your internal systems, dependencies, and deployment workflow.

In other words: passwords open doors. Tokens often open factories.

The real failure was architectural, not individual

When I review incidents like this, I try to separate trigger from cause. The trigger was the leaked token. The cause was an environment that assumed static trust was good enough.

There were four architectural mistakes behind the final loss.

First, the token lived too long. It was effectively permanent until manually revoked. That meant the attacker did not need speed. They needed patience.

Second, the token carried broad scope. It was created for convenience, not precision. Teams often justify this with a simple argument: “We will tighten it later.” Later almost never comes.

Third, the environment chained trust implicitly. One credential unlocked a system that held another credential, which unlocked another. That is how modest access turns into strategic compromise.

Fourth, monitoring focused on infrastructure health rather than permission behavior. The dashboards were excellent at showing CPU, memory, and request latency. They were weak at answering a more important question: why is this identity suddenly creating expensive resources in unfamiliar patterns?

That combination is common. It is also exactly why so many companies still discover token abuse through billing anomalies instead of security telemetry. By the time finance sees the spike, the attacker has already had a productive day.

How the attack actually unfolded

The sequence matters because it shows how little sophistication is required once trust is misconfigured.

Step one was discovery. Attackers scanned repository history, not just current files. This is where many teams get caught: they remove the secret from the latest commit and assume the problem is fixed. It is not. Git remembers.

Step two was validation. The token was tested quietly against the source platform API. No noisy brute force, no malware, no exploit chain. Just a polite request with valid credentials.

Step three was expansion. The attackers enumerated repositories, actions, and connected integrations. They found workflow files, deployment patterns, naming conventions, and references to cloud roles. In a mature environment, that metadata alone is intelligence gold.

Step four was privilege pivoting. The CI system had access to secrets used during releases. Once those were reachable, the attacker no longer needed the original token to do damage. They had entered the trust fabric.

Step five was monetization. Crypto mining is crude, but it is reliable. It converts weak detection into immediate cash flow. If the attacker had chosen exfiltration over mining, the direct invoice would have been lower and the strategic damage much higher.

This is why I call secret leaks an operational issue, not just a security issue. The same design choices that create speed for good automation create speed for abuse.

Why nobody noticed for 72 hours

Executives love the phrase “real-time visibility.” In practice, most teams have real-time exhaust, not real-time visibility.

They had alerts, but the alerts were siloed. Engineering saw unusual compute growth and assumed a batch workload had expanded. Finance saw early spend movement but it was the weekend. Security saw API activity from an allowed identity and had no baseline that marked it as abnormal enough to escalate.

No single signal looked catastrophic in isolation. Together, they formed a very obvious incident. That gap between local normal and global abnormal is where modern breaches live.

This is one of the most important lessons for any company operating at scale: your control plane is a business system. If cloud provisioning, CI execution, and secret access are not monitored as commercial risk, not just technical activity, your response will always lag.

The five-minute fix that prevents most of this

There is no single silver bullet, but there is a simple move that cuts a huge amount of risk fast: stop using long-lived static tokens wherever your platform supports short-lived, identity-bound credentials.

That is the five-minute fix in spirit, and often close to five minutes in practice for the first high-value workflow.

Instead of storing a reusable secret in Git, CI, or a laptop:

• Use workload identity or OIDC federation from your CI system into your cloud provider.

• Issue credentials just in time.

• Limit them to the exact repository, branch, workflow, and action required.

• Set expiry in minutes, not months.

• Log issuance and use as first-class events.

If that sounds minor, good. The best security controls usually are. They remove an entire class of failure instead of adding another dashboard on top of it.

When the credential is short-lived and tightly bound to one workload, a leak becomes far less useful. The attacker has less time, less scope, and fewer pivot paths. You are no longer defending a skeleton key. You are defending a disposable ticket.

The operating model I recommend

If I were walking into a company on Monday to prevent a repeat of this incident, I would focus on six moves in this order.

1. Inventory all machine credentials. Not eventually. Now. You cannot defend what you have not enumerated.

2. Classify by blast radius. Which tokens can touch production, billing, identity, source control, or customer data? Start there.

3. Replace static credentials in CI/CD first. That is where convenience most often overrides judgment.

4. Enforce least privilege by default. Make broad scopes hard to request and easy to detect.

5. Monitor identity behavior, not just system health. A healthy cluster can still be a compromised business.

6. Treat secret exposure as inevitable and rehearse the rollback. Revocation, rotation, reissue, containment. Time matters.

Notice what is not on that list: more employee training. Training matters, but it is not enough. If one copy-paste mistake can still turn into a seven-figure incident, your architecture is carrying too much trust.

The broader lesson

We are entering a phase of technology where the most valuable assets in a company are increasingly machine-to-machine. APIs run revenue. Workflows ship product. Agents will soon make more operational decisions than junior staff ever did. That means the identity layer beneath automation is becoming the real perimeter.

Many teams still defend that perimeter like it is 2016. They vault secrets, rotate some of them quarterly, and congratulate themselves for having MFA on human accounts. Meanwhile, the most privileged identities in the company are scripts, runners, deploy bots, and service accounts with static credentials no human can even remember creating.

That is backwards.

The future of security is not only better detection. It is designing systems where leaked authority decays before it can be exploited. Short-lived credentials. Explicit trust boundaries. Narrow scopes. Continuous verification. Boring controls, rigorously applied.

None of this is glamorous. It will not trend on social media. But it is the kind of operational discipline that separates resilient companies from expensive case studies.

One leaked token should never be able to buy an attacker three days inside your business. If it can, the problem is not the token. The problem is that your infrastructure still confuses access with trust.

Fix that, and you remove one of the cheapest paths attackers have into modern companies.

---