One of our clients was compromised through a single AWS access key. We are sharing what happened, with the client’s identity and identifying details withheld, because the pattern is common, the damage is avoidable, and most of the companies exposed to it do not know they are.
The key
The client had created an IAM access key themselves and given it administrator permissions. It was a long-lived key: a pair of strings that works from anywhere in the world, for as long as it exists, with no second factor.
That key was exposed. The incident began at 11 at night.
The hour is not a coincidence of bad luck. It is the pattern. Attacks like this are automated and run around the clock, so they land when nobody is at a desk: late evening, weekends, public holidays. A company that would notice an odd charge at 11am has no one looking at 11pm, and a service billed by usage can run up a great deal before morning.
How a key gets out varies. Committed to a repository. Pasted into a chat or a ticket. Left in a build setting or on an old laptop. The route matters less than what the key could do once someone else held it. This one could do everything.
What the attacker wanted
They did not start with the client’s data. They started with compute.
The attacker used the key to create Amazon Bedrock resources. Bedrock is AWS’s service for running large AI models. Using someone else’s account to run those models is now common enough to have a name, LLMjacking, coined by the security firm Sysdig in 2024. Sysdig estimated that abusing one expensive model at quota limits across several regions could cost a victim more than US$46,000 a day, and more with newer models.
From an attacker’s point of view it is ideal. The product is immediately resellable, it leaves nothing on disk, and many companies have never opened the Bedrock console, so nobody is watching it.
Then they made sure they could come back
This is the part that turns a bad day into a long week.
An administrator key can create other ways in. The attacker did, adding backdoors through Lambda and through Git.
A Lambda function is a small piece of code AWS runs on demand, under an identity of its own. A function planted by an attacker, with a privileged role attached, can be triggered later to hand access back, long after the original key is dead. Source control is a similar foothold: whatever is committed gets built and deployed with the pipeline’s permissions.
So disabling the leaked key was necessary and nowhere near sufficient. By the time anyone looked, the key was no longer the only door.
What we did
When an administrator credential has been in someone else’s hands, every part of the account is suspect until it has been looked at. The clean-up came down to four pieces of work.
1. Swept every region and every service
The AWS console shows one region at a time. A team that works in one region rarely opens the others, and attackers know it. Resources launched in a region you do not use can run for weeks without anyone seeing them.
So we went through the account region by region and service by service, looking for anything running that the client’s team could not account for: virtual servers, Lambda functions, containers, AI model resources, anything that executes code or accrues charges. Every item was checked against what the client had actually deployed. What they recognised stayed. What they did not was stopped and removed, the attacker’s Bedrock resources and Lambda functions included.
This is tedious work. It is also the only way to be sure. A suspicious server is both a running cost and a possible way back in.
2. Removed every access key from every user
Not only the leaked key. All of them.
Once an administrator key has been exposed, there is no reliable way to know which other keys in the account have been read, copied or newly created. A key that looks legitimate may be one the attacker minted an hour after getting in. Deciding key by key which ones to trust is how a backdoor survives.
So every access key attached to a user was deleted. That closes the door the attacker came through, and any spare ones cut along the way.
It also removes the underlying weakness. A long-lived access key is a password that never expires, works from anywhere, and has no second factor. Where something needs programmatic access, the right tool is a role with temporary credentials, not a key in a file.
3. Moved everyone to multi-factor authentication
With the keys gone, people sign in as themselves, and every sign-in now requires a second factor.
That changes what a leak means. A password or a credential that turns up in the wrong place is no longer enough on its own. An attacker would also need the person’s device.
4. Cleared out the roles
Roles are where a careful attacker hides, because a role does not disappear when a key is deleted. A role that trusts an outside account is a standing invitation to walk back in, and it sits quietly in a list most teams never read.
We reviewed every role in the account and removed two kinds.
Suspicious roles created recently. Anything created in the period after the key was exposed, or that nobody on the client’s team could explain, was examined: who can assume it, what it is allowed to do, when it was last used. Roles the team did not create were removed.
Deprecated roles. Old roles left over from past projects, former suppliers and experiments. They were not part of the attack, but each one is permission nobody is watching. An account that has just been compromised is the right moment to get rid of them.
The backdoor planted through Git was dealt with in the same spirit: find what changed, remove it, and replace any credential it could have touched.
The bill
Then there was the money. The attacker’s usage had been charged to the client’s account, and an AWS bill is the account holder’s responsibility by default, whoever ran the workload.
It does not have to end there. We guided the client through asking AWS to review the charges, and three things made the difference.
A paid support plan. The client was not on one. We had them register for paid AWS support, so the case would be picked up by a support engineer and there would be a proper channel to follow it through. The cost of a support plan is small against a disputed bill.
Documentation. A request to waive charges succeeds on its record. We helped the client set out what had happened: when the unauthorised activity started, which services and regions were involved, which resources the attacker created, and how the access was obtained.
Proof. Alongside the account of events went the evidence for it: that the usage was not the client’s own, and that the account had since been secured. This is where the clean-up pays for itself a second time. Keys removed, multi-factor sign-in enforced, unexplained roles and resources gone. AWS needs to see that the cause has been dealt with before it considers the consequence.
The charges were waived.
That outcome is at AWS’s discretion and nobody should plan on it. What a company can control is how good its case is: move quickly, fix the cause thoroughly, and write everything down as you go. The same notes that guide the clean-up become the evidence afterwards.
What that list has in common
Look at the four steps again. Sweep everything. Remove every key. Require a second factor. Delete what you cannot explain.
None of them is clever. All of them depend on being thorough, and on being willing to remove things that might have been fine. After a compromise at administrator level, “probably fine” is not a category. Either the team can say what something is for, or it goes.
Why cleaning up is not the whole fix
After the clean-up, the account is safe again. It is not yet an account where the same mistake would be harmless. Someone could create a new administrator key next month.
That takes a second layer: limits that sit above the account and apply even to administrators. This is the baseline we now run on every AWS organisation we manage, our own included.
No long-lived keys. People sign in through single sign-on and receive credentials that expire within a working day. There is nothing in a .env file to leak.
Guardrails above the account. Service control policies apply to everyone in an account, administrators included. Ours deny Amazon Bedrock and AWS Marketplace outright wherever the business does not use them, restrict activity to the regions in use, and prevent anyone from stopping the audit trail or removing the account from the organisation.
One detail is worth passing on. A region restriction alone is not enough. In one organisation, the region policy AWS had generated exempted Bedrock from the regional limit. Only an explicit deny on the service closed it. We now test the block from inside each account: listing and invoking models both return access denied.
An audit trail nobody can switch off. One trail covers all accounts and all regions, with log file validation, written to a separate account.
Alerts that reach a person. A budget for each account and a daily spend anomaly alert. These only warn. They do not stop spending, which is why the hard blocks above matter.
Private data paths. Databases are not reachable from the internet. Access goes through a short-lived tunnel authenticated by the same single sign-on, with a throwaway key that is valid for a minute.
Setting this up takes about a day. It costs very little to run.
If you run a company with an AWS account
Ask whoever manages it six questions.
- Does anyone or anything use a long-lived access key? Where is each one stored?
- Which of those keys has administrator access?
- Does every person sign in with multi-factor authentication?
- When did someone last look at the list of roles, and can they explain each one?
- Has anyone ever checked the regions you do not use?
- Who gets an alert when spending jumps, and would they see it at 11pm on a Sunday?
If the answers are uncertain, that is the finding. We wrote up the baseline step by step, and why AI services are the new target.