I've got an estimated bill for $1.7 BILLION over this month. Normal usage is < $5.
Obvs have created an urgent AWS support ticket. Anyone else seeing something like this?
Update: Reddit link: https://www.reddit.com/r/aws/comments/1uyuaw7/help_my_bill_s...
I understand people complaining about large bills, but this is over the top!
It's been 2 hours and I still haven't fully calmed down.
I’m also a little surprised this didn’t trip a circuit breaker. For something as non-real-time as billing, I’m surprised they don’t have an automated kill switch that pauses the billing system and fires a page if variance in bills spikes. Naively some kind of “if the standard deviation of customer bills for this year changes by more than 50%, pause the billing system”. At that number of customers, those numbers should be pretty stable beyond internal billing changes they could normalize for.
Phew.
> You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!
I guess you wanted to say 2^30 which makes 1.5$
While we're being pedantic, 2^30 is 28 in normal programming languages ;)
And a distracted tester? And a distracted pipeline of regression tests?
No, the truth is way worst...
That's one way to cut staff.
It's a shit show.
The number of errors I've seen over the last 30 years seems to say humans not caring is as much of a deal AI use. It's easy to blame AI for humans being lazy, but I do think it comes naturally to us.
It probably shouldn't be legal for banks, hospitals, governments, or any other critical infrastructure to be hosted on AWS if they do things like this.
You should charge your customer 3015000 thousand dollars.
Number felt high so I wanted to double check and I only get 301500.
Literally impossible to tell whether this is parody or an actual response any longer.
I challenge anyone to write something so stupid that an LLM couldn't possibly respond with it. I don't believe such limit exists.
Looks like this is a bug w/ S3
I might be a winner.
Folks can track it directly on AWS Health: https://health.aws.amazon.com/health/status
I'm not currently running anything and have no plans to at the moment. I've always had a mild dread that I'll suddenly get a bill for more than $0.00.
If AWS can goof in a way that causes obviously massive bills (like today), what's to say they can't goof in more subtle ways and start charging small additional amounts that many people may not notice and just pay it.
I remember seeing a story shared by AMS RDS on linked-in that basically said... "Now with predictable pricing"
Huh?
Like “I was scared for a couple minutes on a Friday morning until I saw the vendor status page” is orders of magnitude away from the bar here.
Makes you wonder - what if there really would be an incident where some massive amount of traffic got routed to your infrastructure by some heavyweight player? Say Wikipedia accidentally switches their IP to your CloudFront? Would you really be on the hook for $500k?
"Operational issue - AWS Billing Console (Global) Service - AWS Billing Console Severity Impacted - Inaccurate Estimated Billing Data"
"The rollback of a recent change did not resolve the issue and we are continuing to investigate multiple mitigation paths. Estimated bill updates remain paused."
Wait what if someones actually getting usage spiked
All hail the new generations of our uberployers.
But with AWS costs rising anyway (not by that much but OK), I'm probably not the only one to start reconsidering their cloud strategy. I think this might have just pushed me over the edge.
"If you owe your bank manager a thousand pounds, you are at his mercy. If you owe him a million pounds, he is at yours."
Thanks AWS, no caffeine needed this morning!
Of course, you provided heart stress failure tests for free.
AWS on their support data is reporting this:
Inaccurate Estimated Billing Data
Jul 17 3:03 AM PDT We continue to work to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. We have identified the root cause as an issue with unit pricing within the estimated billing computation subsystem and we are working on a mitigation. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data. We will provide another update by 4:00 AM PDT or sooner if more information becomes available.
Jul 17 2:07 AM PDT Beginning on July 16 7:38 PM PDT, we began displaying incorrect estimated billing data in the Billing and Cost Management Console. Our engineering teams are engaged and investigating root cause. We will provide another update by 3:00 AM PDT or sooner if more information becomes available.
Jul 17 1:33 AM PDT We are investigating issues with Cost Explorer reflecting inaccurate estimated billing data.
Honestly, I would worry more about estimated billing that seems plausible in general, but is way to high for you personally. These ridiculous amounts? Not so much.
$103,515,940,301.79
Use AI => No War Crimes!
Well, actually i guess you can guess what happens next! lol :-D
The messages started as polite and eventually started to get more desperate in tone. At no point were they threatening or adversarial.
Amex realises that threatening would hurt their business trust more than anything. During the great depression, Amex accepted checks from other banks which were falling and paying through their own wallet as a matter of integrity. Amex has always been built around this idea of trust and prestige.
They make most of money from what I have heard on the transaction fees which are more than others (3% compared to 1%). They might get desperate but I am sure that they are one of the last guys who would wanna threaten you if you are paying some large bills for them (as compared to normal credit card companies which might even hire people to extract your loans in some messy situations)
So perhaps be so rich that the credit card company understands it as well and treats ya differently :-D
(I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
This does feel a bit silly for amex to do from what I've heard. Probably 2008 were a weird time in general where trust in systems itself were mostly eroded, whether of people to banking institutions and also vice versa.
> (I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
haha :-)
Host your own people. Host your own.
100K years. Now that's load-bearing ...
At first I was sure it was a phishing attempt. Then went to the console (not using those links) Saw there was an outage where the console was wrong (no mention of email alerts) Then I thought I was hacked - what a perfect cover up for someone to evade detection when the console was wrong. Looked at some logs, realized the incident text was just not exhaustive on the impact. Went back to my cup of coffee.
Note to self- should have looked here first.
"Software Development Engineer II, AWS Invoicing"
https://www.amazon.jobs/de/jobs/10428480/software-developmen...
"...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinforcing areas: intelligent launch readiness — where autonomous AI agents analyze, generate, and validate the information needed to go live in a new market; cloud-native service orchestration — where configuration-driven microservices replace per-launch bespoke engineering with centralized, reusable capabilities so that expanding into a new country becomes a zero-code configuration change rather than a development cycle; and continuous validation..."
"Senior Software Development Manager, AWS Global Bill Generation" https://www.amazon.jobs/de/jobs/10471948/senior-software-dev...
"We're transforming from monthly batch processing and manual war rooms to continuous billing, autonomous agents, and self-healing infrastructure. We believe operational burden is a technical problem, not a staffing problem"
This looks clearly...a staffing problem...
> In this role, you will own end-to-end bill run execution across all AWS partitions, drive the technical vision for autonomous billing operations, and build the team that ensures every customer receives an accurate cost estimated in minutes ...
I think that big tech recently decided that I got 99 problems but staffing ain't one
I guess Nothing is a staffing problem when you make a rule that firing people is always the solution.
And if that weren't enough, the "manual war rooms" alone should be setting off alarm bells.
This is a gold-mine. They need to get sued heavily for this incompetence.
> Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage…
But that’s word for word a 250k+ TC job in the big ‘26.
In this role you will:
- Design and build agentic AI systems that analyze, generate, and validate...
- Build agentic architectures that compose specialized AI agents dynamically...
- Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage...
So instead you take a probabilistic approach - charge the user for a megabyte of data transfer 0.1% of the time, and bill nothing 99.9% of the time.
Now the typical cost is the same, the users bill is probably accurate to the cent, but you have divided the number of billing records by 1000.
This seems statistically invalid insofar as it will tend to overbill potentially by a lot on the minority of cases.
Don't you know how much of the pipe is occupied by a given customers code at any given time or what data is being sent
Right, so invoicing is still a deterministic problem. You can bolt whatever on but in the end it's just product x price x units
The idea is that if we can make small billing mistakes like that, we can make large billing mistakes, and need to invest in the correctness of the systems powering billing.
I have great respect for the engineering culture within AWS during those times. I am glad to have left before seeing it degrade and decline.
I don’t understand why companies still place AI in their system at authority-forming level, if anything it should sit at least behind single guard / validator that is explicitly coded and tested, never in front of presentation layer.
Fuck it, im in.
I love this framing
:)
"Limits except for Storage" seems even easier - I don't think I've ever heard of a storage-based billing story, although I'm sure one or two exist
Also many places I’ve worked, storage is a huge part of the spend but that depends a lot on what you do. e-commerce doesn’t use a ton of it, but if you handle user-generated content or do any kind of training (LLM, computer vision, etc) then you can very much end up in a place where storage becomes a top line number for infra spend.
GitHub pre-Copilot was probably like that. They host a shitload of data, most of which is just at rest the majority of the time. Storage and networking are probably the majority of their infra costs.
I have seen things get hacked for bandwidth, back in the days before you could rent a gbps uplink from the cloud for $0.12. Some scene release groups would hack into universities or companies to do the initial seeding over their super fast links. It used storage, but that wasn’t really the goal.
Incidentaly, smaller competitors solved this issue decades ago, while the big cloud decided it is more convenient never to implement it.
More AI will 100% fix the issue lol
That would mean an outage but that is still better than going bankrupt and teach you a thing or two about monitoring.
Not necessarily. They could imply that your storage becomes inaccessible immediately, but only gets deleted after some time period (say, 1 month). What spending limits do depends on the implementation.
My process went: verify email is not phishing (it was), login to console and check dashboard (same amount), attempt to understand cost (cost management kept contradicting itself), try to log support ticket and only on that part did I notice the status notification. At least I can breathe again now!
Apparently you can trigger an Action (e.g. prevent uploads) when the billing alert triggers, but then my platform wouldn't work anymore, just because AWS had an issue. Also insane that Amazon still hasn't send an email to clarify.
I was on the phone with him and we checked that he didn’t leak any APIs keys but no traffic at all. I even thought of a breach at a vendor he uses for some s3 stuff before I found this thread on HN.
Anyways I didn't need coffee. That produced an adrenaline release unlike any I've experienced before. Thanks AWS
If you owe AWS 437B bucks, that’s a big problem for AWS
It's entirely irresponsible of Amazon to even display such values to the user.
https://edition.cnn.com/2021/07/01/business/robinhood-lawsui...
https://techcrunch.com/2026/04/05/copilot-is-for-entertainme...
I'm not a devops but we are running some s3 buckets. When I saw the number I couldn't even comprehend and thought we had been hacked.
I feel like this is such a bad mistake from AWS and we will never be compensated for it.
Put me off having anything on AWS from now on.
Ugh. No. Never. Do not use Amazon Pharmacy.
I had it for three months, and each month it was unable to deliver pills before I ran out, so I cancelled and switched to the brick-and-mortar pharmacy down the street.
A year later, suddenly Amazon Pharmacy starts sending unauthorized prescription refill requests to my doctor.
I still have the account cancellation confirmation e-mail from Amazon Pharmacy, but Amazon won't close my account. Amazon's account rep says it cannot close accounts for "legal reasons." Bullshit. He can't say what the legal reasons are, or point me to a document stating these conditions.
Now my doctor's office just ignores all refill requests from Amazon Pharmacy.
Never trust your health to big tech.
The account was probably real, made for some purpose 15 years ago. I had ignored the charge for years because it was like $7 a month. Then it went to $300/month, making it worth the time. I could find no invoice email and none of my AWS accounts lined up with the bill.
I tried contacting support, but without the account number involved they had no way to help. I disputed the charge, the bank refunded that month and then went right back to charging the next month(my credit card helpfully accepted the charges despite the dispute).
I had to cancel the credit card entirely to make it stop.
Yeah, this most certainly is bad code wrapping around a value. AWS will post a notice soon if they haven’t already.
But so is imprisoning or exterminating all humans for their own good, as most AI dystopias end up as.
The crypto network you hosted should pay for itself in 10-20 years just like LLMs. Don’t worry. Consider Bank of America until then if you are good on credit score.
$151 billion the number for me.
Zimbabwe redenominated its currency three times in 2006, 2008, and 2009, dropping a total of 25 trailing zeros in the process. the 4th Zimbabwean dollar in 2009 was worth 10^25 of the first zimbabwean dollar.
$1,299,988,247,332.56!
That was a fun set of emails to wake up to, figured they had to be phishing for how outrageous of a number it was. But nope! Fun little incident they've got going over there.
Insane
>The second path involves rolling back a recent change to the billing computation subsystem.
Want to bet AI code was involved?
https://www.theregister.com/software/2026/02/24/aws-would-ra...
AI: "No problem, let me change how we bill and fix the tests for the new increase in value"
How is 1 + 1 = 2 hard to test?
Vibe coded implementation paired with vibe coded tests made to fit the implementation?
If I got an email with that amount I would've just laughed at the incompetence that lead to a bug that lead to me getting the email. But I write as a backseat email recipient.
People make similar pronouncements after every us-east-1 outage makes the news, but I feel like AWS would be going out of business by now if people followed through.
It reminds me of airlines, where after a particularly grueling irregular ops experience, a few dozen people file off the plane swearing "Never again, <airline name>!" but really, we all must know deep down that the airlines are all subject to the same external inciting factors, internal profit motivations, and human imperfection, and thus all pretty equally likely to cause us a bad day or ruined trip. The effort spent to avoid one isn't really worth it.