Everytime I read something like this , I get nervous about the cloud providers and Google. Since this is a relatively high profile customer standards, shouldn't they explain what caused them to suspend the account ?
Are they really "high profile"?
I've never heard of them until this incident or maybe the other one with the database.
They just seem... Loud, publicly. And always about failure.
https://hn.algolia.com/?dateEnd=1779148800&dateRange=custom&...
72M deploys from 3M customers across 10M services in the last month.
And vice versa. Company accounts have been deleted because an employee's personal account tripped Google's random number generator. Again, only Google does this.
Also if this rumor is true, they also get massive white glove engineering support: https://www.cnbc.com/2025/08/21/google-scores-six-year-meta-...
it's not like they're the only provider, or even the #1
Their security track record is pretty exceptional compared to the other two.
If Railway isn't satisfied with the explanation, they're able to say so publicly, yes?
They're a web host; it could be any number of plausible mundane things that triggered automated action. This is a big recurring problem for any shared hosting provider.
an entire gcp project deleted along with its persistent disks.
how does that make any sense? nobody thought to call them or anything
No matter how damaging the behavior?
> an entire gcp project deleted along with its persistent disks.
Railway doesn't say that - "persistent disks inaccessible", followed by "persistent disks restored to ready state". It was a suspension, not a wipe.
there's like 0% chance there was domething super damaging going on where they couldn't get anybody on the phone yet within 10 mins or so were able to get the restoration process going with their account managers
I dont see what could be going on where an automated process would have to step in except for something like suddenly provisioning infinite amounts of resources, but quota limits should hit first so...
I'd also expect a story around how it is this happened w/o a human spending at least an hour working his/her way through a call list to reach someone at Railway. Starting with ops and escalating to the ceo if necessary.
If you are saying they should be required to by law, then no I disagree.
But, given that this incident unjustly caused real damages to another company, I am pretty certain that Google will be required to make some sort of response to this, and if it ends up in the courts, it will be public.
That statement is the last 15 or so years.
Maybe AWS is the only player in town now? I don't know. Google doesn't instill confidence with these incidents, same with those cases of insurmountable bills caused by simple mistakes where there should be a way for smaller customers to cap usage.
Unless imposed by employer to use, run a mile.
These sorts of things have happened before with Google and the other expensive providers.
Are cheaper providers known for doing this? I would have thought they would be less lively to, as they are smaller and therefore every customer is relatively more important to them, and they are therefore more likely to check before turning services off.
In my experience, the reason they're cheaper is because they offer fewer features (cut down on ongoing expenses) and because they aggressively enforce TOS (the margins are thinner, so you're less able to afford people using more resources than allocated).
Thanks.
You're already placing blame here.
We don't know the details other than what Railway has said.
If Railway did something wrong, then letting that be known may help other customers avoid the same ~mistake.
Is that possible? Absolutely.
Has Google also done similar things before? Absolutely.
I certainly wouldn't bet my house on either possiblity.
You can't rely your business on GCP. Honestly, this is the most silly way to kill your own business.
For context, copied from my post 3 years ago.
March 10, 2023 | hide | past | favorite As a 4 years customer, our production severs have been suspended by Google Cloud because we didn't fill up some information on-time. Contacted support but they expect us to wait for 24-48 hours to get it resolved while all our servers are down. Anyone linked with someone powerful in google cloud can help?
======
- Running production on google cloud for 4 yrs with my startup. 100% legit SaaS business.
- Always pay bills on-time no issue. Good customer never open tickets, ask for help or what just quietly pay my bills each month.
- Our servers was abruptly suspended yesterday midnight and my whole business is now down for > 10hrs.
- We run a SaaS business that other ecommerce stores rely on and have hundreds of paying merchants.
- My customers have been grilling me and I don't feel gcloud's trust and safety team understand/care how urgent the issue is.
======
Why were our servers suspended? Because we didn't fill up information in time?
- See https://imgur.com/a/x0Y3RJl
- Apparently they dropped us an email 10 days back that I missed out
- Titled "Important Information Regarding Your Google Account" with no indication of suspension or what in title.
- Given the number of subprocessor "Important" emails they send it's too easy to miss out the email.
- 10 days gone by and our servers were abruptly shutdown with zero suspension notification or what.
- We've been paying $400-$700/mo for the past 4 yrs consistently and they shut us down because we didn't fill up some information?
When I tried to ask them to at least temporarily get our servers back while the verification is ongoing, I didn't get any answers.
Google Cloud have zero empathy for customers.
It's not like my account got suspended for fradulent issue or what. It's suspended because I didn't fill up some information on-time and they don't even allow me to temporarily reactivate my services or what. Especially when I had to wait for hours to get their team to verify my details before I can get my servers back.
You can't trust them with your business. Don't run any production stuffs with Google Cloud, ever.
Google/GCP can only make very general statements and in this case we want more than that.
They need to tell Railway and Railway needs to tell us, or Railway can tell us that Google is refusing to tell them.
Either way, we need to hear about this from Railway.
The bigger point though is Google really need to flag any business account as not subject to these suspensions until checked into by several humans. Back when I had a team that used a lot of App Engine they would even call us when we caused all their pagers to go off, and then conspire to keep the lights on while things got fixed. It's sad they have ended up like this.
That case is called:
> or Railway can tell us that Google is refusing to tell them.
I'm not paying you (which let's face it, that's what an NDA is) just to find out why you messed up about as severely as one can imagine. In theory, there was a contract here: $ for cloud services, and the rug got pulled. One should get a very clear, and very apologetic explanation as to why, with no strings attached, or one should be voting with their wallet.
Now, whether Railway will do any of that, who knows.
"We take full responsibility for the architectural decisions that allowed a single upstream provider action to cascade into a platform-wide outage, and detail below what happened, how we recovered, and the changes we are making to prevent this from happening again."
My guess is they will be switching away from Google Cloud. Because anything else would be nuts.
The calibration of the alarms might be off, and that's acceptable, but in the end if something can be held wrong, somebody will hold it wrong.
I did the same thing in the past, albeit in a much smaller scale. There's no shame in being wrong and admitting as long as it results in progress, so this stance of "we do nothing wrong" from both parties is getting a bit old now.
To me, what it sounds like is that a Google Cloud system identified Railway as a misbehaving customer. Spam, hackers, that sort of thing. Often this happens for "platform as a service" companies, because Railway themselves probably do host some spammers and hackers, and they have their own systems for dealing with it.
So, it's quite possible that according to the Google team, Railway violated the terms of something or other, and according to the Railway team, they did not, and now everyone has to argue about it.
But who knows, this is just me guessing based on some experience running a PaaS that itself was running on top of AWS.
Railway plays around vibecoding as they go along, and their tech practices don't inspire confidence either. Unlike other PaaS like Render or Heroku, I doubt Railway has adequate rate-limiting to stop bad apples.
1. There may be no simple rule of thumb like "suddenly using tons of bandwidth"
2. Bad actors can open up so many accounts, you have to close them automatically
3. Malware can infect a good actor, who is unaware or struggling to deal with it
I don't know what happened in this case, there's a chance it wasn't Google's fault but it doesn't matter, Google already lost all benefit of the doubt long ago.
Nobody doubts Google's engineering ability. It's their soft skills that are lacking. Google is culturally unable to understand the concept of customers.
That is a huge problem at Google: "we're google; we know what you want and need better than you do." That and an unshakable confidence in their own infallibility.
When I was there, I remember saying, "we do some cool stuff, but always remember that hubris is the death of empires."
The product is great. However I have GCP blacklisted because it's basically a vending machine. If your drink gets stuck in a vending machine all you can do is hit it hopelessly.
If I have a problem on AWS I'm fairly sure I can talk to a human and resolve it. As GP mentioned AWS support is pro-active and will even reach out to you of their own initiative. This gives me a warm fuzzy feeling.
Amazon understands that cloud services are a trust-based business. Their customers trust them with the power of life or death over their business.
I'm interpreting that bit from Railway's blog to mean it wasn't just them that was impacted.
Every regulated firm running on GCP is going to spend Monday explaining to their board how their resilience plan accounts for a hyperscaler that operates this opaquely. The compliance paperwork is the easy bit - the honest answer is we trusted that a hyperscaler would behave like a utility and they didn't.
So yes - they owe a statement. The whole point of paying hyperscaler prices is the assumption you won't wake up suspended with no explanation.
#1. you will most likely never be as big a customer as railway. if they did it them, you're fucked.
#2. if shit hits the fan, even a company as large as railway can get no human being on the phone. that's insane.
how is gcp still a thing after this event? if you're in charge of tech at your company, how do you stomach choosing this? how do you defend your choice to your CEO?
This is a small unknown startup.
Incident Report: May 19, 2026 – GCP Account Suspension - https://news.ycombinator.com/item?id=48204770 - May 2026 (144 comments)
Previously:
Incident Report: Railway Blocked by Google Cloud [resolved] - https://news.ycombinator.com/item?id=48201484 - May 2026 (344 comments)
This would have been an amazing conversation to witness.
Completely different group of people, culture and incentives.
Google Cloud is another company under Alphabet.
Why would you do that?
Seriously, what are some good reasons to choose GCP over vendors that have demonstrated to be much more reliable? Are they much cheaper than the alternatives? Do they have unbeatable features?