Google Cloud Is Down

https://status.cloud.google.com/incident/compute/19003Status page reports all green, however the outage is affecting YouTube, Snapchat, and thousands of other users.Status page reports all green, however the outage is affecting YouTube, Snapchat, and thousands of other users.We're having what appears to be a serious networking outage. It's disrupting everything, including unfortunately the tooling we usually use to communicate across the company about outages.There are backup plans, of course, but I wanted to at least come here to say: you're not crazy, nothing is lost (to those concerns downthread), but there is serious packet loss at the least. AWS tries to lock people in to specific services now which makes it really difficult to migrate. https://aws.amazon.com/s3/sla/Here are the S3 numbers: https://aws.amazon.com/s3/sla/ I understand this is long-since resolved (I haven't tried building a service on Amazon in a couple years, so this isn't personal experience), but centralized failure modes in decentralized systems can persist longer than you might expect.(Work for Google, not on Cloud or anything related to this outage that I'm aware of, I have no knowledge other than reading the linked outage page.)(Work for Google, not on Cloud or anything related to this outage that I'm aware of, I have no knowledge other than reading the linked outage page.) I don’t think the networking SRE folks are as impacted in their direct communication, but they are (obviously) not able to get the word out as easily.Still, it seems reasonable to me to use tooling for most outages that relies on “the network is fine overall”, to optimize for the common case.Note: the status dashboard now correctly highlights (Edit: with a banner at the top) that multiple things are impacted because Networking. The Networking outage is the root cause.Still, it seems reasonable to me to use tooling for most outages that relies on “the network is fine overall”, to optimize for the common case.Note: the status dashboard now correctly highlights (Edit: with a banner at the top) that multiple things are impacted because Networking. The Networking outage is the root cause.Note: the status dashboard now correctly highlights (Edit: with a banner at the top) that multiple things are impacted because Networking. AWS refunded me in the first reply on the same day!GCP sales rep just copy pasted a link to a self support survey that essentially told me, after a series of YES or NO questions that they can't refund me.So why not just tell your customers like it is? Google Cloud is super strict when it comes to billing. Apparently the $300 Trial with Google Cloud did not include some critical products, AWS Free tier makes it super clear and even still I sometimes leave something running on and discover it in my invoice....I've yet to receive a reply from Google and its been a week now.I do appreciate other products such as Firebase but honestly for infrastructure and for future integration with enterprise customers I feel AWS is more appropriate and mature.GCP sales rep just copy pasted a link to a self support survey that essentially told me, after a series of YES or NO questions that they can't refund me.So why not just tell your customers like it is? Google Cloud is super strict when it comes to billing. Apparently the $300 Trial with Google Cloud did not include some critical products, AWS Free tier makes it super clear and even still I sometimes leave something running on and discover it in my invoice....I've yet to receive a reply from Google and its been a week now.I do appreciate other products such as Firebase but honestly for infrastructure and for future integration with enterprise customers I feel AWS is more appropriate and mature.So why not just tell your customers like it is? Google Cloud is super strict when it comes to billing. Apparently the $300 Trial with Google Cloud did not include some critical products, AWS Free tier makes it super clear and even still I sometimes leave something running on and discover it in my invoice....I've yet to receive a reply from Google and its been a week now.I do appreciate other products such as Firebase but honestly for infrastructure and for future integration with enterprise customers I feel AWS is more appropriate and mature.I'm now back to AWS and still on a Free Tier. Apparently the $300 Trial with Google Cloud did not include some critical products, AWS Free tier makes it super clear and even still I sometimes leave something running on and discover it in my invoice....I've yet to receive a reply from Google and its been a week now.I do appreciate other products such as Firebase but honestly for infrastructure and for future integration with enterprise customers I feel AWS is more appropriate and mature.I've yet to receive a reply from Google and its been a week now.I do appreciate other products such as Firebase but honestly for infrastructure and for future integration with enterprise customers I feel AWS is more appropriate and mature.I do appreciate other products such as Firebase but honestly for infrastructure and for future integration with enterprise customers I feel AWS is more appropriate and mature. https://www.whoishostingthis.com/#search=status.cloud.google... It's always interesting to see these outages at large cloud providers spider out across the rest of the internet, a lot of the world depends on Google to stay up. https://techcrunch.com/2018/02/27/apple-now-relies-on-google... https://downdetector.com/status/level3 https://www.zdnet.com/article/some-internet-outages-predicte... https://status.cloud.google.com/incident/cloud-networking/19...Incident #19009 began at 2019-06-02 12:53. https://status.cloud.google.com/incident/cloud-networking/19...Times are US/PacificIncident #19009 began at 2019-06-02 12:53. https://status.cloud.google.com/incident/cloud-networking/19...Times are US/PacificTimes are US/Pacific It might be something security related if it triggers a mandatory identity confirmation.edit: I tried to send me a mail from another account and it worked but out of 4 or 5 mail checks at least two failed giving the same error.[23:44:27] POP< -ERR [SYS/TEMP] Temporary system problem. Please try again later.The problem seems much more complex.edit: I tried to send me a mail from another account and it worked but out of 4 or 5 mail checks at least two failed giving the same error.[23:44:27] POP< -ERR [SYS/TEMP] Temporary system problem. Please try again later.The problem seems much more complex.[23:44:27] POP< -ERR [SYS/TEMP] Temporary system problem. Unless something is really fucked (like both GCP and AWS being down for us-east) incidents like these are not going to impact them at all.The cost of either migrating to the other provider or, even worse, migrating to more traditional hosting companies is enormous and will require much more than "service was down for 2 hours in 2019". The contracts also cover cases like this and even if they don't, Google and Amazon can and will throw in some free treat as an apology.On one hand I find this quite sad, but from a pragmatic point of view it makes sense.The cost of either migrating to the other provider or, even worse, migrating to more traditional hosting companies is enormous and will require much more than "service was down for 2 hours in 2019". The contracts also cover cases like this and even if they don't, Google and Amazon can and will throw in some free treat as an apology.On one hand I find this quite sad, but from a pragmatic point of view it makes sense.On one hand I find this quite sad, but from a pragmatic point of view it makes sense. AWS had the S3 incident affecting all of us-east-1: “Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable.”https://aws.amazon.com/message/41926/https://aws.amazon.com/message/41926/ I'm overall happy with it, but if I needed to run a service with a 99.95% uptime SLA or higher, I wouldn't rely solely on GCP. https://www.google.com/appsstatus#hl=en&v=statushttps://i.imgur.com/pcqwwA4.pnghttps://i.imgur.com/pcqwwA4.png There appears to be some irregularities on consumer services as well that are of course certainly related, youtube was behaving a bit oddly for me.The impact seems to be cascading down from just GCE to other services as well - that status page certainly does not reflect the reality of the situation. You can't even sign into GCP right now, and things that run on GCE, like appengine seem impacted.The impact seems to be cascading down from just GCE to other services as well - that status page certainly does not reflect the reality of the situation. You can't even sign into GCP right now, and things that run on GCE, like appengine seem impacted. Stuff like this could make you loose your bussiness, especially if some entity like google doesn't communicate and as a result, you do not have a answer for your own customers.Medium sized private cloud providers are a lot better at this, considering the communication lines are a lot shorter.Medium sized private cloud providers are a lot better at this, considering the communication lines are a lot shorter.