Amazon’s Cloud Is Down Again, Taking Heroku and GitHub With It

Some cloud computing operations from Amazon Web Services have crashed, taking down services as varied as Heroku and GitHub with it. (For awhile it appeared GitHub had gone down too but that wasn’t the case. See the update below.)

Amazon’s EC2 operations at its data center in Northern Virginia appears to be the culprit, according to its status board. The last time something like this happened was in July, when a significant electrical storm took out backup power at the same location, and knocked AWS offline for a weekend. In that outage, sites including Instagram, Pinterest and Netflix all crashed.

Here’s the message from the status board for the Elastic Cloud Compute service, a.k.a. EC2:

11:26 AM PDT We are currently experiencing degraded performance for EBS volumes in a single Availability Zone in the US-EAST-1 Region. New launches for EBS backed instances are failing and instances using affected EBS volumes will experience degraded performance.

11:11 AM PDT We can confirm degraded performance for a small number of EBS volumes in a single Availability Zone in the US-EAST-1 Region. Instances using affected EBS volumes will also experience degraded performance.

10:38 AM PDT We are currently investigating degraded performance for a small number of EBS volumes in a single Availability Zone in the US-EAST-1 Region.

Also this, regarding Amazon’s relational database service:

11:03 AM PDT We are currently experiencing connectivity issues and degraded performance for a small number of RDS DB Instances in a single Availability Zone in the US-EAST-1 Region.

And finally this, from Amazon’s Elastic Beanstalk service:

11:45 AM PDT We are continuing to see delays launching, updating and deleting Elastic Beanstalk environments in the US-East-1 Region.

11:06 AM PDT We are currently experiencing elevated API failures and delays launching, updating and deleting Elastic Beanstalk environments in the US-East-1 Region.

More as it becomes available.

Update: True to the history of prior Amazon Web Services outages, scores of other services that depend on AWS crashed along with it. Among them Reddit, AirBNB, Pinterest, Foursquare, but also smaller sites like ifttt.com, Foopets.com. The launch also interfered with the launch plans of a site called AssetsintheSeats.com. See the Tweet below:

Here’s the latest from Amazon’s status board as of 2:20 PM Pacific Time:

2:20 PM PDT We’ve now restored performance for about half of the volumes that experienced issues. Instances that were attached to these recovered volumes are recovering. We’re continuing to work on restoring availability and performance for the volumes that are still degraded.

We also want to add some detail around what customers using ELB may have experienced. Customers with ELBs running in only the affected Availability Zone may be experiencing elevated error rates and customers may not be able to create new ELBs in the affected Availability Zone. For customers with multi-AZ ELBs, traffic was shifted away from the affected Availability Zone early in this event and they should not be seeing impact at this time.

Update: GitHub was the subject of some early initial reports as having been affected. Apparently itwasn’t GitHub itself that went down, but some aspect of its status report page was affected, as you can see here:

02:17 PM PST

The status website is misreporting availability of services again. Git and Downloads are both available.

Here’s a funny example of a residual outage. A bartending app called GetOnTheBar crashed for awhile, but is as of this writing starting to come back online. It’s hosted on Heroku, which was of course affected by the AWS outage. See the Tweet below.