+- Southperry.net (https://www.southperry.net)
+-- Forum: Main (https://www.southperry.net/forumdisplay.php?fid=11)
+--- Forum: Site Announcements (https://www.southperry.net/forumdisplay.php?fid=18)
+--- Thread: Extended Outage (/showthread.php?tid=65354)
Extended Outage - Eos - 2013-06-30
Long story short, an entire street of data centers in Atlanta lost power and every host, and thereby every hosting company and every hosted site, went out. The outage was long enough that UPS' couldn't have sustained them but could've bought them time to do graceful shutdowns, had they had any, and of course any major host normally has actual backup generators to run indefinitely during outage, but apparently our latest had nothing whatsoever, so we went down hard, for a good six hours, and when we came up the database was understandably upset and had to be talked back down off a ledge before it'd run.
Once the database was restored we were still dead though because the ipv6 bridge between our web server and db server stopped functioning. Somewhere in the provider's network is a piece of equipment someone forgot to turn back on I think, so that had to be hunted down and traced through to determine whether it was something that died in our configs or an actual network issue as it was, and then we had to reroute everything back to ipv4 until it can be corrected.
Now we're back up and semi running while I try to find any gaping holes in our data other than whatever was occurring at the exact minute we died.
Extended Outage - Arrol - 2013-06-30
Oh I know that feel. My power went off yesterday as well.
Extended Outage - ayumilove - 2013-06-30
Good timing! I was refreshing this site a few times and I'm glad is back up
Waited a couple of minutes when it was down.
Extended Outage - SoR0XaS - 2013-06-30
oh, so that's why I tried to access the site since 6 hours ago and failing miserably.
Glad the site's back up!
Extended Outage - byakugan - 2013-06-30
Time to get back on my addiction to this site U_u
Extended Outage - Five Second Pose - 2013-06-30
ayumilove Wrote:Good timing! I was refreshing this site a few times and I'm glad is back up
SoR0XaS Wrote:oh, so that's why I tried to access the site since 6 hours ago and failing miserably.
Glad the site's back up!
protip : to check if a site's down use http://www.isup.me
Extended Outage - Eos - 2013-06-30
Five Second Pose Wrote:protip : to check if a site's down use http://www.isup.me
I prefer http://isitup.org/
Extended Outage - Eos - 2013-06-30
A better explanation of how exactly this managed to happen for those who were as baffled as I;
Quote:The cause of the loss of utility power is still being investigated. Approximately 35 minutes after the datacenter switched to generator power the generator failed, and subsequently the UPS battery backup power was exhausted about 15 minutes later. Approximately 2 hours after this occured, the generator was repaired and services were mostly online overnight until around 11 am local time, when the generator experienced another failure. Approximately 1 hour later utility power was restored to part of the datacenter, including the part that we are in, and we have been online since that time
And as a nice relief;
Quote:As this is the most severe outage our Atlanta location has experienced in the last 3 years since we launched services here, we have decided to credit all clients in Atlanta with 1 free month of service, which should be reflected right now in your service next due date. Even though we did not cause this outage we reconize that any failure of this magnitude regardless of fault is unacceptible and we take full responsibility for the outage you have experienced yesterday evening and again this morning.
That'll definitely help us with the recent issues we've had with advertising outages and whatnot on top of this.
Extended Outage - KhainiWest - 2013-07-01
It's refreshing to see customers treated as customers
Extended Outage - Polantaris - 2013-07-04
Quote:Approximately 35 minutes after the datacenter switched to generator power the generator failed, and subsequently the UPS battery backup power was exhausted about 15 minutes later. Approximately 2 hours after this occured, the generator was repaired and services were mostly online overnight until around 11 am local time, when the generator experienced another failure.
I hope they're shopping for a new generator.
Extended Outage - D2Girls - 2013-07-10
There were power outages for all kinds of services all accross North America. My cellphone provider lost service and my phone has been pretty much useless for the past two days. Thank god its fixed now.