Feb 13, 2009, 6:43 AM
Post #1 of 6
(4915 views)
Shortcut
DZ.com Downtime: THURSDAY 2/19, 4:30PM Pacific
Edit: Scroll down for the latest!
Hi all, We're bringing new development and back-up environments online tonight (Friday 2/13), and word is we'll be down around 8PM US-eastern for, more or less, an hour. Very sorry for the inconvenience! Jay
(This post was edited by j_ung on Feb 19, 2009, 9:03 AM)
Well, we got the back up running just in time, because the main server went down last night. The back up worked, but not ideally, so the Gossamer team is massaging the kinks out now. While we're in there mucking about, we're going upgrade a few other things as well.
In the meantime, things are working, but if database queries get a little on the large side, things bottleneck and page-load times slow down. You might even see a proxy error or two. The team is working on it until it's fixed. In the meantime, I've very sorry for yet another inconvenience.
Ooookaaay... we're back. The Gossamer boys just put 12 more gigs of ram into the back up, which is why were down for the last 15 minutes. Word is now they're going to work on the main.
It's hard to describe how un-thrilled I am, but they're working their asses off right now to stabilize the patient. Bear with us. There will undoubtedly be more hiccups today.
1. The primary drive of the dropzone database server crashed and was not recoverable. It was an older server.
2. We fell over to our new back up.
3. The back-up could not handle the traffic well, and performance was poor during peak traffic times.
4. Yesterday, our developers upgraded the back up from 4 GB of ram, to 16 GB of ram, and performance improved. It has handled traffic for the last 24 hours or so without issue.
5. We upgraded the main hardware and RAM to give us protection from drive failures and additional performance improvements.
As of now, we're still on the back-up, which is working fine. Obviously, though, we need to switch back over to the main server. When that happens, there's going to be another outage for probably 20 minutes. This'll be around 4:30 PM Pacific time, Thursday.
Thanks all. I hate, hate, HATE the inconvenience all this caused.
DZ.com Downtime: THURSDAY 2/19, 4:30PM Pacific
Thread Options
