0
j_ung

DZ.com Downtime: THURSDAY 2/19, 4:30PM Pacific

Recommended Posts

Edit: Scroll down for the latest!

Hi all,
We're bringing new development and back-up environments online tonight (Friday 2/13), and word is we'll be down around 8PM US-eastern for, more or less, an hour. Very sorry for the inconvenience!
Jay

Share this post


Link to post
Share on other sites
Well, we got the back up running just in time, because the main server went down last night. The back up worked, but not ideally, so the Gossamer team is massaging the kinks out now. While we're in there mucking about, we're going upgrade a few other things as well.

In the meantime, things are working, but if database queries get a little on the large side, things bottleneck and page-load times slow down. You might even see a proxy error or two. The team is working on it until it's fixed. In the meantime, I've very sorry for yet another inconvenience.

Jay

Share this post


Link to post
Share on other sites
Ooookaaay... we're back. The Gossamer boys just put 12 more gigs of ram into the back up, which is why were down for the last 15 minutes. Word is now they're going to work on the main.

It's hard to describe how un-thrilled I am, but they're working their asses off right now to stabilize the patient. Bear with us. There will undoubtedly be more hiccups today.

Share this post


Link to post
Share on other sites
Okay, I have the official word.

Here's what happened:

1. The primary drive of the dropzone database server crashed and was not recoverable. It was an older server.

2. We fell over to our new back up.

3. The back-up could not handle the traffic well, and performance was poor during peak traffic times.

4. Yesterday, our developers upgraded the back up from 4 GB of ram, to 16 GB of ram, and performance improved. It has handled traffic for the last 24 hours or so without issue.

5. We upgraded the main hardware and RAM to give us protection from drive failures and additional performance improvements.

As of now, we're still on the back-up, which is working fine. Obviously, though, we need to switch back over to the main server. When that happens, there's going to be another outage for probably 20 minutes. This'll be around 4:30 PM Pacific time, Thursday.

Thanks all. I hate, hate, HATE the inconvenience all this caused. :|

Share this post


Link to post
Share on other sites
0