Forums: General: Announcements:
DZ.com Downtime: THURSDAY 2/19, 4:30PM Pacific

 


j_ung

Feb 13, 2009, 6:43 AM
Post #1 of 6 (3861 views)
Shortcut
     DZ.com Downtime: THURSDAY 2/19, 4:30PM Pacific  

Edit: Scroll down for the latest!

Hi all,
We're bringing new development and back-up environments online tonight (Friday 2/13), and word is we'll be down around 8PM US-eastern for, more or less, an hour. Very sorry for the inconvenience!
Jay


(This post was edited by j_ung on Feb 19, 2009, 9:03 AM)


j_ung

Feb 18, 2009, 7:25 AM
Post #2 of 6 (3526 views)
Shortcut
     Re: [j_ung] DZ.com Downtime: Friday 2/13, 8PM Eastern [In reply to]  

Well, we got the back up running just in time, because the main server went down last night. The back up worked, but not ideally, so the Gossamer team is massaging the kinks out now. While we're in there mucking about, we're going upgrade a few other things as well.

In the meantime, things are working, but if database queries get a little on the large side, things bottleneck and page-load times slow down. You might even see a proxy error or two. The team is working on it until it's fixed. In the meantime, I've very sorry for yet another inconvenience.

Jay


j_ung

Feb 18, 2009, 9:29 AM
Post #3 of 6 (3519 views)
Shortcut
     Re: [j_ung] DZ.com Downtime: Friday 2/13, 8PM Eastern [In reply to]  

Ooookaaay... we're back. The Gossamer boys just put 12 more gigs of ram into the back up, which is why were down for the last 15 minutes. Word is now they're going to work on the main.

It's hard to describe how un-thrilled I am, but they're working their asses off right now to stabilize the patient. Bear with us. There will undoubtedly be more hiccups today.


j_ung

Feb 18, 2009, 3:03 PM
Post #4 of 6 (3507 views)
Shortcut
     Re: [j_ung] DZ.com Downtime: Friday 2/13, 8PM Eastern [In reply to]  

Just an update that there's no other update right now. I'm logging off for the night, but I'll check back when I can. Things seem to be working...


j_ung

Feb 19, 2009, 9:07 AM
Post #5 of 6 (3484 views)
Shortcut
     Re: [j_ung] DZ.com Downtime: Friday 2/13, 8PM Eastern [In reply to]  

Okay, I have the official word.

Here's what happened:

1. The primary drive of the dropzone database server crashed and was not recoverable. It was an older server.

2. We fell over to our new back up.

3. The back-up could not handle the traffic well, and performance was poor during peak traffic times.

4. Yesterday, our developers upgraded the back up from 4 GB of ram, to 16 GB of ram, and performance improved. It has handled traffic for the last 24 hours or so without issue.

5. We upgraded the main hardware and RAM to give us protection from drive failures and additional performance improvements.

As of now, we're still on the back-up, which is working fine. Obviously, though, we need to switch back over to the main server. When that happens, there's going to be another outage for probably 20 minutes. This'll be around 4:30 PM Pacific time, Thursday.

Thanks all. I hate, hate, HATE the inconvenience all this caused. Unimpressed


j_ung

Feb 19, 2009, 5:43 PM
Post #6 of 6 (3457 views)
Shortcut
     Re: [j_ung] DZ.com Downtime: Friday 2/13, 8PM Eastern [In reply to]  

We're back up and running! We're on the main server and all appears to be going according to plan. I'll check in again later. G'night!



Forums : General : Announcements

 


Search for (options)