j_ung 0 #1 February 13, 2009 Edit: Scroll down for the latest! Hi all, We're bringing new development and back-up environments online tonight (Friday 2/13), and word is we'll be down around 8PM US-eastern for, more or less, an hour. Very sorry for the inconvenience! Jay Share this post Link to post Share on other sites
j_ung 0 #2 February 18, 2009 Well, we got the back up running just in time, because the main server went down last night. The back up worked, but not ideally, so the Gossamer team is massaging the kinks out now. While we're in there mucking about, we're going upgrade a few other things as well. In the meantime, things are working, but if database queries get a little on the large side, things bottleneck and page-load times slow down. You might even see a proxy error or two. The team is working on it until it's fixed. In the meantime, I've very sorry for yet another inconvenience. Jay Share this post Link to post Share on other sites
j_ung 0 #3 February 18, 2009 Ooookaaay... we're back. The Gossamer boys just put 12 more gigs of ram into the back up, which is why were down for the last 15 minutes. Word is now they're going to work on the main. It's hard to describe how un-thrilled I am, but they're working their asses off right now to stabilize the patient. Bear with us. There will undoubtedly be more hiccups today. Share this post Link to post Share on other sites
j_ung 0 #4 February 18, 2009 Just an update that there's no other update right now. I'm logging off for the night, but I'll check back when I can. Things seem to be working... Share this post Link to post Share on other sites
j_ung 0 #5 February 19, 2009 Okay, I have the official word. Here's what happened: 1. The primary drive of the dropzone database server crashed and was not recoverable. It was an older server. 2. We fell over to our new back up. 3. The back-up could not handle the traffic well, and performance was poor during peak traffic times. 4. Yesterday, our developers upgraded the back up from 4 GB of ram, to 16 GB of ram, and performance improved. It has handled traffic for the last 24 hours or so without issue. 5. We upgraded the main hardware and RAM to give us protection from drive failures and additional performance improvements. As of now, we're still on the back-up, which is working fine. Obviously, though, we need to switch back over to the main server. When that happens, there's going to be another outage for probably 20 minutes. This'll be around 4:30 PM Pacific time, Thursday. Thanks all. I hate, hate, HATE the inconvenience all this caused. Share this post Link to post Share on other sites
j_ung 0 #6 February 20, 2009 We're back up and running! We're on the main server and all appears to be going according to plan. I'll check in again later. G'night! Share this post Link to post Share on other sites