RESTORED: AGS, TS, DR, M4D, KOTF + forums Outage

[[DRuG]NikT]

*edit* 24/8/11 12.30am Systems are back online - TS back up, forums all working.. Jeff may need to restart his zombie server... enjoy folks

From about 4:30pm today the AGS host started rebooting by itself - faster and faster, with no events in the logs, then stopped responding altogether, so is offline.

I managed to get the databases for teamspeak, KOTF and AGS forums before it stopped responding... when I arrived home it had rebooted, so I immediately became suspicious after 8 months of solid uptime.

Backups began immediately after the third reboot - that's about 5 mins after I returned home for a quiet Friday night. - I continued reconnecting for about an hour in between random reboots (with the disk response/data seeming ok when the server was up) until they became more frequent until it eventually stayed offline.

I have managed to save a percentage of the (randomly rebooting) server so far - the databases & Theme folders for DR, KOTF & AGS, Jeff's zombie server, the teamspeak database and license key, the KOTF web.

The last backup of all of AGS I took, 16/12/10 (updated: original was from about a year ago)... wasn't really much different & the databases will have all the new posts, coz I definitely got them Things are looking up the more we back up - your help suggesting things I might miss will really help.

It may just be that the disk is OK, but we won't know that until the host responds. Mouldy has been up > 12 hours and headed to bed, and now so am I, so now we just cross our fingers and hope the disk is ok when we hear back from the host.

More info as it emerges.

[[DRuG]NikT]

I've just managed to get a backup of the zombie and DR servers' databases.... it seems we haven't hosted m4d in a while, or am i just trippin - there's no m4d folder on the ags server.. well there's a user.. has that server been running?

*edit* got the Blu-Tech theme for the DR forums backed up

Also the AGS forum theme & root of /var/www

I've backed up kotf's web, databases & forum theme - blackrain203

I've backed up Slick's www folder - is there anything else you need from here & can u maybe ask sk0t what he needs?

Jeff I got the wblog theme for the zombie forums - Anything else particular anyone wants me to grab?

I've managed to backup your samp03 folder in your user directory.

The ags server is up and down like a yoyo, but im managing to get stuff off it. It's now 2.30am and I'm gonna take a break until morning, I've been at it solid since I got home..not even eaten.

[[DRuG]Slick]

Cheers Nik, you won't have to worry about my stuff, the PictureStack files are no longer required there and the rest is in the main var/www dir which you probably already have.

I am pretty sure sk0t has everything of his backed up, maybe even more. Let me know if you're unable to get everything of importance as we might have a copy ourselves.

[[DRuG]NikT]

Yeh I've just backed up all of DRuG's forums, web, databases, mta server + web, everything.. on a removable drive that is usually off.

I managed to get pretty much everything @ AGS (yes I did get a solid copy of your www folder slick) in terms of live forum databases etc I think... but the host isn't responding any more so I damn hope so.

I'm gonna help Mouldy find a provider that offers incremental backups after this... he's losing more than 1 drive a year.. it's getting stupid.

[]DR[Jeff]

Thanks for that NikT. I think that is all I guess, does the backup include all the posts and settings of the zombie section?

[[DRuG]NikT]

yep.. and the theme 8)

worst case scenario if the box has totally assploded HDD and all, we can do a fresh install of smf then import the database backup.

[]DR[Jeff]

The worse case scenario sounds brutal lol. Thanks again for getting the data off

[[DRuG]Ryant]

cheers nikt, mouldy, everyone involved

[[M4D]FULL_METAL]

M4D is fine, as we have everything through comatose now, so no worries from us

hope everything goes well nikt and mouldy

[sk0t]

I just thought I should make an appearance here as most of the stuff on the box is I was managing, before I left.

As long as the following things we're backed up:

[*]The entire /var/www DIR (Not just the forum themes the whole show I made major major modifications to the whole of SMF and that version of SMF isn't available anymore and I think it's safe to say most of the mods we used aren't too)
[*]The /home/gta DIR
[*]The /home/zombie DIR
[*]And all the SQL databases

You guys should be fine everything else on the box was useless or no longer used.

Also I'd like to just say that, the issues with the server are not caused by our host I can vouch for the validity of the servers VD's hosting with the following uptime taken from a server about two servers down from the AGS one with the same specs: 22:36:18 up 403 days, 22:04, 0 users, load average: 27.96, 27.47, 26.52 - That's my data processing server it's constantly writing to the hard-drive and sending out data packets and I've never had an issue with it... EVER.

It cannot be a matter of bad luck this is the four time we've had an issue like this happen... To be totally honest I saw it coming that's why I begged Mouldy to allow me to re-install the services and start again, the box was neglected, out-dated and under pressure contently by unnecessary packages and incorrectly config'd services.

Anyway its up-to you guys now to solve this problem but I can be contact on Steam by KF if you want my help doing it, nice work taking the back ups I hope you took enough and if you didn't that VD can restore the hard-drive. If Mouldy can he should try and upgrade the server to something with RAID-1 instead of looking for another host that takes back ups, that shouldn't fall back on the host that should fall back on us and not many hosts do it for dedicated server because of privacy issues. You'd be lucky to find another server with the specs at VD in the same price range anyway.


Side note I noticed I'm still listed as member on the DRuG members page I also still have a members tag here on the forums could that be fixed please?