PUBLIC FORUMS/GENERAL FORUM

Topic Title: Server Downtime (Simple Topic)

Topic starter: System Administrator

Topic started: 00:24:40 31st Aug 2010

Posts: 10 Last post: 12:36:53 1st Sep 2010 by EL

System Administrator Posted: 00:24:40 31st Aug 2010

Posts: 48

Topics: 50

Location: United States



Has passed. I'm exhausted. Been sleeping far too little these past few days - a rundown of what happened therefore.

I have SkyLords set to replicate to my desktop computer, just incase anything should fail on the hard disk, I have backups to the very moment it failed, if I am booted into Linux. If I am booted into Windows mode, those updates are saved. I have to reboot weekly into Windows mode on my desktop computer however, what was the weekly downtime during the (fortunately briefish) time SL was hosted from my desktop - nobody was very fond of that solution, and it's permanent home is much nicer anyway, without pesky things like graphics cards and speakers, monitors and mice.

Well, as is amazingly common, we were under an attack from the internet on Thursday, some dumb cracker trying to guess our passwords, 10 per second or the like. Relatively boring stuff - but it was filling up our log files with failed login attempts. What ended up happening was a combination of factors which should never occur again. On Thursday morning, the disk filled up. This caused the white screen that many of you saw for a few days while I tried to free up disk space.

However, there was a lot of things, cron jobs mostly, waiting to happen. These sucked up disk space as fast as I could reclaim it, until finally I had most of the jobs pushed through - however, there were three jobs, written during my absence, I'll leave it to you to imagine how I feel about the author, that didn't fail gracefully - they continued to suck up disk space in an infinite loop. I found and fixed these final points on Friday night(?).

I wasn't content however to just leave it at that - I wanted to ensure that this triple disk hogging couldn't ever use up all of the 7 gb allocated... Far, far, too little, but I didn't think about that when I set it up back when. So, I tried to resize the disk. Again and again, I tried various tools. Finally giving up, I opened the case, pulled the disk, and resized it from my desktop machine :P.

When I put it back in however... It didn't boot. This was Saturday night around 3 am, and I needed to hit church the next day (was excellent however). Since I decided to let it rest for a day, I got back to it this evening after work (I was late to work already this morning, not able to work on the server). Now, past midnight, with less than 24 hours till the end of the round, I have fixed the booting problem (it was a strange kernel update that broke compatibility with SATA for some motherboards - I was unlucky to have the update from the brief time it wasn't fixed).

SkyLords now is running off of a ... much larger partition on a borrowed disk, and should be fine there for the foreseeable future. Just a note - before the downtime, the SkyLords server was running fine for almost a full 70 days without a single interruption. No reboots, hiccups, or needing to restart apache. SkyLords is running on some pretty awesome hardware, with a very wide Internet connection, and I dare you to NOT try to disrupt it please thank you very much :]. We are safe from any more accidental DOS attacks now however :P..

Thank you for your patience.

~Iiridayn


Omnibus Posted: 02:17:08 31st Aug 2010

Posts: 30

Topics: 10

Gender: Male



Thanks Iiri!!!


TallDude3rd Posted: 03:12:27 31st Aug 2010

Posts: 1850

Topics: 37

Location: Netherlands

Gender: Male



Bruteforce attack on our server? That is just stupid




__________________________
UNOFFICIAL BIG KAHUNA OF THE UNIVERSE

EL Posted: 03:25:42 31st Aug 2010

Posts: 1705

Topics: 182

Location: United Kingdom

Gender: Male



So glad you got it fixed Iiri thank you :)

Yes it it pretty stupid attacking the server but at least we are now prepared for any future attacks


Tom Bomadial Posted: 05:11:13 31st Aug 2010

Posts: 423

Topics: 76

Location: United States

Gender: Male



Thank you.
Not only for fixing the system.
But also for going to church.




__________________________
"Is not easily provoked"
Translated:
It is not good to wake a sleeping bear

TallDude3rd Posted: 06:10:00 31st Aug 2010

Posts: 1850

Topics: 37

Location: Netherlands

Gender: Male



He is just going to church to program a text-speech so the preacher-man can take a break when reading the bible... lol




__________________________
UNOFFICIAL BIG KAHUNA OF THE UNIVERSE

Zerohours Posted: 10:04:37 31st Aug 2010

Posts: 2443

Topics: 287

Location: Canada

Gender: Male



TY Iiri




__________________________
I am here to help in anyway I can! Let's have fun!

Pray Posted: 17:02:34 31st Aug 2010

Posts: 2255

Topics: 147

Location: United States

Gender: Male



LOL I-Flight for sermons. Extreme & Deep.




__________________________
A storm is coming, a storm is coming in..

20:48:19 | Professor planet 0 from paladin
20:20:21 | Arcanix planet 14 Arcanix 01

Diplomacy & Trading Forum Moderator.



me naam is m Posted: 04:10:17 1st Sep 2010

Posts: 985

Topics: 37

Location: Netherlands

Gender: Male



*lol @ TD+Pray*

Anyway, I am wondering what or who was trying to ddos?




__________________________
Where has the rum gone!?

FORTRANshadow Posted: 08:36:19 1st Sep 2010

Posts: 383

Topics: 17

Location: United States

Gender: Male



Thanks, Iiri! I know how hard it is to keep things going in the big bad internet. Please know that we appreciate your efforts.

You may want to look at installing denyhosts on your server. Although it is normally configured just for ssh, I think the scripts can be set to review all login attempts in your auth.log.


EL Posted: 12:36:53 1st Sep 2010

Posts: 1705

Topics: 182

Location: United Kingdom

Gender: Male



It would work.


Previous 1  Next

PUBLIC FORUMS
TICKETS
LANGUAGE FORUMS
SKYLORDS CHAT