A day that made me curse my existence..

So last night one of the web servers at my job took a dump.  It ended up coming back up before I could take a look, so I just kept an ear out for trouble…

This morning, same server drops…  I reset it since I can’t actually get into it, some time goes by, another server drops…  I check the firewall – nothing really standing out, so I follow up on some errors in the server logs, and make some suggested performance tweaks…  Then after a quiet hour or two… the feces hit the oscillator.

Two specific webservers start getting SLAMMED.  That cascades in a way that affects a few other servers.  Now mind you, the root of this entire problem is a denial of service attack. Once I stop that, we’re good.  I dig into the firewall, and start to identify the attackers.  As they hit me, I’m blocking them.  It’s tedious, but besides limiting the number of connections per second, my only real recourse.

In the middle of all of this, identifying attackers, blocking them, bringing the servers back up, and trying to make sure that this all can’t happen again… I get interrupted at least every 20 minutes by someone who doesn’t understand that the problems they want me to fix are being CAUSED by the f’n DoS attack.

I finally manage to block the attack after nearly screaming at the person who kept interrupting, and in the process manage to mess something up with my firewalls.  I spend a few more hours trying to fix what I messed up, but decide to call it a night, go home for some dinner and a bit of a break before I head back in after everyone’s gone home to fix the problem.

I call in my consultant friend, he tries for about 40 minutes but kicks it up to support.  I call support, my contract is for Monday through Friday from 8am-5pm…. it’s now 10pm.  Still, I explain my story to the tech, and he helps me… by about 10:40, the firewalls are back to normal… no server issues since the late afternoon…peace and relative quiet.

Only redeeming fact of this entire day was being out on the bike in the cool, clear evening air.  Was a great way to decompress from the stress of all this BS.

Hoping tomorrow that I’ll get out of work around lunch, go to the gym, then come up and make some GF cupcakes for the potluck at my church.  ~fingers crossed~