Automatically Restart HTTPD when server overloads

November 8, 2010

I have several dedicated servers running WHM with cpanel on centos. I sell web space to the general public who with them bring their own developers and programmers and of course crappy code.

Quite often there will be sites that just EAT up CPU and of course I only know about it when the servers overloads itself and crashes. I do have CSF installed on all servers which does have an early warning system for server load, but if I am asleep in the middle of the night, its no help.

So, how can I prevent cpu overload on my server and bring my server back to normal use – AUTOMATICALLY?

After thoroughly looking around and it took me some time, I finally found some scripts I was looking for.

The scripts below is run by cron job every 60 seconds and detects the current server load. If the server load is above 25 then it will automatically stop the httpd server, send me an email informing me. Then 60 seconds afterwards, o the next cron run, it will check the server load again and if the server is back down to a load of 4 (an acceptable load on our system), the httpd service will automatically start again.

Usually once httpd is stopped, the load drops down within 60/100 seconds so there is hardly anytime down time, but it does give you time to investigate where the issue is, especially if it happens all the time.

Totally perfect! It means that my system will not grind to a halt without me knowing. Its saved me a few times this month already since implementing it.

Obviously its not the be all and end all perfect solution, but it works and prevents serious overloading of the server.

To install, run the following in ssh and this will pull the file from my server over to yours.

mv /root/loadmonitor.txt /root/
chmod 755 /root/

Then you need to add the cron to the cron tab so that it checks your server every 60 seconds.
Follow these instructions to edit the root cron file

1. open the cron file
pico /var/spool/cron/root

2. add the following line to the cron file so it runs every minute
* * * * * /root/ > /dev/null 2>&1

3. restart the cron with this command
/etc/init.d/crond restart

Thats it!

The only thing you need to do now is change a couple of settings within the file to set your email address.

Do the following to edit your settings.

pico /root/

This will open the file and then you only need to edit the following settings:

my $contacts – the email address to send alerts to
my $from_address – the custom name of the server so you know which server its coming from