Apache Server Errors

So I run a server at home hosting several sites, including a few personal sandboxes for development. I went to check one of sites, and noticed it was taking an incredibly long time to load. When I logged into the machine, I got the dreaded Usage of /: 99.9% of 27.50GB disk space error. Now, this can happen, and for all sorts of reasons. Maybe a program got overzealous writing out logs, or backing up files, or any one of a thousand rouge operations. I have run into this error before, and thankfully I had a few quick commands to run to determine the reason.

Find Largest Folders

	du -kx / |sort -nr|head -10

Find Largest Files

	find . -type f -exec ls -s {} \; | sort -n -r | head -5

Find Largest Folders and Files

	du -a /var | sort -n -r | head -n 10

Problem

Eventually I found the culprit in my apache error and access logs. After some poking around in the huge files and rolled over logs, I determined that some bots were trying to gain access to the admin sections of one of my hosted sites. I found hundreds of thousands of lines looking like the below (identifying information has been removed).

	[Fri Aug 16 22:34:52 2013] [error] [client XX.XX.XX.XX] File does not exist: /var/www/XXXXXXXXXX/phpMyAdmin
	[Fri Aug 16 22:34:52 2013] [error] [client XX.XX.XX.XX] File does not exist: /var/www/XXXXXXXXXX/PMA
	[Fri Aug 16 22:34:52 2013] [error] [client XX.XX.XX.XX] File does not exist: /var/www/XXXXXXXXXX/pma
	[Fri Aug 16 22:34:53 2013] [error] [client XX.XX.XX.XX] File does not exist: /var/www/XXXXXXXXXX/admin
	[Fri Aug 16 22:34:53 2013] [error] [client XX.XX.XX.XX] File does not exist: /var/www/XXXXXXXXXX/dbadmin
	[Fri Aug 16 22:34:53 2013] [error] [client XX.XX.XX.XX] File does not exist: /var/www/XXXXXXXXXX/sql
	[Fri Aug 16 22:34:53 2013] [error] [client XX.XX.XX.XX] File does not exist: /var/www/XXXXXXXXXX/mysql
	[Fri Aug 16 22:34:54 2013] [error] [client XX.XX.XX.XX] File does not exist: /var/www/XXXXXXXXXX/myadmin

Not only were these attacks filling up my server with garbage in the log files, but they were causing degradation in page load times. I had already setup a robots.txt file to keep proper crawlers in the correct locations, so I needed to come up with a better solution.

Solution

After much analysis of the issue, I determined that all of these attacks were coming from overseas from Chinese and Russian IPs based on IP lookups. After some more research I decided to try implementing mod_geoip2. While this will disallow any traffic from the countries I will disallow, I examined the usual traffic using Google Analytics, and determine that no traffic generally originates from either country I was having issues with. I followed the installation instructions on maxmind. Based on the examples provided, I setup an .htaccess file looking like the below:

	GeoIPEnable On
	GeoIPScanProxyHeaders On
	GeoIPDBFile /etc/apache2/GeoIP.dat

	SetEnvIf GEOIP_COUNTRY_CODE CN BlockCountry
	SetEnvIf GEOIP_COUNTRY_CODE RU BlockCountry

	Deny from env=BlockCountry

Conclusion

So far my logs have been clean. Other than the occasional bad link from old/moved/cached files, I have had no requests for files that don’t exist. Performance issues can be a problem, but I haven’t noticed any degraded performance within the system. In fact, I’ve noticed much quicker response from the pages loading.