OMNINOGGIN

Every time I search the web for information on how to block spam bots, scrapers, and harvesters, I always see an Apache .htaccess file or some code to dump into httpd.conf to achieve this. I’m a bit against using this method for blocking evil bots. I do respect Apache for being a flexible & modular web server (that’s why I still use it), but I do not have much to boast about Apache’s speed and efficiency.

To achieve the same purpose on my server with greater efficiency, I made use of my Varnish reverse proxy configurations (located under /usr/local/etc/varnish/default.vcl).

In this post, I will only be discussing about vcl_recv subroutine, which gets called when a client request is received.
Read on…

Okay I lied, eAccelerator gives a pretty darn high ROI, but setting up a proxy cache gives a comparable or higher ROI. I chose to use Varnish as my proxy cache.

Once installed, Varnish will keep a cache of all objects requested by internet users (e.g. post-generated PHP pages, CSS, javascripts, images) with the goal of off-loading some work from your web server (remember: we won’t want big Apache to do the work only if it has to). Also Varnish takes full advantage of the OS’s virtual memory and advanced I/O features on FreeBSD 6.x making it the optimal choice for my setup.

There were many confusing instructions on the web about how to configure Varnish. Here are the steps I took to setting up Varnish for a signal machine running both Varnish and the web server: Read on…