Tag: firewall

Originally published May 27, 2008, I’ve bumped this up a bit in the queue after some edits.

Spambots and spiders that ignore robots exclusion file can kill your site both in bandwidth and by potentially exposing information you don’t want ‘harvested.’ With that in mind, here is a quick-n-dirty guide to blocking spambots and rogue search engine spiders by using .htaccess. First the essential example codeblock, followed by a working example:

essential example codeblock

# redirect spambots & rogue spiders to the end of the internet
Options +FollowSymlinks
RewriteEngine On
RewriteBase /
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^spambot
RewriteRule ^(.*)$ http://www.shibumi.org/eoti.htm#$1 [R=301,L]

Next is to read my article on how to quickly check your error logs for oddities … which should provide you with a list of all sorts of unusual user agents worth blocking.

With said list, all that is left to do is create a working version that instead of sending people to the end of the internet, blocks them outright – which is probably a better move then sending the traffic elsewhere:

real-world/working example

# redirect spambots & rogue spiders to the end of the internet
Options +FollowSymlinks
RewriteEngine On
RewriteBase /
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailSearch [OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft\ URL [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector
RewriteRule .* - [F,L]

Note I provide 4 examples:

1. ^$,
2. ^EmailSearch
3. ^Microsoft\ URL
4. ^Web\ Image\ Collector

All to demonstrate how to use perl-like regular expressions parse out the user agent. For example:

1. ^ – identifies the beginning of the user agent string
2. $ – identifies the end of the user agent string
3. \ – that is a slash with a space afterwards tells the parser to include the space between words
4. [OR] – is placed after each of the multiple entries, except the last
5. [NC,…] – is sometimes placed after an entry to scan it w/out concern to upper or lower case

In the process, I’m intentionally blocking empty user agents using .htaccess – “^$” – a search string that uses a regular express to test for nothing between the beginning “^” and end “$” of a user agent token. Sorry, but if you’re not willing to tell me who/what you are, I’m not willing to show you my content.

Also, be aware the above requires that you have mod_rewrite installed on your Apache server, and that you have privileges to create your own rewrite rules in your own .htaccess file. If you’re not sure, check with your hosting service and/or system administrator.

In most cases, such privs & access exists – but your mileage may vary – as they might in how your particular .htaccess file actually works in-the-wild.

That said, more tomorrow or Thursday on how to create cron job to list those “unusual user agents” ‘automagically‘ for easy identification – and if needed -anti-spam remediation.

Using a blog to manage a website’s content is a flexible and affordable solution more and more churches are employing to effectively present their message online. There is however one drawback – in that some of the open source blogging solutions used as content management on the cheap also tend to attract attention from nere-do-wells who attack the comment and content functions of application such as WordPress and MovableType with robotic floods of advertisements offering anything from enlarging various appendages to curing male baldness all while losing your life’s saving playing poker online.

What’s worse is that many of these attacks these days come from servers in countries where you have absolutely no legal, let alone social, recourse to stop said attacks. Take for example a recent slam of attacks on a new dedicated server I’ve been working on – all which failed due to recent preventative security endeavors – but all incoming from a block of related IP addresses from a server in China all of whose addresses had 218.25.161… in common.

And while these unwanted advances were successfully thwarted by various server hardening practices implementations – the best way to avoid trouble from said attacker is to just deny access to anything on the server by denying the range of IP addresses indicated in my security logs.

With that in mind, I thought I’d share two approaches to blocking a range of IP addresses. One solution at the firewall level – the path I prefer on dedicated servers, the other solution is blocking IP blocks via the .htaccess file, which are employed on sites hosted on a shared server.

Using APF firewall, I simply create an entry that defines the block – in this case:

218.25.161.0/24

In the .htaccess file:

<Limit GET HEAD POST>

order allow,deny

deny from 218.25.161

allow from all

</LIMIT>

Both implementations block IP addresses from 218.25.161.0 through 218.25.161.255. But what happens if I only want to block addresses from a smaller set of addresses? Like those coming from someone abusing their DSL services whose range of dynamically assigned IPs may only be a range of 216.12.201.150 through 216.12.201.200.

That becomes trickier as is requires both a knowledge of the ‘CIDR notation’ and the bit mapping that goes along with it. Which is why I recommend instead using this nifty little online tool from Mikero.com. An easy-to-use service which performs all the bit-blasting, while also “aligning” the range so it can be expressed in correct CIDR notation.

Or in laymen’s terms, I add the following generated range to my firewall:

216.12.201.128/25

Or where no such firewall access is available, the following line in my .htaccess file:

deny from 216.12.201.128/25

Below are some tools and links on the topic of how to block a range of IP addresses if you want to dig into it a bit further.

Online tools to calculate an IP address range (CIDR):

• Mikero.com IP address range calculator
• IPAddressLocation.org subnet mask calculator

Online tools to check/verify your CIDR notation:

• Grox’ WhatMask IP subnet/CIDR Calculator
• Security Tools Online Painless CIDR calculator

Tutorials on blocking IP addresses and CIDR subnet masks:

• WebGeek: .Htaccess IP Banning – Block Bad Visitors
• WhatMask? from Laffey Computer Imaging

Pre-fabricated blacklists to block IP addresses of entire countries:

• WizCraft – Server .htaccess Blocklists for exploited servers (e.g. Russia and China)
• Block a country.com – block by ip address, deny access by country

A bit more on .htaccess and mod_access:

• Apache Project: Module mod_access (v. 1.3) – allow
• ClockWatchers.com – .htaccess Tutorial how to block an IP address

Just remember to keep good backups of whatever files you’re working on – and try not to lock yourself out while experimenting with changes!