Bots, short for robots, are computer programs that browse (surf) websites all over the internet and automatically perform specific tasks. Like almost everything else on the internet, there are good bots and bad bots. In this article we will explain what bots are and how to block bad bots with .htaccess files.
Good bots are used by search engines such as Google and Yahoo to crawl your website, learn about it and use such information so your website can rank in the search engine results. Bad bots are typically used to harvest email addresses from websites, which later are used by spammers. Bad bots are also used to find security vulnerabilities in websites. When vulnerabilities are found by these bad bots, they are later exploited by hackers.
Why you need to protect WordPress from bad bots?
The main reasons why you would need to protect your WordPress from bad bots are spam and bandwidth, which costs money.
Htaccess file to block bad bots
You can block a single bad bot from accessing your WordPress by using an htaccess file. By using the htaccess file in example below, we are going to block a bad bot with the user-agent string evilbot.
RewriteEngine on RewriteCond %{HTTP_USER_AGENT} ^evil RewriteRule ^(.*)$ http://no.access/
The above htaccess sample is checking the user-agent of the bad bot, and if it matches evil it will be redirected to a non-existing website http://www.no.access. If you would like to block multiple bad bots from accessing your WordPress, use the [OR] operand in the htaccess file and add a line for each bad bot you want to block, as shown in the below example.
RewriteEngine on RewriteCond %{HTTP_USER_AGENT} ^evilbot [OR] RewriteCond %{HTTP_USER_AGENT} ^spambot [OR] RewriteCond %{HTTP_USER_AGENT} ^virusbot RewriteRule ^(.*)$ http://no.access/
WP White Security Webmaster Tip: Bad bots are like pests; it is impossible to get rid of them. You can have the most extensive list of bad bots but new ones will appear every day. I wrote this article so you can effectively block bad bots which are badly affecting your WordPress website and not to block all the bad bots on the internet.
4 comments
Thank you for this topic.
Can we have the opposite approach, BLOCK ALL Bots and trust or allow ONLY the legitimate Bots?
Is there any code or formula to do this?
Thx
You are welcome David. Unfortunately it is not possible to block all bots and only allow legitimate ones. There is no list of legitimate bots list, and even if there was, rest assured that bad bods will try to emulate good bots.
With the latter script are we covering everything bad or this is just an example?
For instance I see this when I’m looking at my report Unknown robot (identified by ‘bot’ followed by a space or one of the following characters _+:,.;/\-)
Am I covering these bots with the latter script?
Hi there,
Thank you for reaching out!
These scripts are just basics and are used as an example in this article. Having said that, it’s almost impossible to cover all bots.
However, you can update the htaccess files to work for your website as per your requirements.
Regards,