What is AhrefsBot and how do I block it?

Learn what AhrefsBot is and how to block the bot from crawling your website using the .htaccess and robots.txt files.

AhrefsBot

AhrefsBot is a web crawler that powers a link database for Ahrefs online marketing toolset. The bot constantly crawls the web to fill their database with new links and to check the statuses of the previously found ones. In other words, AhrefsBot is a bot that is primarily used as a backlink checker.

AhrefsBot

According to ahrefs.com, every 24 hours the bot crawls over 6 billion web pages and updates Ahrefs’ index every 15 – 30 minutes.

Link data collected by Ahrefs is used for a variety of purposes and for digital marketers to plan, execute, and monitor online marketing campaigns.

Many consider AhrefsBot to be a bad bot that does not add value to your website. Blocking AhrefsBot will not negatively affect your rank and can actually be beneficial by clearing up bandwidth usage. It can also restrict your competitors from checking your backlinks.

The reasons why some consider Ahrefs’ web crawler to be a bad bot is because it often does not respect your robots.txt file and disallow and allow rules. It can bypass the rules and crawl your website even though you do not want it to. The bot can also use up your website’s bandwidth and make your website slower. Another concern with the bot is that it is used by spammers that participate in referrer spam indexing in order to spam your website with unwanted referral traffic.

Other people have reported that the company behind the bot is known to be dishonest and provide misleading support to those who have questions about the bot.

How to block AhrefsBot

You can block AhrefsBot using the robots.txt and .htaccess files. Below are solutions to block the bot from crawling your website.

Method 1: Use the robots.txt file

The robots.txt file gives permission to crawlers to crawl a website and adding code to the file can stop bots like AhrefsBot from crawling your site. Most crawlers will abide by the rules of the robots.txt file; However, some will not including bad bots. AhrefsBot may or may not abide by the rules.

  • Add this to the robots.txt file. The robots.txt file is located in your site’s files and can be found in your website’s root folder.
User-agent: AhrefsBot

Disallow:/

Method 2: Use the httaccess file

Using the htaccess file is a great method you can utilize to block AhrefsBot and other bots from crawling your website. With the .htaccess file you can block bad bots by IP addresses, or in this case, IP ranges since AhrefsBot uses several IP address and ranges.

  • Add this to the .htaccess file. Alike the robots.txt file, the .htaccess file is found at the root of the files of your site.
deny from 173.199.64.0 - 173.199.127.255

Other IP ranges used by Ahrefs:

54.36.148.0/24
54.36.149.0/24
54.36.150.0/24
195.154.122.0/24
195.154.123.0/24
195.154.126.0/24
195.154.127.0/24

If the bot has crawled your website from another IP address, simply add the same code to your .htaccess file with the updated IP addresses but keep this range in there.

Another solution

In addition, add this code to the .htaccess file to block AhrefsBot.

RewriteEngine On

RewriteCond %{REQUEST_URI} !/robots.txt$

RewriteCond %{HTTP_USER_AGENT} ^.*AhrefsBot.*$ [NC,OR]

RewriteRule ^.*.*

Hopefully, you found this information helpful and have successfully blocked AhrefsBot from crawling your site. Leave a comment below of you have any questions or suggestions.

Sean Doyle

Sean Doyle is an experienced tech author with experience in cybersecurity, privacy, malware, and more. He has written for numerous online publications.