AhrefsBot is a web crawler that powers a link database for Ahrefs online marketing toolset. The bot constantly crawls the web to fill their database with new links and to check the statuses of the previously found ones. In other words, AhrefsBot is a bot that is primarily used as a backlink checker.

According to ahrefs.com, every 24 hours the bot crawls over 6 billion web pages and updates Ahrefs’ index every 15 – 30 minutes.

Link data collected by Ahrefs is used for a variety of purposes and for digital marketers to plan, execute, and monitor online marketing campaigns.

Some people claim that Ahrefs’ web crawler is a bad bot that does not respect the robots.txt file and bypasses disallow and allow rules. However, Ahrefs disputes this claim by saying “We absolutely strictly respect robots.txt: both disallow and allow rules. Please see https://ahrefs.com/robot for full transparency on our crawler.”

The bot can also use up your website’s bandwidth and make your website slower. Another concern with the bot is that it is used by spammers that participate in referrer spam indexing in order to spam your website with unwanted referral traffic.

Other people have reported that the company behind the bot is known to be dishonest and provide misleading support to those who have questions about the bot but we cannot validate these claims.

How to block AhrefsBot

You can block AhrefsBot using the robots.txt and .htaccess files. Blocking AhrefsBot will not negatively affect your rank and can actually be beneficial by clearing up bandwidth usage. It can also restrict your competitors from checking your backlinks.

These are solutions to block Ahrefs bot from crawling your website:

Method 1: Use the robots.txt file

The robots.txt file gives permission to crawlers to crawl a website and adding code to the file can stop bots like AhrefsBot from crawling your site. Most crawlers will abide by the rules of the robots.txt file; However, some will not including bad bots. AhrefsBot may or may not abide by the rules.

• Add this to the robots.txt file. The robots.txt file is located in your site’s files and can be found in your website’s root folder.

User-agent: AhrefsBot

Disallow:/

Method 2: Use the httaccess file

Using the htaccess file is a great method you can utilize to block AhrefsBot and other bots from crawling your website. With the .htaccess file you can block bad bots by IP addresses, or in this case, IP ranges since AhrefsBot uses several IP address and ranges.

• Add this to the .htaccess file. Alike the robots.txt file, the .htaccess file is found at the root of the files of your site.

deny from 173.199.64.0 - 173.199.127.255

Other IP ranges used by Ahrefs:

54.36.148.0/24
54.36.149.0/24
54.36.150.0/24
195.154.122.0/24
195.154.123.0/24
195.154.126.0/24
195.154.127.0/24

If the bot has crawled your website from another IP address, simply add the same code to your .htaccess file with the updated IP addresses but keep this range in there.

Another solution

In addition, add this code to the .htaccess file to block AhrefsBot.

RewriteEngine On

RewriteCond %{REQUEST_URI} !/robots.txt$

RewriteCond %{HTTP_USER_AGENT} ^.*AhrefsBot.*$ [NC,OR]

RewriteRule ^.*.*

Hopefully, you found this information helpful and have successfully blocked AhrefsBot from crawling your site. Leave a comment below of you have any questions or suggestions.