I’m currently running FreeFixer.com on a shared Dreamhost server. Dreamhost has a monitoring service that  keeps an eye on the total resource usage for each user account. If some user consumes to much resources on the server, the monitoring service starts killing off processes for that user and an email report is sent. This is great since it saves me much of the performance problems caused by other users on the same server.

Some time ago, the resource usage for freefixer.com started hitting the limit but I didn’t notice any additional traffic when I examined the Google Analytics report. This led me to investigate Apache’s access.log file. Here are two example entries from the log:

157.55.39.252 - - [25/Jun/2019:02:37:05 -0700] "GET /library/file/UninstallTP.exe-154295/ HTTP/1.1" 200 17986 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

163.172.64.171 - - [25/Jun/2019:02:37:10 -0700] "GET /b/tag/fake-flash-software/ HTTP/1.1" 200 18719 "-" "Barkrowler/0.9 (+http://www.exensa.com/crawl)"

The first entry (157.55.39.252) claims to be the bingbot and the second (163.172.64.171) is a crawler called Barkrowler (exensa.com).

When examining the access.log a bunch of questions are raised:

1. Let’s say the crawler claims to be BingBot or GoogleBot, but is it the real one coming from one of Google’s or Microsoft’s data centers, or is it a bot that falsely set its user agent to GoogleBot or BingBot?
2. What about all the other bots out there? Their crawling uses quite a lot of resources, but do they bring any value or users to your web site.
3. What about all the other high usage IP-numbers that claims to be ordinary users? Are their claims correct, or are they just bots in disguise?

How To Determine If a Bot is Fake

Let’s say you see an entry in the log coming from 157.55.39.252 and it claims to be bingbot. How can we determine that the traffic is from a real bingbot? We can do this using the following two steps:

1) First we do a reverse DNS lookup using the IP from the log.

$ host 157.55.39.252

252.39.55.157.in-addr.arpa domain name pointer msnbot-157-55-39-252.search.msn.com.

The DNS responds with [msnbot-157-55-39-252.search.msn.com].

2) Then we do a forward DNS lookup on the hostname we got from the reverse lookup.

$ dig +short msnbot-157-55-39-252.search.msn.com

157.55.39.252

So, to summarise: 157.55.39.252 points to [msnbot-157-55-39-252.search.msn.com] which is owned by Microsoft. And the [msnbot-157-55-39-252.search.msn.com] hostname resolves back to 157.55.39.252 which we started with. Excellent, we now know that we are dealing with a legitimate bingbot.

Another way to check if an IP belongs to bingbot, if you don’t have the host and dig command line tools available, is to use Bing’s Verify Bingbot Tool. You simply type in the IP address, in this case 157.55.39.252, and solve the captcha.

I’m not aware of web verification tools for the other search engines such as Google or Yandex. If you know about such a tool, please let me know.