Crawlers database

Amazonbot/0.1 is Amazon's web crawler designed to enhance services like Alexa by indexing web content, thereby improving the accuracy and comprehensiveness of responses to user queries.

Identifying Amazonbot/0.1:

The user-agent string for Amazonbot/0.1 includes "Amazonbot" along with additional agent information. An example user-agent string is:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)

Controlling Amazonbot's Access:

Amazonbot/0.1 adheres to the robots.txt protocol, allowing webmasters to manage its access to their sites. For instance, to prevent Amazonbot from crawling specific directories, you can include the following in your robots.txt file:

User-agent: Amazonbot
Disallow: /do-not-crawl/

However, Amazonbot does not support the crawl-delay directive or robots meta tags like "nofollow" and "noindex".

Verifying Amazonbot's Authenticity:

To confirm that a crawler accessing your server is the official Amazonbot, you can perform DNS lookups:

1. Identify the accessing IP address from your server logs.
2. Use the host command to perform a reverse DNS lookup on the IP address.
3. Ensure the retrieved domain name is a subdomain of crawl.amazonbot.amazon.
4. Perform a forward DNS lookup on the retrieved domain name.
5. Verify that the returned IP address matches the original IP address from your server logs.

For example:

$ host 12.34.56.789
789.56.34.12.in-addr.arpa domain name pointer 12-34-56-789.crawl.amazonbot.amazon.

$ host 12-34-56-789.crawl.amazonbot.amazon
12-34-56-789.crawl.amazonbot.amazon has address 12.34.56.789

For more detailed information, you can visit Amazon's official page about Amazonbot.

United States(US) , Australia(AU) , Netherlands(NL) , The Netherlands(NL) , Luxembourg(LU) , France(FR) , Spain(ES) , South Korea(KR) , Singapore(SG) , Taiwan(TW) , Germany(DE) , Switzerland(CH)

[1] Amazonbot/0.1 ** [2] Amazonbot/0.1 **