Database
There are currently 1670 real crawlers registered in the database.
| Nr. | Name | Host | Rating | Updated | Info |
|---|---|---|---|---|---|
| 61 | Crawl4AI/1.0 | - | Lower | 2026-02-16 11:00:17 | |
| 62 | Crawler | - | Lower | 2026-07-26 16:16:59 | |
| 63 | Crawler bot | https://llmstxtscan.org | Lower | 2026-07-16 16:00:07 | |
| 64 | Crawler@obsrvr | https://obsrvr.net/about | Lower | 2026-07-30 00:01:48 | |
| 65 | CrawlmBot/1.0 | https://github.com/rodyherrera/Crawlm | Lower | 2026-07-16 09:30:12 | |
| 66 | Creditreform-ESG-Bot | - | Lower | 2026-01-28 14:00:12 | |
| 67 | DENIC-Crawler/42 | https://www.denic.de/crawler | Lower | 2026-01-17 21:19:45 | |
| 68 | DataBot/1.0 | - | Lower | 2026-03-28 02:00:09 | |
| 69 | DataForSeoBot/1.0 | https://dataforseo.com/dataforseo-bot | High | 2026-08-10 06:26:20 | |
| 70 | DataFoundationResearch/0.1 | https://example.com | Lower | 2026-07-28 10:00:05 | |
| 71 | DataHub | https://www.hubspot.com | Lower | 2025-08-09 19:00:08 | |
| 72 | Dataprovider | https://Dataprovider.com | Lower | 2026-08-07 17:25:38 | |
| 73 | DeCrawler/1.0 | - | Lower | 2026-04-18 08:22:42 | |
| 74 | Deadweb-prober/1.0; | https://github.com/Crawlora-org/dead-web-commoncrawl | Lower | 2026-08-11 13:00:07 | |
| 75 | DeepSeekBot/1.0 | https://www.deepseek.com/bot | Lower | 2026-08-03 05:02:26 | |
| 76 | Developers-facebook-crawler | https://developers.facebook.com/docs/sharing/webmasters/crawler | Lower | 2026-05-24 16:57:46 | |
| 77 | Developers.snap | https://developers.snap.com/robots | Lower | 2026-04-29 15:53:04 | |
| 78 | Diffbot/1.0 | - | Lower | 2026-07-16 19:34:07 | |
| 79 | Discordbot/2.0 | https://discordapp.com | Lower | 2026-07-07 18:13:58 | |
| 80 | Dmbot/1.1 | - | Lower | 2026-08-01 21:42:38 |
Crawler bot
Rating:
Lower
Updated(UTC):
2026-07-16 16:00:07
Crawler@obsrvr
Rating:
Lower
Updated(UTC):
2026-07-30 00:01:48
CrawlmBot/1.0
Host:
https://github.com/rodyherrera/Crawlm
Rating:
Lower
Updated(UTC):
2026-07-16 09:30:12
DataForSeoBot/1.0
Host:
https://dataforseo.com/dataforseo-bot
Rating:
High
Updated(UTC):
2026-08-10 06:26:20
DataHub
Rating:
Lower
Updated(UTC):
2025-08-09 19:00:08
Dataprovider
Rating:
Lower
Updated(UTC):
2026-08-07 17:25:38
Deadweb-prober/1.0;
Host:
https://github.com/Crawlora-org/dead-web-commoncrawl
Rating:
Lower
Updated(UTC):
2026-08-11 13:00:07
Developers-facebook-crawler
Host:
https://developers.facebook.com/docs/sharing/webmasters/crawler
Rating:
Lower
Updated(UTC):
2026-05-24 16:57:46
Developers.snap
Host:
https://developers.snap.com/robots
Rating:
Lower
Updated(UTC):
2026-04-29 15:53:04
Discordbot/2.0
Rating:
Lower
Updated(UTC):
2026-07-07 18:13:58