API - Documentation
The Crawler Data API provides a constantly updated database of known web crawlers, bots, and scrapers. All data is collected, processed, and updated automatically 24/7.
Use this API to
- Identify bots in server logs
- Classify IP addresses
- Detect unwanted scrapers
- Improve firewall and security rules
API Endpoint
https://crawlers-info.de/get-crawlers-data
The API is accessed using HTTP GET requests with URL parameters.
Retrieve the Full Database
Required Parameter:
| Parameter | Description |
|---|---|
| key | Your API key or free public API key 5555 (It is currently the same) |
Optional Parameters:
| Parameter | Description |
|---|---|
| format | json (default) or xml |
| rating | lower, medium, high or all (default) |
Examples:
cURL
curl -G -d "key=5555" https://crawlers-info.de/get-crawlers-datacurl -G -d "key=5555" -d "format=json" -d "rating=all" https://crawlers-info.de/get-crawlers-datacurl -G -d "key=5555" -d "format=xml" -d "rating=lower" https://crawlers-info.de/get-crawlers-dataURL
https://crawlers-info.de/get-crawlers-data?key=5555https://crawlers-info.de/get-crawlers-data?key=5555&format=json&rating=allTargeted Search
You can also search for a specific crawler by name or IP address.
Required Parameter:
| Parameter | Description |
|---|---|
| search | Bot name or IP address (e.g. ahrefsbot or 54.36.148.6) |
Optional Parameter:
| Parameter | Description |
|---|---|
| format | json (default) or xml |
Examples:
cURL
curl -G -d "search=54.36.148.6" -d "format=xml" https://crawlers-info.de/get-crawlers-dataURL
https://crawlers-info.de/get-crawlers-data?format=xml&search=ahrefsbotUpdate Frequency & Usage
The database information is updated once a day between 1:00 a.m. and 3:00 a.m. (Europe/Berlin, UTC+2). Multiple requests per day are not necessary. We recommend storing the API response locally (JSON or XML) and updating it daily. There is currently no limit on the number of requests, but this may change in the future.
Targeted searches always return new and up-to-date information.
Data Quality
All data is collected, analyzed, and maintained automatically without human intervention. If you find incorrect information, please report it using the contact form.
Support the Project
You are free to use this data for your own projects, services, or security systems. If you find this API useful, we would greatly appreciate your support to cover the following costs:
- Server costs
- AI processing costs
- Further project development
You can find more information about donations here.