Crawlers database

BrightEdge Crawler/1.0 is a web crawler employed by BrightEdge Technologies to gather data for its suite of SEO and content marketing tools. This crawler is integral to services such as Site Audit/Content IQ, which analyzes on-page SEO, technical, and usability issues; On-page Recommendations, offering prescriptive actions to optimize web pages; and SearchIQ, which identifies areas to prioritize for SEO and analyzes current performance. (support.brightedge.com)

The user agent string for this crawler is "BrightEdge Crawler/1.0 (crawler@brightedge.com)". It operates by starting crawls based on specified URLs, opening links from each page for further crawling, and caching these URLs to enhance future crawl efficiency and provide a basis for comparison between crawls. (support.brightedge.com)

If you wish to control the crawler's access to your site, you can manage this through your site's robots.txt file. To disallow the BrightEdge Crawler, add the following lines:

User-agent: BrightEdge
Disallow: /

To explicitly allow it, use:

User-agent: BrightEdge
Disallow:

Ensure that a robots.txt file exists in each hosted directory, including subdomains, to effectively manage crawler access. (support.brightedge.com)

For any questions about whitelisting or BrightEdge crawlers, you can contact BrightEdge support at support@brightedge.com. (support.brightedge.com)