Crawlers database

Comscore's Web Crawler is a specialized software tool designed to access and analyze the content of web pages. Its primary function is to perform contextual content analysis, enabling advertisers to match campaigns with relevant page content. The crawler operates by identifying itself through the user-agent string:

Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)

This identification ensures transparency and allows website administrators to recognize and manage the crawler's activities.

Key characteristics of Comscore's Web Crawler include:

• Content Analysis: The crawler examines static, textual content on web pages to assess their relevance and context.

• Compliance with Robots.txt: It adheres to the directives specified in a site's robots.txt file, respecting the preferences of website owners regarding automated access.

• Selective Data Extraction: The crawler does not extract or store source code; instead, it gathers data about publicly available content, such as content language, content ratings (e.g., G, PG13, R), and relevant IAB categories (e.g., "Real Estate::Buying/Selling Homes").

• Non-Execution of JavaScript: It does not execute JavaScript to generate ad impressions, focusing solely on static content.

• Controlled Crawling Rate: The crawler operates at a slow rate by default to minimize the impact on website performance.

For website administrators who wish to allow the crawler to access their sites, adding the following lines to the site's robots.txt file will whitelist the crawler:

User-agent: proximic
Disallow:

This configuration permits the crawler to access all areas of the site.

In summary, Comscore's Web Crawler is a tool that facilitates the contextual analysis of web content, aiding advertisers in delivering relevant and targeted campaigns while respecting website owners' preferences and maintaining minimal impact on site performance.

United States(US)