AI & Machine Learning

crawlcensus.com

Analyzes and audits websites' robots.txt and accessibility for AI crawlers, providing reports and statistics on AI agent access.

1 endpoint6 known toolsFirst detected August 23, 2026Last detected August 23, 2026

ENDPOINT 1

https://crawlcensus.com/mcp

No auth detected

MCP server metadata

Name: crawl-census
Version: 1.0.0

Capabilities

tools

Server instructions

Audit how AI crawlers and answer engines can read a website, and query the public census of who blocks whom.

Known tools 6

scan_site

Run a live AI-accessibility audit of a domain: robots.

Inferred read-only

site_report

Return the most recent stored audit for a domain without triggering a new scan.

Inferred read-only

census_stats

Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.

Inferred read-only

crawl_preflight

Decide whether a crawler may fetch a list of domains before spending requests on them.

Inferred read-only

agent_profile

What this census measures and publishes about one AI crawler: how often it is disallowed in robots.

Inferred read-only

census_facts

Every headline finding from the census as discrete, dated records rather than prose.

Inferred read-only

CONNECT WITH APPROVAL

Client installation

Review this server and its permissions before adding it. Secret placeholders must be set locally.

Codex

~/.codex/config.toml

[mcp_servers.crawl-census]
url = "https://crawlcensus.com/mcp"
enabled = true

Claude Code

.mcp.json

{
  "mcpServers": {
    "crawl-census": {
      "type": "http",
      "url": "https://crawlcensus.com/mcp"
    }
  }
}

Claude Desktop

Settings → Connectors → Add custom connector

Name: crawl-census
Remote MCP URL: https://crawlcensus.com/mcp

Add this remote URL as a custom connector in Claude Desktop. Availability depends on the user plan and workspace policy.

Cursor

.cursor/mcp.json

{
  "mcpServers": {
    "crawl-census": {
      "url": "https://crawlcensus.com/mcp"
    }
  }
}

Visual Studio Code

.vscode/mcp.json

Add to Visual Studio Code

{
  "servers": {
    "crawl-census": {
      "type": "http",
      "url": "https://crawlcensus.com/mcp"
    }
  }
}

Generic MCP

Client-specific MCP configuration

{
  "name": "crawl-census",
  "transport": "streamable-http",
  "url": "https://crawlcensus.com/mcp"
}

MCP Inspector

Run the official MCP Inspector locally and enter the indexed Streamable HTTP endpoint.

TRUST AND VERIFICATION EVIDENCE

Loading Trust v2 evidence…

Checking the associated registrable domain. The BuiltWith key remains server-side.

Indexed

Evidence is source-attributed and does not guarantee that a third-party server is safe. Risk labels are conservative metadata heuristics.