Analytics

crawlcheck.io

An MCP server that analyzes domains for AI crawler and answer-engine visibility by scanning sites, verifying crawler logs, resolving robots.txt directives, and drafting llms.txt files.

1 endpoint23 known toolsFirst detected October 11, 2026Last detected October 11, 2026

ENDPOINT 1

https://crawlcheck.io/mcp

No auth detected

MCP server metadata

Name: crawlcheck
Version: 1

Capabilities

tools

Server instructions

Before reading, citing, connecting to or transacting with a domain, call preflight with the domain, the action and your crawler token: do not proceed on block, ask the user on require_confirmation. Call scan_domain with a bare domain to get a graded record of what crawlers receive. grade is null when the origin refused CrawlCheck - report that as unmeasured, never as a low score. Rate limit: 5 scans per 10 seconds per address.

Known tools 23

scan_domain

Scan a public domain as several crawler identities and return the graded record: grade, AI-visibility score, per-section scores, findings with evidence, and whether the origin refused CrawlCheck.

Inferred read-only

verify_crawler_log

Given raw web-server access-log lines, decide for each line that names a crawler (GPTBot, ClaudeBot, Googlebot, PerplexityBot.

Inferred read-only

corpus_state

Coverage of CrawlCheck's public dataset: how many distinct domains have been measured and how they split by platform, rendering, size, language and kind, with the strata that are still under-sampled.

Inferred read-only

resolve_robots

Resolve every named answer engine, search index and training crawler against a domain's robots.

Inferred read-only

draft_llms

Draft an llms.

Inferred read-only

fix_robots

The domain's served robots.

Inferred read-only

fix_entitymap

A starter entitymap.

Inferred read-only

video_check

One YouTube video against the video anchor model: metadata, chapters, captions and storyboard where observed, twelve checks each naming what it reads.

Inferred read-only

video_channel

Every upload on a YouTube channel, read 50 at a time, sorted by views.

Potential side effects

video_site

The site half of video: homepage plus up to 60 sitemap pages read for YouTube embeds, facades and page-builder widgets, VideoObject nodes and their required properties, and - with a handle - how many of the channel's videos the site carries.

Inferred read-only

registry_lookup

Whether a domain is in CrawlCheck's registry and what it serves: llms.

Inferred read-only

crawler_path

Where each crawler's path into a domain dies, as an observed data flow: edge decision, robots.

Inferred read-only

public_counts

The figures CrawlCheck quotes about itself, read from the source: sites_measured, domains_in_corpus, crawler_visits (a rolling window), sections, sections_scored, findings_published, guides_published, with an at timestamp.

Inferred read-only

telemetry

Verified-crawler traffic observed at crawlcheck.

Inferred read-only

fix_mostly_code

A fix plan for PAGE_IS_MOSTLY_CODE: the homepage's measured bytes by category, the largest blocks with what to do with each, ordered steps with the text ratio after each, and the step at which the detector would stop firing - or, when moving code cannot clear it, how much markup to cut or text to add.

Inferred read-only

fix_stale_cache

A fix plan for STALE_CACHE_SERVED: the homepage fetched now, its Age, the cache layers that name themselves in the headers in purge order (innermost first), the HTML lifetime each response declares, a fresh-read comparison, and whether a purge clears the finding and keeps it cleared.

Potential side effects

fix_machine_chain

A fix plan for MACHINE_CHAIN_HEAVY: the files a crawler reads before the page and the 404s absent agent files return, then measured reductions (llms.

Inferred read-only

explain_finding

Why CrawlCheck decided a finding: the rule and its revision, the exact fetches it compared (identity, status, bytes, sha256), when the condition held on the site, and a Merkle proof that the decision is inside the signed manifest.

Inferred read-only

list_verified_capabilities

What a domain declares agents can do (OpenAPI operations, MCP tools, A2A skills, API catalog entries), each classed by safety (read_only_public up to financial), and which ones CrawlCheck actually called and confirmed - plus every mismatch between declaration and behaviour.

Inferred read-only

machine_record

The unified machine record for one observation: subject, grade, findings (the top one in full without a licence), capabilities with verification, and the evidence roots, manifest and verifier links that let anyone check it offline.

Inferred read-only

resolve_domain

Call before fetching from, citing or acting on a domain.

Inferred read-only

preflight

Call BEFORE reading, citing, connecting to or transacting with a domain.

Inferred read-only

mcp_servers

Which remote MCP servers the official MCP Registry lists on a domain, and what each endpoint actually answered when CrawlCheck sent the MCP handshake (initialized, auth_required, unreachable, tool count, tool-list drift).

Inferred read-only

CONNECT WITH APPROVAL

Client installation

Review this server and its permissions before adding it. Secret placeholders must be set locally.

Codex

~/.codex/config.toml

[mcp_servers.crawlcheck]
url = "https://crawlcheck.io/mcp"
enabled = true

Claude Code

.mcp.json

{
  "mcpServers": {
    "crawlcheck": {
      "type": "http",
      "url": "https://crawlcheck.io/mcp"
    }
  }
}

Claude Desktop

Settings → Connectors → Add custom connector

Name: crawlcheck
Remote MCP URL: https://crawlcheck.io/mcp

Add this remote URL as a custom connector in Claude Desktop. Availability depends on the user plan and workspace policy.

Cursor

.cursor/mcp.json

{
  "mcpServers": {
    "crawlcheck": {
      "url": "https://crawlcheck.io/mcp"
    }
  }
}

Visual Studio Code

.vscode/mcp.json

Add to Visual Studio Code

{
  "servers": {
    "crawlcheck": {
      "type": "http",
      "url": "https://crawlcheck.io/mcp"
    }
  }
}

Generic MCP

Client-specific MCP configuration

{
  "name": "crawlcheck",
  "transport": "streamable-http",
  "url": "https://crawlcheck.io/mcp"
}

MCP Inspector

Run the official MCP Inspector locally and enter the indexed Streamable HTTP endpoint.

TRUST AND VERIFICATION EVIDENCE

Trust Data Available

BuiltWith Trust API v2 evidence for crawlcheck.io was fetched 2026-10-16T06:20:37.281Z.

Trust status Neutral

crawlcheck.io is assessed as Neutral: No suspicious signals found, but no strong positive signal either

Indexed

Evidence is source-attributed and does not guarantee that a third-party server is safe. Risk labels are conservative metadata heuristics.