About

The AI Crawler Honeypot is an automated observatory measuring how AI search engines, training scrapers, and autonomous agents interact with web infrastructure. It collects request telemetry, verifies claimed identity against published network ranges, exposes tools that agents can call, and publishes open, transparent access logs.

Zero Raw IP Retention

Privacy is architected into the system by default. Real client IP addresses are never written to disk or long-term databases. When a request is received, the IP is hashed using SHA-256 combined with a daily rotating salt (salt:{YYYY-MM-DD}). The salt expires automatically after 24 hours and is purged daily at 00:05 UTC, making irreversible reconstruction mathematically impossible.

Vendor CIDR Verification

When a request claims a user-agent string associated with known AI crawlers (such as OpenAI, Anthropic, Google, or Perplexity), the connecting IP is checked against official published CIDR blocks and reverse DNS records. Requests are categorized as:

• verified: Connecting IP belongs to the official published infrastructure of the claimed vendor.
• claimed / unverified: User-agent claims vendor affiliation, but connecting IP is outside published ranges.
• mismatch: IP is confirmed to belong to a conflicting network or third party.
• unverifiable: User-agent does not belong to a provider with public verification endpoints.

Robots.txt Conformance & Telemetry

The honeypot passively tracks whether agents respect robots.txt directives without dropping or resetting connections. Requests to disallowed paths are logged with the matching rule. Observational data is classified for cadence and sequential anomalies, giving researchers clear insight into automated crawler behavior.

Tool Registry & Agent Interaction

The honeypot exposes a registry of tools that AI agents can discover and call. Agents that read /tools.json can invoke tools via POST /tools/call, including:

• ask — ask a question about this domain
• check_crawler — check whether a specific domain has been observed here
• lookup_log — query the public log
• summary — return the current summary counts

Every tool call is logged with the same schema as request entries. The tool-call log is public at /tools/log.

Public Logs & Data Access

All activity is made publicly accessible across several views:

• Daily Public Log (/log/today) — Rolling 24-hour log with real-time status.
• Cumulative Search (/log/all) — Complete searchable and filterable historical log.
• Violations (/log/violations) — Requests to paths disallowed by robots.txt.
• Telemetry Summary (/log/summary) — Aggregate metrics by vendor and day, including robots_ignored_by_verified.
• Tool Calls (/tools/log) — Public log of tool invocations.
• Tool Registry (/tools.json) — MCP-compatible registry of available tools.
• Monthly Data Exports (/export/index) — Downloadable raw JSON and CSV monthly archives.
• Help (/help) — Navigation for AI agents and humans.
• Privacy Policy (/privacy) — Full UK GDPR Article 6(1)(f) documentation.

About

A public AI crawler observation honeypot. Factual observation only — no judgments, no editorialising, no claims about whether any agent is "safe" or "unsafe." Every entry is a timestamped field/value record.

Contact: vic@fixseo.uk