Methodology

This page documents how the AI Crawler Honeypot works. It describes the verification pipeline, the logging schema, the privacy architecture, the x402 monetization layer, and the public record.

— What this is —

A public AI crawler observation honeypot. It logs HTTP requests to this domain, verifies claimed identity against published vendor infrastructure, and publishes a public record.

— Zero raw IP retention —

Real client IP addresses are never written to disk or long-term databases.

When a request is received:

1. The source IP is truncated to /24 (IPv4) or /48 (IPv6).
2. The full IP is hashed using SHA-256 combined with a daily rotating salt.
3. The salt is stored at salt:{YYYY-MM-DD} with a 24-hour TTL.
4. The salt is purged automatically at 00:05 UTC each day.

After 24 hours, the salt no longer exists. The hash cannot be reversed. Historic entries remain intact with their computed hash, but no link to the original IP is retained.

— Vendor CIDR verification —

When a request claims a user-agent associated with a known AI crawler, the connecting IP is checked against the vendor's published infrastructure.

Categories:

• verified — Connecting IP belongs to the official published infrastructure of the claimed vendor.
• claimed / unverified — User-agent claims vendor affiliation, but connecting IP is outside published ranges.
• mismatch — Connecting IP does not match the published ranges of the vendor claimed in the User-Agent string.
• unverifiable — User-agent does not belong to a provider with public verification endpoints.

— What “mismatch” means —

A mismatch means the source IP address did not fall within the published network ranges of the vendor claimed in the User-Agent string.

This is a factual observation, not an accusation. A mismatch can occur for several reasons:

1. Client-side agents: Interactive AI tools (like Claude Code, ChatGPT-User, or Cursor) run on a human's local machine or VPS. When they fetch a page, they use the human's IP address, not the AI company's infrastructure.
2. Spoofing: A script or scanner is intentionally using a fake User-Agent to bypass filters or impersonate a legitimate AI crawler.
3. Proxies: The request is being routed through a corporate proxy, VPN, or CDN that masks the original source network.

The log records the mismatch. It does not guess the intent.

— Robots.txt conformance —

The honeypot tracks whether agents respect robots.txt directives. It does not drop or reset connections. It serves a 200 response to every open request, including those to disallowed paths.

Requests to disallowed paths are quietly logged with the matching rule inside the telemetry streams, allowing the record to factually show which agents respected the rules and which did not.

— DeepSeek behavioural classification —

Requests to disallowed paths and requests to /ask are classified using DeepSeek. The classification records:

• behavior_pattern: mechanical, human_like, or unknown
• cadence_anomaly: boolean
• path_sequence_anomaly: boolean
• notes: factual observations only

Compliant requests (robots_rule == "none") do not call DeepSeek. They receive a stub classification: unclassified — skipped — routine traffic.

— Machine-to-Machine x402 Monetization —

Deep telemetry archives and data extractions are guarded by an open, HTTP-native x402 Payment Gate.

When an AI crawler or automated data-scraping agent attempts to access deep archival endpoints under /export/*, the server issues an HTTP 402 Challenge (Payment Required). To clear the gate and unlock the payload, the client must submit valid, verified on-chain cryptographic proof of a $0.01 USDC transaction settled on the Base Mainnet network (eip155:8453), routing directly to the domain's public ledger address.

— Log schema —

Every entry contains the following fields:

• timestamp (UTC, ISO 8601)
• claimed_agent (User-Agent string, treated as a claim not identity)
• verification (verified, claimed, mismatch, or unverifiable)
• source_network (truncated IP prefix, /24 or /48)
• ip_hash (SHA-256 of IP + daily salt)
• requested_path
• robots_rule (the matching disallow rule, or "none")
• response_code
• deepseek_classification (JSON object)
• tool_name, tool_input, tool_output (for tool calls only)

No other fields are added. The schema is stable.

— Data retention —

Public log (log:public:{date}): 24-hour TTL. Entries expire automatically.

Archive (log:archive:{timestamp}:{uuid}): permanent, no TTL. Retained for 12 months, then reviewed under the retention policy at /privacy.

Salt (salt:{date}): 24-hour TTL. Purged daily at 00:05 UTC.

— Public views —

The record is accessible at the following paths:

• /log/today — Rolling 24-hour open log
• /log/summary — Aggregate counts, all time and per day (free summary hook)
• /tools.json — Public open definition of active tools
• /export/index — Gated monthly JSON and CSV telemetry archive index (Requires x402 execution)

— What the record does not do —

No judgments. No scores. No labels like “safe,” “unsafe,” “malicious,” or “threat.” Every entry is a factual field/value record.

No claims about whether any agent is safe or unsafe. No claims about intent. Verification describes the source network, not the agent's purpose.

— Contact —

Corrections, removal requests, and general enquiries: vic@fixseo.uk