Stored
- URL — scheme + host + path (query stripped)
- IP — for range checks
- User-agent — bot identity claim
- Timestamp — UTC; shown in your profile timezone
How to send crawl events, what we store, and how verification works.
Replace <TOKEN> with the token from your domain setup. The Worker posts four fields and lets the origin request continue.
export default {
async fetch(request, env, ctx) {
ctx.waitUntil(ingest(request, env));
return fetch(request);
}
}
async function ingest(request, env) {
try {
await fetch("https://ingest.crawlerlogs.com/ingest/<TOKEN>", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
url: request.url,
user_agent: request.headers.get("user-agent"),
ip: request.headers.get("cf-connecting-ip"),
timestamp: new Date().toISOString()
})
});
} catch (_) {}
}Signup shows the exact ingest URL for your account. See also how it works and the Worker vs JS guide.
We match user-agent + IP against known bots and published ranges. Ranges change; treat this as evidence, not a court ruling.
Catalog identity and IP inside that crawler’s published ranges.
Looks like a known bot in the UA, but the IP is not in range.
IP is in a crawler range; UA does not claim that crawler the usual way.
We recognize the bot, but there is no usable published range to check.
Same four fields for page loads that run the script. It will not see crawlers that skip JavaScript. Prefer the Worker when you can.
<script src="https://www.crawlerlogs.com/c.js?t=<TOKEN>" async></script>Setup in the app shows the Worker snippet with your token filled in. Read API keys live under Settings after signup.