Easily capture every bot that visits your website.
Googlebot,
Bingbot, GPTBot, AI crawlers, all of them.
Real-time dashboard. Zero log files.
Already on Cloudflare? You can start for free in under 5 minutes.
Server logs are critical. In practice, they usually are not available in the form you need.
Most teams do not have clean access to raw bot traffic. Hosting providers hide logs, platforms sample them, engineering teams do not want to hand over full request data, and once you do get the files they often contain far more than you want from a privacy and security standpoint. CrawlerLogs skips the log pipeline entirely and captures only the crawler event you actually need.
<script src="https://js.crawlerlogs.com/latest.js" data-cl-token="YOUR_API_KEY"> </script>
Simple, quick, and useful. You will still miss bots that never execute JavaScript.
export default { async fetch(request, env) { const res = fetch("https://ingest.crawlerlogs.com/ingest", { method: "POST", headers: { "Content-Type": "application/json", "X-Api-Key": "YOUR_API_KEY" }, body: JSON.stringify({ ip: request.headers.get("cf-connecting-ip"), ua: request.headers.get("user-agent"), url: request.url, ts: Date.now() }) }); return fetch(request); } }
If you are already on Cloudflare, this is the cleanest and most complete setup path.
We collect exactly 4 data points per request.
That is the whole privacy model: do not ingest full server logs when all you need is crawler intelligence.
No cookies, no request bodies, no payloads, no user session data. We reduce each bot visit to four fields so you can verify crawlers, understand coverage, and avoid dragging PII or other sensitive request details into the workflow.
That's it. No user tracking. No session data. No PII. Just crawler intelligence.
Watch your crawl data flow in.
Everything you need to understand your crawl.
CrawlerLogs turns raw bot visits into actionable intelligence. See which bots are crawling what, how often they return, which pages they’re ignoring, and whether AI crawlers are scraping your content — all without touching a single log file.
Track Googlebot, Bingbot, GPTBot, ClaudeBot, AppleBot, and 50+ known crawlers automatically.
Cloudflare Worker captures at 300+ PoPs worldwide. No origin hit. Sub-millisecond logging overhead.
See which pages get crawled, how often, and by whom. Spot trends and anomalies instantly.
Find pages crawlers never visit. Identify structural issues before they impact indexing.
GPTBot, ClaudeBot, PerplexityBot — see exactly what AI crawlers are scraping from your site.
We only collect URL, IP, User-Agent, and Timestamp. No cookies, no payloads, no personal data. Nothing to leak, nothing to worry about.
Learn how bots visit sites, what we track, and why the data is so insightful.
We put together a practical guide covering how search bots, AI crawlers, and other automated visitors reach your pages, why server logs are so hard to use in the real world, and why a four-field tracking model gives teams clearer operational visibility.
Start free. Upgrade when you're ready.
Common questions.
For most sites, it's completely free. Cloudflare's free plan includes 100,000 requests per day (~3M/month). Bots typically account for a small fraction of your traffic, so you'll stay well within limits. Sites with over 3M monthly page views may need Cloudflare's $5/mo paid plan, which includes 10M requests. Even at 100M requests/month, you'd pay ~$32/mo to Cloudflare — a fraction of what traditional log analysis tools cost.
Google Search Console shows you what Google indexed. CrawlerLogs shows you what bots actually requested — every URL, every bot, in real-time. GSC doesn't track Bingbot, GPTBot, ClaudeBot, or any AI crawlers. CrawlerLogs tracks 50+ bots, including all the AI crawlers scraping your content.
No. The Worker fires an async POST with 4 small data points (URL, IP, User-Agent, Timestamp) and immediately returns the original request. There's zero blocking — your site's response time is completely unaffected. The POST happens in the background after the response is already sent to the visitor.
We collect exactly 4 things: URL, IP address, User-Agent string, and Timestamp. No cookies, no request bodies, no form data, no personal information, no payloads. There's literally nothing sensitive to leak. We don't track your visitors — we track bots.
Use CrawlerLogs Lite — a lightweight JavaScript tag you can deploy via Google Tag Manager or paste directly into your HTML. It works like a tracking pixel and captures Google and Bing crawl data. It's more limited than the Worker, but gives you valuable visibility without any server changes.
Yes. The free plan doesn't require a credit card. Paid plans are billed annually and you can cancel before your next renewal. Your data remains accessible until the end of your billing period.
50+ bots including Googlebot, Googlebot-Image, Bingbot, GPTBot, ClaudeBot, AppleBot, PerplexityBot, Meta-ExternalAgent, Bytespider, AhrefsBot, SemrushBot, MJ12bot, DotBot, and more. With the Cloudflare Worker, we capture every bot — if it makes a request, we see it.
Get started free with 7 days of rolling data. No credit card. No log files. Just insights.
