Choose one collection point
Browser analytics cannot observe crawler requests reliably. Install server-side observation separately. A Cloudflare route Worker is the recommended option because it can observe traffic at the edge, including CDN cache hits. Express middleware is an alternative for a long-lived Node application when edge integration is unavailable.
Use one collector per request path. Installing both on the same traffic can double-count visits. Neither integration sees requests blocked before it runs, and neither changes the response your website sends.
Generate the project-specific integration
In Tideflow AI, open your project → Settings → AI Bots Detection and select Copy token. To view the value first, select Reveal token. Save this project token as TIDEFLOW_CRAWLER_TOKEN in your Cloudflare Worker secrets or server environment; never include it in public website code.
To get the installation code, use your connected MCP client and call get_crawler_setup with no arguments. It returns worker, middleware, secretName, secretValue, instructions, middlewareInstructions, and integrationGuidance. Use the returned code as your installation module; you do not need to build your own bot detector.
The crawler credential is scoped to the project and website and belongs only on the server. It is separate from the browser installation token and the MCP token. Generating this setup does not deploy it.
{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/call",
"params": {
"name": "get_crawler_setup",
"arguments": {}
}
}Install the Cloudflare Worker
Save the returned worker source in a dedicated Worker project. Configure a route covering your proxied domain and the pages you want to observe. Replace example.com in this sample with your own zone and route; use your actual source filename. Store the project token copied from Settings → AI Bots Detection (or the returned secretValue) interactively as TIDEFLOW_CRAWLER_TOKEN, then deploy.
If a Worker already handles that route, merge the generated observation logic after its actual origin response instead of replacing the existing routing. Preserve its response and background-reporting lifecycle. Requests must pass through this Worker; an unrelated workers.dev deployment will not observe your website.
{
"name": "tideflow-crawler",
"main": "src/index.js",
"compatibility_date": "2026-09-01",
"routes": [
{
"pattern": "example.com/*",
"zone_name": "example.com"
}
]
}npx wrangler secret put TIDEFLOW_CRAWLER_TOKEN
# Paste the token from Settings > AI Bots Detection at the prompt.
npx wrangler deployInstall the Node and Express middleware
For a long-lived Node 20+ server, save the returned middleware as tideflow-crawlers.js in a server-only location. Set TIDEFLOW_CRAWLER_TOKEN in the server environment using the returned secretValue. Register the middleware before routes and static handlers, as shown below.
The generated middleware observes recognized bot GET requests when the response finishes and reports the actual response status. The events appear in the same AI Bots dashboard as Worker observations.
import express from "express";
import { tideflowCrawlerMiddleware } from "./tideflow-crawlers.js";
const app = express();
app.use(tideflowCrawlerMiddleware);
// Register your existing routes and static handlers afterward.
app.use(express.static("public"));
app.listen(3000);Understand what is observed
Express only sees requests reaching the application. Upstream cache hits, upstream blocks, and aborted responses are not recorded by this middleware. Keep the server running long enough to send observations after responding; immediate process shutdown can lose them. This is not a serverless lifecycle adapter: serverless integrations need their platform’s background-work lifecycle and the actual route response status.
Detection covers supported identities including OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, and others. A matching user agent is an observation, not authenticated identity. Cloudflare observations are marked verified only when trusted Cloudflare bot-management metadata says verifiedBot is true. Express observations remain unverified and do not trust client-supplied verification headers.
Verify the installation
Make a controlled test request to a page on your installed domain. This example intentionally impersonates a crawler user agent to exercise collection; it is not evidence of an actual OpenAI visit. Use a known test page so you can recognize the observation.
Open AI Bots in the matching Tideflow AI project and inspect the page, crawler identity, HTTP status, and date range. A received event verifies the reporting path, while a page timeline provides access history. A bot request does not establish indexing, citation, or a conversion. If nothing appears, check route coverage, proxied DNS for the Worker, the server secret, and whether traffic actually reaches your collector.
curl -A "OAI-SearchBot" -o /dev/null -s -w "%{http_code}\n" \
https://example.com/your-test-page