Kept AI

Agent Analytics

Know which AI agents visit and what they read

Log every AI crawler that reaches your site, read its purpose, and decide which bots deserve access.

Sign in to run the check on your domain

Northwind overview

Example data

Six months of fictional brand trends. Use the metric tabs and focus a month to inspect sample values.

April 2025
Rosebay65%
Bluehaven62%
Northwind47%
Sunfield32%
Pinegrove21%

Northwind’s competitors

Compare the fictional brands in this sample

BrandVisibilitySentimentPosition
1Rosebay65%862.7
2Bluehaven62%622.9
3Northwind47%893.6
4Sunfield32%763.9
5Pinegrove21%882.3

Domains

URLs ↗
DomainTypeUsedAvg. citations
Fforum.exampleUGC32%41%
Jjournal.exampleEditorial43%38%
Gguide.exampleComparison31%30%
Nnorthwind.exampleCorporate28%24%
Aanswers.exampleOther19%17%

Domains by type

Most cited domains, grouped by type

  • UGC32%
  • Editorial26%
  • Corporate20%
  • Comparison14%
  • Other8%

↗ Fictional brand trends

Illustrative workspace · Example data

Filter by intent

Split training crawlers from bots that fetch pages for live AI answers, and focus on the traffic that counts.

Catch what's broken

List the URLs that fail when AI bots request them, then close those gaps.

Audit your robots.txt

Kept AI shows how your robots.txt treats each crawler — blocked, allowed or not covered — so you never shut out the bots you actually want.

See behavior over time

Follow AI crawl trends per bot, model and URL from one week to the next.

From insight to action

  1. Locate an access problem

    Start with a failed URL or a crawler rule that needs review.

    Example interface: sample requests grouped into 200, 404 and 403 responses
    32 requests3 status codesReview errors
  2. Verify the intended policy

    Confirm whether that crawler should access that page.

    Example interface: neutral project domain, region and weekly sampling settings
    1 projectSet scopeReview setup
  3. Fix and watch new requests

    Review later status codes; access alone does not guarantee a citation.

    Example interface: a three-step sample collection, review and recheck history
    3 milestonesKeep a record7 days

Connect the gateway in three steps

Use the domain-specific records in your connection flow. Confirm origin and certificate readiness before changing production DNS; a tracked project alone is not a connected gateway.

  1. Configure the gateway CNAME

    Prepare the host record and gateway target shown in your connection flow.

    Example interface: neutral project domain, region and weekly sampling settings
    1 projectSet scopeReview setup
  2. Verify the connection

    Complete domain control validation and check DNS, certificate, and origin status in the app.

    Example interface: sample requests grouped into 200, 404 and 403 responses
    32 requests3 status codesReview errors
  3. View incoming requests

    After connection, inspect crawler requests and status codes in Agent Analytics.

    Example interface: sample crawler requests and HTTP status codes
    Read-only log3 pathsHTTP status

Example DNS record. Replace the host and follow the target supplied by the app.

www.example.com.  CNAME  gateway.freem.ai.

Integrations

Bring the evidence into your workflow

Use the data available in your project. MCP reads results; review any assistant-generated draft before exporting or sharing it.

  • CSV / JSON exportsDownload project results into your own sheets and BI tools.
  • Read-only APIPull the same observations into internal dashboards.
  • MCP serverLet an AI assistant read results without write access.
  • CloudflareSend crawler requests to Agent Analytics, or upload a raw access log.

A role-based working example

“I would fix a blocked guide first, then inspect the next crawl before claiming an improvement.”
Technical SEO lead · Web teamExample Illustrative scenario, not a customer testimonial.

FAQs

What is Agent Analytics?

Agent Analytics is the Kept AI module that monitors the key AI bots visiting your site, including those run by ChatGPT, Gemini, Perplexity and Claude. It reports which bots arrive, which pages they fetch, and whether robots.txt lets them in or keeps them out.

How do I start tracking AI crawlers?

Connect a supported provider such as Cloudflare in a few steps, or upload a raw access log. Once connected, Kept AI starts logging each request AI bots make to your site right away.

Why does AI crawler traffic matter?

Every AI answer begins with a fetch. When bots can't reach a page, or run into errors, that page can't show up in AI results. Watching AI bot requests lets you catch and fix those failures, put effort into the content that matters, and link what bots read to what ends up cited.

How do AI crawlers differ from traditional search engine crawlers?

Classic search crawlers such as Googlebot index pages for ranked search results. AI crawlers such as GPTBot, Claude-Web and PerplexityBot fetch content so ChatGPT, Claude or Perplexity can answer questions directly. Kept AI groups visits by intent, so you can see whether a page was fetched for model training, for a retrieval index, or live in response to a user's question.

Which AI crawlers does Kept AI recognize?

Kept AI recognizes leading AI crawlers for retrieval and training, including those behind ChatGPT, Gemini, Perplexity, Claude, Google AI Overviews, AI Mode and Copilot.

How does crawler data improve my AI search visibility?

Once you see which AI bots visit, what they fetch and where they are blocked, you can fix errors, tune robots.txt and focus on the pages that feed AI answers. Kept AI links crawler activity to the pages that end up cited.

Can I export my crawler data or build on it?

Yes. Use the MCP server or the API to bring the data into Claude, Cursor and the rest of your stack.

See what AI tells buyers about you

Example interface: overview of mentions, sources and open tasks