Kept AI

What Server Logs Can Tell Your AI Search Team

Kept AIEditorial team
GEO Know-How

Start with what a request proves

Server logs provide a record of requests that reached the layer producing those logs. They can help explain access problems and request patterns. They do not show every decision an AI system makes after receiving content. A successful response is evidence of delivery at that layer, not proof that a page was understood, indexed, or cited.

Begin an investigation with a bounded question: can a particular crawler access a newly published guide? That is more answerable than asking whether AI likes your website. Write down the page paths, observation window, and relevant infrastructure before opening a large log collection.

Locate the layer that holds the evidence

An origin log and an edge log describe different parts of the request journey. A cached response may be served without a new origin request. A security rule may block access before your application runs. Record which layer each extract represents, and avoid treating a missing origin entry as proof that nobody requested the page.

Export only the fields needed for the question: timestamp with timezone, host, path, method, status, response size, user agent, and a safe request identifier where available. Keep sensitive query parameters and personal data out of shared extracts. Sampling and retention limits belong in the analysis notes.

Classify crawler identity with care

A user-agent string is a claim made by the requester. It is useful for grouping records but is not sufficient identity verification by itself. When attribution matters, compare available network evidence with the provider’s current published verification guidance. If you cannot complete that check, label the group as matching a user-agent pattern.

Keep verified identity, suspected automation, and ordinary browser traffic separate. Also preserve uncertainty about request purpose. Even an identified provider may operate several systems, so a request should not automatically be labeled as training, answer retrieval, or a particular research feature.

Trace one path before aggregating

Consider an illustrative guide that returns a redirect to a revised URL. A summary count might show successful requests, while a path review reveals that the destination returns a permission error. Inspect the entire redirect chain and final content before concluding that the guide is accessible. This is a hypothetical diagnostic example, not a reported customer incident.

  1. Select one canonical page and its redirects, then filter a short time window at the relevant logging layer.
  2. Inspect status codes alongside response size and content. A success code alone does not rule out an empty shell or a challenge page.
  3. Compare robots rules, access controls, and security events for the same path. Distinguish an intentional restriction from an accidental failure.
  4. After an authorized fix, repeat the narrow check and retain both extracts. Describe the verified access change without claiming a visibility increase.

Use agent analytics as investigation context

Kept AI’s native Agent can analyze crawler activity, crawlability, paths, robots rules, and a thirty-day conversion summary. Use that context to prioritize a path investigation, then verify the relevant evidence. An aggregate summary cannot identify the cause of every blocked request or establish that an AI answer sent a specific visit.

Treat robots overrides and goals as deliberate configuration changes. Goals are limited to ten, and confirmation is the default unless configuration bypasses it. Review the intended rule and scope before applying a change; removing a restriction solely to increase request counts is not a meaningful access strategy.

Draw the boundary between access and outcomes

Illustration · Request evidence → access diagnosis → separate answer observation Illustrative evidence chain, not measured funnel data; server requests do not by themselves establish citations or conversions.

Report a finding someone can test

A useful report names the affected path, logging layer, time window, observed response, and next check. State whether identity was verified and what data was missing. Keep citations and downstream visibility in a separate observation record. This makes a technical fix reviewable without turning an access log into an unsupported marketing attribution claim.

References