Kept AI

Auditing Site Access for Deep Research Workflows

Kept AIEditorial team
Research Notes

State the audit question precisely

A research-oriented assistant may consult several documents while preparing an answer, but your website does not necessarily expose every useful document in the same way. An access audit asks a narrower question: can an authorized reader reach and interpret the material needed for a particular research task? It does not attempt to reconstruct a provider’s entire internal workflow.

Choose a task with a clear evidence requirement. An illustrative example is comparing a service’s published retention policy with its deletion instructions. The audit should identify which pages support each statement, whether the documents agree, and what an external reader can actually access without private account permissions.

Inventory evidence rather than every URL

Create a compact document inventory with the canonical URL, document purpose, format, access requirements, and owner responsible for its accuracy. Include important downloadable documents and linked appendices. A polished overview may be insufficient if the exception that changes the answer appears only in a separate attachment.

For each document, write the specific question it answers. This reveals gaps that a crawl count cannot: a reachable page can still omit the operative definition or describe an outdated process. Flag conflicting versions before testing assistants, because a successful fetch does not resolve contradictory source material.

Compare the available content surfaces

Inspect the initial response, the content visible to a normal authorized reader, and any downloadable version separately. Note where important information depends on scripts, an expanded panel, authentication, or an embedded file. Do not assume a research assistant receives exactly the same representation as your browser.

Use the retention-policy example to make this concrete. If the overview is readable but the deletion exception is trapped in an inaccessible attachment, record a partial evidence path. Consider adding an accessible explanation with a link to the authoritative policy, while retaining any necessary access restrictions. Do not remove legitimate controls merely to satisfy an audit.

Run a bounded research session

If you test an assistant, use a controlled prompt describing the research question and permitted public sources. Save the exact prompt, session conditions, start and end times, final answer, and citations. Keep the test free of confidential facts that could appear in captures or shared reports.

  1. Confirm that each required document is intended for public access and identify any policy restrictions before the session.
  2. Ask the same substantive question against the documented evidence set. Record omissions without silently rewriting the prompt after seeing the answer.
  3. Compare cited passages with the actual documents. Check whether exceptions survived summarization and whether the cited source supports the conclusion.

Correlate requests without overclaiming

Review narrowly scoped edge or server records for the session window, including a reasonable surrounding interval. Identify the logging layer and timezone. A nearby request is a lead to investigate, not automatic proof that the test caused it. Cached material, background traffic, redirects, and incomplete logging can complicate the timeline.

Describe user-agent matches as such unless identity has been independently verified. Likewise, absence of a matching request does not prove the assistant ignored the document. Keep three evidence columns separate: what your server recorded, what content was accessible during inspection, and what the answer actually cited.

Show the audit model

Illustration · Document inventory + access inspection + answer evidence Illustrative audit model, not measured data or a trace of a provider’s internal research system; each evidence stream has separate limits.

Write a reproducible finding

Finish with the smallest defensible conclusion: which document was inaccessible, which explanation was missing, or which answer misrepresented an exception. Assign a corrective action and retain the relevant version for a focused retest. Improving access and source consistency is verifiable work; promising that a research assistant will read every page or cite your preferred URL is not.

References