Kept AI

Testing the Usefulness of llms.txt and Markdown Pages

Kept AIEditorial team
Research Notes

Turn a format proposal into a testable question

A proposed llms.txt file or a Markdown alternative is an experiment in presenting information, not evidence that an answer system will use it. Before creating either, state the problem you expect the format to solve and what observation would change your mind.

For example, a documentation team might suspect that navigation clutter obscures a short installation procedure. Its hypothesis is that a concise text representation makes the same verified instructions easier to retrieve and inspect. That is narrower than promising better AI visibility across all providers.

Separate an index from the content it describes

Treat an llms.txt experiment as a maintained description of selected resources. Treat a Markdown experiment as an alternative representation of particular content. Neither name tells you whether a specific crawler requests it, whether retrieval includes it, or whether a generated answer relies on it.

Define the relationship to the public documentation before publishing. Readers need to know which page is authoritative when two representations differ. A plain text version that silently drops warnings, prerequisites, or version conditions can make a procedure easier to read and less safe to follow.

Prototype one procedure before a whole library

Choose one stable guide with an observable task, such as configuring a local test connection. Preserve the exact prerequisites, command arguments, expected output, and cleanup instructions. Use clearly labeled sample values so nobody confuses illustrative credentials with working secrets.

Use the checklist below to make the prototype reviewable. Test a public copy without an authenticated session, and keep the original page available for comparison. The experiment should reduce uncertainty about one format, not create a second undocumented publishing system.

  1. Compare each procedural step and warning against the authoritative guide.
  2. Check that sample commands remain complete and links resolve correctly.
  3. Assign an update owner and define when an outdated copy must be removed.

Observe access without overinterpreting logs

A successful request establishes access under the tested conditions. A logged request to the new resource establishes that some client requested it, subject to the reliability of your logging and client identification. Neither observation establishes that the content influenced an answer.

Inspect response status, redirects, readable text, and relevant robots rules. Kept’s native Agent can help investigate crawler and crawlability questions, robots, and paths. Do not describe a successful crawl check as universal support for llms.txt or Markdown ingestion.

Illustration · Four distinct levels of evidence Illustrative evidence ladder, not measured data: accessibility, observed requests, retrieved content, and answer influence require different checks; one level does not prove the next.

Compare task accuracy as well as citations

Save a baseline answer to a question requiring the guide’s specific procedure. After making the alternative available, repeat the question with the same recorded context where possible. Inspect whether the answer preserves prerequisites and failure warnings, not just whether a new URL appears.

If the answer improves, retain the underlying samples and mark the conclusion as provisional. Provider changes and ordinary response variability can explain differences. If there is no visible change, distinguish a lack of observable benefit from proof that no system could ever use the format.

Count the cost of keeping copies aligned

Every alternative representation needs an owner and a change path. Ask what happens when a required permission changes or a command becomes obsolete. A lightweight generated copy may be maintainable; a hand edited duplicate can become an attractive but misleading source.

Include upkeep in the decision record alongside observed usefulness. If reviewers repeatedly find missing warnings, fixing the primary guide may offer a clearer benefit than expanding the experiment. Format work should earn its place against other documentation needs.

Choose a stopping rule before expanding

Keep the experiment when it serves a verified reader or retrieval need at an acceptable maintenance cost. Revise or retire it when those conditions fail. This decision does not require declaring the format a universal solution or a hoax; it requires evidence proportionate to the claim.

References