Kept AI

Two Illustrative Team Setups for Testing AI Search Ideas

Kept AIEditorial team
Team Scenario

Two illustrative setups, one standard of evidence

The two team setups below are illustrative planning scenarios, not accounts of customers or measured results. One represents an in-house group that can edit its own documentation. The other represents an agency coordinating work across client approval boundaries. Both need a falsifiable question before they need a larger dashboard.

A workable question is whether clarifying a specific product limitation improves the accuracy of answers to related prompts. A weak question is whether more content improves AI search. The first identifies a change, an audience need, and an observable response; the second leaves too many explanations open.

Give the in-house team a narrow release

In the in-house scenario, a content owner drafts a clarification to one documentation page while a product reviewer checks its accuracy. These are responsibilities in the example, not claims about real employees. Keep surrounding navigation and unrelated pages stable so the change can be inspected without a confusing bundle of edits.

Before release, save a baseline set of answers and the source page as it stood. Include prompts likely to expose the limitation and a small comparison set about untouched topics. The comparison set cannot remove every confounder, but it can reveal when many answers changed for reasons unrelated to the edit.

Give the agency an explicit approval contract

In the agency scenario, the analyst prepares an evidence packet while the client retains publication authority. Agree which project information may be read, where drafts can be stored, and who approves factual claims. Separate each client’s records so a reusable template never becomes accidental sharing of another organization’s material.

An external connected assistant may help summarize authorized Kept AI MCP observations or draft a proposed action. Its access remains read-only through those tools. Send the client a concise page-level proposal with supporting answers, the intended correction, and a verification plan rather than implying the assistant already changed the website.

  1. Write one hypothesis and one answer-quality criterion before editing.
  2. Assign a factual reviewer and a separate publication decision where approval boundaries require it.
  3. Save baseline and follow-up evidence, then document a keep, revise, or stop decision.

Use the same experiment card

Both setups benefit from an experiment card that someone else could follow. Record the hypothesis, changed page, exact prompts, output collection conditions, review owner, and stopping rule. Define success as a content outcome such as fewer factual contradictions, rather than an unexplained movement in a combined score.

For example, the card can ask whether an answer correctly states that a feature requires a particular deployment configuration. Reviewers should judge that criterion against approved product documentation. A brand mention without the configuration caveat is not a successful answer simply because the brand appeared.

Illustration · Two routes to the same review gate Illustrative team workflow, not measured data: an in-house draft and an agency proposal both pass factual review, publication approval, and evidence collection before a follow-up investment decision.

Schedule collection around evidence readiness

First verify that the approved change is live and accessible. Then collect follow-up answers under conditions comparable to the baseline, retaining failures and ambiguous responses. If source discovery has not occurred, mark that uncertainty; do not shorten the experiment until a favorable output happens to appear.

Track effort as well as answer quality. A clarification that takes little maintenance may be worthwhile even when its visibility effect remains uncertain. A recurring manual review with no decision attached may deserve cancellation. Neither team should interpret work completed as evidence that the hypothesis was correct.

Make the next investment conditional

Close the experiment with a short decision memo: what changed, what the observations support, what remains unknown, and what action follows. The in-house team may keep a useful documentation correction. The agency may recommend a second bounded test, with the client approving its scope independently.

Small observational experiments do not isolate model updates, source changes elsewhere, or answer randomness. State those limitations plainly. Their value is disciplined learning and better information for users, not a claim that either organizational structure guarantees citations or that every successful answer was caused by the latest edit.

References