Separate three different language signals
A Chinese answer can cite an English page, and a Spanish prompt can produce a visible search query in another language. These observations describe different stages of a workflow. Record prompt language, observable query language, and answer language separately instead of treating any one as proof of the others.
Source language is a fourth useful field. A page may contain translated navigation around an untranslated technical document, so inspect the cited passage rather than assigning language from the domain name. Mixed-language pages should remain mixed when a single label would conceal relevant detail.
Design prompts around equivalent needs
Start with a task that genuinely matters in each target market, such as finding setup instructions for a product sold there. Ask a fluent reviewer to create equivalent prompts. Literal translation can introduce unnatural wording, different implied expertise, or a product name that local readers do not use.
An illustrative pair could ask, in English and Chinese, how to export project records while retaining source links. Keep the product, desired outcome, and access conditions equivalent. Record deliberate differences, such as a local legal requirement, as separate experimental conditions rather than hiding them inside the translation.
Hold collection conditions steady
Use a fresh conversation for each independent test unless conversational context is the variable under study. Record the provider, displayed model identity when available, search setting, interface, collection window, and any location or locale setting you can actually observe. Do not infer hidden routing from the answer alone.
Interleave languages within a collection window instead of collecting one language now and another much later. Repeat the matched prompts enough to notice variation, while documenting the repetition policy in advance. Repetition helps describe instability; it does not reveal every hidden query or make a small sample representative.
Capture only what the interface exposes
Visible search activity is evidence about the exposed activity, not a complete transcript of internal retrieval. If no query is shown, label query language as unobserved. A citation to an English page is insufficient evidence that the assistant issued an English query.
Build a compact observation sheet using this checklist. Preserve the original wording alongside any analyst translation so another reviewer can check the language classification and spot whether a translation removed a meaningful ambiguity.
- Save the exact prompt and an identifier for its equivalent-language pair.
- Record visible queries verbatim, or explicitly mark them unobserved.
- Save the answer, cited URLs, passage languages, and any collection failure.
Investigate mismatches without overclaiming
When a local-language prompt repeatedly cites another language, inspect the available documentation. The alternative language may have a more complete guide, a clearer title, or the only accessible explanation. Those are hypotheses to test through page review, not conclusions established by the language mismatch itself.
A useful follow-up changes one condition: explicitly request local-language sources while keeping the task unchanged. Compare source relevance and answer usefulness as well as language. A response that obeys the language instruction but cites an irrelevant page is not automatically an improvement.
Translate the finding into a bounded decision
Report the observed pattern with its prompt cohort and collection conditions. Do not generalize it to every user, country, or future provider version. Language, geography, and account configuration can interact, and a single interface rarely gives complete visibility into those interactions.
For content planning, repair a documented local-language gap when it serves readers: translate missing prerequisites, clarify regional terminology, or connect equivalent documentation pages. Then repeat the same test. Better local documentation is a defensible outcome even when citation behavior remains variable.