Kept AI

Tracking Query Expansion Changes Across Countries and Time

Kept AIEditorial team
Research Notes

Define the expansion you can observe

Query expansion describes the additional searches an assistant may use while working on a request. A study can analyze exposed query strings, but it cannot assume that the interface reveals every retrieval step. Begin by naming the observable unit: a displayed query associated with one recorded answer attempt.

Keep query count, query length, and query purpose separate. More queries do not necessarily mean broader research, and a longer string does not necessarily express more concepts. An assistant might repeat a search, add a product identifier, or narrow a location without changing the underlying task.

Separate country from language and access

A country comparison needs more than a country name in a spreadsheet. Record how the country condition is established: explicit prompt wording, observable account settings, or a documented collection location. These conditions are not interchangeable, and none establishes every aspect of a provider’s internal location handling.

Use equivalent tasks and keep language as an explicit variable. If all prompts for one country use English and all prompts for another use Chinese, an observed difference cannot be cleanly attributed to country. A smaller crossed comparison can be more informative than a larger confounded collection.

Freeze a panel and version the method

Choose a stable panel of tasks that remain meaningful across observation windows. Archive the exact prompts, inclusion rules, and collection procedure before starting. Add new topics to a separate exploratory panel rather than replacing difficult or unproductive prompts inside the historical baseline.

For example, an illustrative study could compare requests for software deployment guidance across two country conditions. Keep the software version and desired deployment outcome fixed. If a regional requirement changes, annotate it and decide whether the affected prompt still belongs in the comparable panel.

Specify length and structure measurements

Character counts are simple to reproduce but are not directly equivalent across writing systems. Whitespace-separated word counts can be especially misleading for Chinese. Preserve raw strings, declare the segmentation method, and compare within a language before interpreting a combined multilingual length trend.

A structural codebook can classify whether a visible query adds a location, entity, constraint, comparison, or follow-up clarification. Permit multiple labels when appropriate. Test the codebook on ambiguous examples and retain reviewer disagreements rather than forcing every query into an apparently tidy single category.

  1. Save each raw query in its original order within the answer attempt.
  2. Mark unexposed queries as unobserved, never as zero-length searches.
  3. Store the measurement version and any manual classification decision.

Compare windows without hiding collection breaks

Collect country conditions within comparable windows and keep a log of failures. A provider interface change can alter what is exposed without changing the underlying search behavior. If visibility changes, show a collection break and avoid drawing a continuous trend line across it.

Inspect paired prompt changes before pooling results. A shift driven by one task family may disappear in other tasks. Report the number of observed attempts, missing observations, and the spread of query lengths so a central value does not hide a highly uneven distribution.

Use the study to choose an investigation

A detected expansion change is a lead, not an explanation of ranking or commercial outcomes. It may reflect interface behavior, product updates, task composition, or genuine retrieval changes. The observational design usually cannot isolate those causes without additional controlled comparisons.

Use recurring constraints to inspect whether your documentation answers the corresponding reader questions. If searches increasingly expose implementation prerequisites in the observed sample, review those instructions for completeness. Publish the study method and limits alongside the finding, and avoid promising that matching query phrasing will earn citations.

Illustration · A controlled comparison across observation windows Illustrative study layout, not measured data: fixed prompt panels cross country and language conditions, while method changes create labeled breaks between observation windows.

References