Define the relationship you want to study
A brand appearing near the top of a source list and appearing in an AI answer are two observations, not yet a causal story. Better-known brands may receive both placements. A provider may prefer the source itself, or the answer may draw from another page entirely.
Start with a narrow question: within a defined topic and collection window, how does mention frequency vary with position on eligible source lists? Define a mention as a recognizable brand reference in the answer body. Record citations separately rather than counting every cited page as a brand endorsement.
- Write the inclusion rules and the mention definition before reading the answer results.
- Keep source, brand, prompt, and run identifiers so repeated observations remain traceable.
- Report the main comparison alongside a source-exclusion check and the unresolved alternative explanations.
Choose an observation unit
One possible unit is a brand, source page, prompt, and answer run combination. Keep identifiers for all four. Without those identifiers, repeated runs can look like independent evidence even when they reuse the same list and ask almost the same question.
Preserve the source list as it appeared when collected. Record whether it has an explicit ranking, a neutral alphabetical order, sponsored entries, or an editorial grouping. Position three on a ranked shortlist means something different from the third item in an alphabetical directory.
Build a comparison that can fail
For a practical pilot, select one software category and a fixed set of evaluation prompts. Capture eligible lists before examining which brands the answers mention. This reduces the temptation to select only sources that make an interesting result look stronger. Document exclusions before calculating any relationship.
Compare ranked and unranked lists separately. Also compare answers that cite the recorded page with answers that do not. A position pattern present only when a page is cited is a different finding from a pattern seen throughout the sample; neither alone proves that moving an item would change an answer.
Account for repeated evidence
Repeated answers from one prompt share context. Brands within one list share an editor and source reputation. Treat these as clusters when estimating uncertainty, or summarize results at a coarser level if your sample cannot support a reliable model. More rows do not automatically mean more independent information.
Inspect whether a single popular source drives the apparent pattern. Recalculate after excluding that source, then inspect the direction and stability of the relationship. This sensitivity check does not repair every bias, but it reveals when a broad conclusion rests on one editorial choice.
Separate description from intervention
An observational analysis can describe associations within its sample. Testing an intervention requires a different design, such as a legitimately controlled editorial experiment with comparable pages and a prespecified collection plan. Do not rearrange public rankings deceptively just to manufacture an experiment. Reader usefulness remains a requirement.
Even a before-and-after result can reflect provider changes, new sources, or an unrelated product launch. Keep a change log and inspect unaffected comparison prompts where possible. If those prompts also move, the explanation may extend beyond the edited page. State that uncertainty in the result.
Publish evidence readers can inspect
A useful research note includes inclusion rules, the collection window, prompt wording, source-order definitions, missing data, and the analysis method. Share representative answer excerpts only when permitted, with enough context to understand them. Label any diagram that explains the method as illustrative rather than measured evidence.
Conclude at the level the design supports: a relationship observed under specified conditions. Do not turn it into a universal rule that first position wins, or an instruction to buy placements. The valuable outcome is a better question for the next study, alongside an honest account of what remains unknown.