Researchers investigated if modifying a single element of a source affects AI search citations while maintaining all other factors constant. The primary finding was that altering the order of matched sources had minimal impact, with no noticeable difference in citation frequency.
The paper, uploaded to arXiv on September 14, has not undergone peer review. It discusses a GPT-5.4 search agent that utilizes Exa as its search engine, with archived conversations and no real-time webpage modifications.
How The Test Operated
The researchers directed the GPT-5.4 agent to respond to 130 frequently asked questions by conducting its own internet searches. They documented all interactions and search findings related to 129 questions. By examining these records, they selected pairs of websites that showed up in the same search results and were deemed to back the same information. This process was carried out to locate instances where either website could be appropriately referenced. Upon discovering accurate matches, any discrepancies in acknowledgment were a result of how the model attributed credit between the two sources, which both verified the same information.
113 pairs remained after that. Subsequent verification by a blinded individual confirmed 103 of them as legitimate matches. The researchers then reviewed each stored conversation in four different formats, with variations in layout and presentation. The final response was the only one regenerated.
Both versions of the text were produced by AI rewrites of the original content, primarily by Grok 4.3, except for one pair where GPT-5.4 was used as a backup. The authors ensured the accuracy of the information through a separate review by Grok to confirm consistency, although there are differences in wording between the two versions.
The difference in initial positions was greater than the impact of swaps.
Pages at the beginning of a search call were referenced 85.1% of the time in saved transcripts, while those in the fifth position were cited only 42.8% of the time, resulting in a 42.3 percentage point disparity.
The term ‘position’ in this context refers to the sequence of the five Exa results displayed in a search, rather than the ranking of a page on Google or its placement on the live web.
Search engines typically prioritize more relevant pages by placing them at the top, so the observed variation is influenced by both page ranking and quality. When the researchers elevated a page within its pair, the likelihood of it being cited increased by 7.9 percentage points. Nonetheless, this outcome was not deemed statistically significant once adjustments were made for multiple tests.
Another test set of 56 pairs, where only the sequence was changed, indicated a measurement of 0.0 points, with a 95% confidence interval ranging from -5.4 to +5.4.
The study clarifies that the raw gap and swap outcomes assess distinct elements. It indicates that while position did impact citations in specific instances, the averages derived from raw position data are not dependable.
Structured revisions received more recognition but did not lead to clearer input.
Pages that were revised with headings and lists resulted in an increase of 0.50 citation markers per answer compared to pages written in plain paragraphs, within a confidence interval of 0.20 to 0.84. The test answers had a substantial median of 29 markers across six documents.
The authors believe that credit is now more concentrated on the revised page, as the total number of citations per answer remained the same and there was minimal change on the other page.
The researchers’ primary experiment aimed to determine if the page received any citations. They discovered that employing structured text raised the chances by 4.5 percentage points, within a 95% range of -1.4 to +10.4. The study indicates that this finding is not definitive and notes that effects of around 8.5 points or higher could be reliably detected.
A stricter comparison, maintaining every word but adjusting the layout to one sentence per row in a list, increased citation rates for all 113 pairs. However, when they conducted the test again with a smaller group, the effect was the opposite.
In the paper’s discussion section, the authors provided these viewpoints:
“This serves as a caution about attribution sensitivity, rather than a strategy for optimization.”
Citation outcomes were influenced by repeated airings.
The researchers reevaluated 120 responses with the same inputs and discovered that the choice to reference the target page shifted in 15% of these instances, approximately one out of every seven.
The typical impact stayed the same in all repeated runs, with approximately 45% of the variability in an individual run attributed to model randomness.
The authors suggest repeating citation tests several times and noting the consistency of the results across those repetitions.
SparkToro found that ChatGPT and Google’s AI Overviews rarely generated identical brand lists when provided with the same prompt multiple times, according to a report in January.
Why This Is Important
The test revealed a significant difference in the position gap compared to the typical impact seen when researchers exchanged the order of sources. An Ahrefs study in May found that pages referenced by AI tended to have JSON-LD schema about three times more frequently, yet the inclusion of schema did not definitively lead to more citations.
This brings up concerns about whether a correlation in a vendor report or your tracking was actually verified by altering the variable, and relying on just one response is not a strong enough reason to categorize a citation as either successful or unsuccessful.
The research does not provide evidence that reformatting a webpage while it is live increases citations, as the changes were made only to existing text and did not involve crawling, retrieving, or ranking processes.
Planning for the future
The researchers repeated the saved searches on Grok 4.3 and found that the structured rewrites showed similar results. Yet, fewer than 50% of Grok’s initial responses adhered to the proper citation format.
The authors suggest conducting additional studies to examine various scenarios multiple times, try out different search providers and models, and focus on the frequency and presence of citations on a page.
Featured Image: Accogliente Design/Shutterstock