admin 
A new preprint tested whether tweaking just one part of a source changes AI search citations, with everything else kept the same. In the raw numbers, the top result got cited about twice as often as the fifth. When researchers Sriram Selvam and Anneswa Ghosh reversed the order of matched sources, the impact was much smaller and measured zero in a follow-up test.
Posted to arXiv on September 14, the paper isn’t peer-reviewed. The study covers one GPT-5.4 search agent that uses Exa as its search provider, featuring offline replayed conversations and no live webpage edits.
How The Test Worked
The researchers prompted the GPT-5.4 agent to answer 130 common questions by having it perform independent web searches. They recorded every message and search result from the 129 questions it addressed. From these transcripts, they chose pairs of pages that appeared in the same search outcomes and were both screened as supporting the same fact. This screening aimed to find situations where either page could be fairly cited. When true matches were identified, any credit differences were due to how the model apportioned recognition between the two sources, both confirming the same fact.
That left 113 pairs. A later blinded human check confirmed 103 of them as genuine matches. The researchers replayed each saved conversation four ways, placing one page above or below the other and showing its text either as plain paragraphs or rewritten with headings and lists or a table. Only the final answer was generated again.
Both versions of the text were generated by AI rewrites of the original page. Grok 4.3 created nearly all of them, with GPT-5.4 used as a fallback for one pair, and a separate Grok review checked that the facts matched. The wording varies between the two versions, so the authors note that the test compares two rewrites but does not specifically isolate formatting differences.
Raw Position Gap Was Larger Than Swap Effects
In the initial position of a search call, pages were cited 85.1% of the time in saved transcripts, compared to 42.8% for pages in the fifth position. This creates a difference of 42.3 percentage points.
Here, ‘position’ simply means the order of the five Exa results returned in a single search, not where a page ranks on Google or its position on the live web.
The study highlights that search providers usually put more relevant pages at the top, so the raw difference reflects both the position and the quality of the pages. When the researchers moved the same page higher within its pair, the chance it was cited at all went up by 7.9 percentage points. However, this finding wasn’t considered statistically significant after accounting for multiple tests.
Another testing set with 56 pairs, where only the order was switched, showed an estimate of 0.0 points, with a 95% confidence interval from -5.4 to +5.4.
The study explains that the raw gap and the swap results measure different aspects. Overall, it suggests that position did influence citations in some cases, but averages from raw position data aren’t reliable.
Structured Rewrites Got More Credit, Not Clearer Entry
Pages that were rewritten with headings and lists received an average of 0.50 more citation markers per answer compared to the same pages written as plain paragraphs, with a 95% confidence interval from 0.20 to 0.84. The answers in the test were heavily cited, with a median of 29 markers across six documents.
The total number of citations per answer didn’t rise, and the number on the other page barely changed. The authors see this as credit being focused more on the rewritten page.
The main test the researchers conducted, which they planned before starting the experiment, was to see if the page got cited at all. They found that using structured text increased that likelihood by 4.5 percentage points, with a 95% interval from -1.4 to +10.4. The paper points out that this result isn’t conclusive and mentions that the study could reliably detect only effects of about 8.5 points or more.
A more strict comparison, where every word stayed the same but the layout was adjusted to one sentence per list row, boosted citation rates across all 113 pairs. When they repeated the test with a subset, the effect reversed.
In the discussion section of the paper, the authors shared these insights:
“This is an attribution-sensitivity warning, not an optimization tactic.”
Reruns Changed Citation Outcomes
The researchers tested 120 responses again using the same inputs, and found that the decision to cite or not for the target page changed in 15% of these cases, roughly one in seven.
The average count effect remained consistent across these reruns. They estimate that about 45% of the variation in a single run’s effect is due to model randomness.
The authors recommend rerunning citation tests multiple times and sharing how consistent the results are across those runs.
Additionally, SparkToro reported in January that ChatGPT and Google’s AI Overviews each produced the same brand list less than 1% of the time when given the same prompt repeatedly.
Why This Matters
The raw position gap in this test was much larger than the average effect observed when researchers swapped source order. An Ahrefs report from May showed pages cited by AI were about three times more likely to include JSON-LD schema, but adding schema didn’t clearly increase citations.
This raises questions about whether a correlation in a vendor report or your tracking was ever tested by changing the variable, and a single answer is a weak basis for labeling a citation as won or lost.
The study can’t confirm if reformatting a live page boosts citations, since rewrites only applied to text already retrieved, excluding crawling, retrieval, and ranking processes.
Looking Ahead
The researchers re-ran the saved searches on Grok 4.3, discovering that the structured rewrites leaned the same way. However, less than half of Grok’s first replies followed the correct citation format.
The authors recommend more research to test each scenario a few times, explore different search providers and models, and pay attention to both how often citations occur and if a page is cited at all.
Featured Image: Accogliente Design/Shutterstock
