admin 
Testing is everywhere in SEO: Our industry started testing titles, then moved on to internal linking, content formats, templates, and now we are increasingly curious about the impact of AI-generated changes.
But running a test and learning something useful from it aren’t necessarily the same thing.
You can make a change, watch a metric move, and still have very little evidence that what you implemented caused it. Equally, you can see no measurable movement and incorrectly conclude that your change had no effect.
This is where SEO can borrow something from science.
I ended up in SEO by chance, like many of us do. And I brought my research background with me, along with the curiosity of knowing why I was observing certain things, rather than just limiting myself to report on it.
That’s what made me particularly interested in running SEO tests and eventually made me train teams to think like a scientist when running one.
Scientific experiments are designed not simply to produce results, but to make it harder for us to fool ourselves about what those results mean. That requires more than a testing tool or a sparkly before-and-after graph. It requires:
- Asking the right question.
- Defining what evidence would answer it.
- Designing a test capable of producing that evidence.
- Being willing to accept an answer you didn’t expect.
Most SEO testing mistakes happen when we skip one of those steps.
So, here are the ones that can make the difference between simply running a test and actually knowing whether your SEO change worked – and yes, I made most of them myself at some point, but we learn from experience, right?
1. Not Having A Solid, Falsifiable Hypothesis
Good research always starts with a question, not a hypothesis.
Our industry prides itself on data-driven decisions, so what is one thing that you need an answer to, in order to make a decision that benefits your business?
Commercial challenges (and how they manifest themselves in organic performance) are usually what drives most fruitful tests. A real uncertainty or problem in the business is where you want to start from. For example, why is traffic down on product pages since last quarter? Did your AI mentions drop in the past month, and would targeted content improve visibility? What kind of initiatives are most likely to move the needle, if you were to create a roadmap?
A good experiment starts with something the business genuinely doesn’t know and that matters enough to investigate. Don’t start by asking what you can test. Start by asking what you need to learn in order to make better decisions, and turn those into specific, falsifiable hypotheses.
“Improving page titles will improve SEO performance” is not really a hypothesis. What constitutes “improving”? What is “performance”? What outcome would show that you were wrong? Be as specific as you can be, even if it means you have to make some educated guesses.
A useful hypothesis needs to make a prediction that the experiment could conceivably contradict. So before you run the test, ask yourself: What result would prove my hypothesis wrong? If there isn’t one, you’re not really testing it.
For example, your hypothesis could be “Updating the titles to include a country of reference on 700 pages will generate an additional 10% clicks over a four-week period, as compared to the four weeks preceding the change.”
Obtaining a 12% uplift will prove your hypothesis; 9% would falsify it (although that doesn’t mean your result isn’t meaningful – read on to understand why).
2. Not Isolating Variables Or Having A Valid Comparison
One of the first things you learn when designing an experiment is that observing a change is not enough. You need to be reasonably confident about what caused it.
This is also one of the hardest principles to translate into SEO, because websites are not sterile lab environments where everything is controlled for.
While you are changing titles or testing a new content placement on a template, a number of other things could be happening:
- Your Product team deploys a new feature that affects the whole site.
- Marketing launches a campaign.
- Search demand changes.
- Google rolls out an update.
- Your devs release something on the same pages you are treating.
And while they are all tied to different business goals and events, the principle remains the same: the more things you change at once, the less confident you can be about which one produced the result.
Researchers call these alternative explanations “confounding variables,” and I have fallen victim to them myself.
In one of my recent studies, I was interested in whether people responded differently to AI-generated and human-created content. Except I eventually realized I couldn’t confidently attribute the differences I was observing to the origin of the content. The materials differed in other meaningful ways, including their messaging and how familiar participants were with them.
So I scrapped the original design and went back to the drawing board.
Painful? Yes, and it took me significantly longer to build stimuli that minimized confounding variables. But considerably less painful than confidently getting to the wrong conclusion.
SEO testing requires the same discipline, even if perfect experimental control is rarely possible.
That means asking:
- Do I have a suitable control or comparison?
- If I do, are the groups actually comparable (or am I comparing page templates with a completely different intent and traffic history)?
- What else changed during the experiment?
- Did seasonality, a campaign, product launch, or algorithm update coincide with it?
- If performance changed, what alternative explanations could account for it?
Unfortunately, we cannot stop everything while we test, but we can keep the lines of communication open with other teams and understand what they are doing. Keep a log of major changes across SEO, engineering, product, and marketing if you can – it helps more than you know when you run your first analysis after the test launch.
The goal isn’t to pretend your website is the sterile laboratory that it’s not. It’s monitoring the other plausible alternative explanations so that you can have reasonable confidence in what caused what.
3. Deciding What Success Looks Like After Seeing The Results
There is an old joke in SEO: If traffic goes up, it’s because of your work. If traffic goes down, it was a Google update.
That’s exactly one of the risks of testing a hypothesis you deeply care about: falling victim to our own confirmation bias.
Researchers have their own version of this problem. If you run enough analyses, look at enough outcomes, or keep changing the question, eventually you will find something interesting.
This is partly why preregistration exists in research: Researchers document what they intend to test and how before seeing the results. I’m not suggesting you preregister your next title test on the Open Science Framework (unless it’s revolutionary; in that case please do and send us all a link), just borrow the principle. Before running the experiment, commit to the following:
- What are you measuring?
- What is your primary outcome?
- Over what period?
- What comparison matters?
- And what would count as evidence for or against your hypothesis?
Otherwise, it becomes very easy to take any signals that confirm your hypothesis as evidence in favor, irrespective of what the actual outcome is.
For example:
- “Well, traffic didn’t move, but impressions did.”
- “Impressions are down, but CTRs are up.”
- “Average position is down, but look at these six new keywords we gained!”
I don’t mean to rain on anyone’s parade, but if you give yourself enough metrics and enough freedom after the fact, something will eventually look positive. You just cannot conclude it’s definitely because of your change.
That’s why I stress that a hypothesis shouldn’t be something you aim to definitely prove, but something you want to test. Even if this means you might eventually end up falsifying it.
If you enter an experiment thinking “I know this works, and I’m going to demonstrate it,” you have immediately given yourself something to defend. You start looking harder for explanations when the result is negative than when it’s positive. You extend the test because perhaps it just needs another week. You remember the successful experiments and hide the unsuccessful ones under the rug – not acknowledging that even the tests we “lose” can teach us something about our audience, and what works or doesn’t on our website.
But your hypothesis isn’t your strategy; it just informs it.
Scientists are not supposed to be emotionally attached to their hypotheses surviving. In reality, we’re human, so of course it happens. That’s exactly why research methods try to protect us from ourselves – and SEO teams sometimes need the same protection.
4. Drawing Conclusions Your Test Can’t Support
Imagine: weeks of running an experiment, collecting data, analyzing it, and getting to the end only to discover that your answer is: “We don’t know.”
Welcome to research. We’ve all been there. And it’s ok.
In SEO, we often want tests to produce a binary answer: it worked, or it didn’t (even if we end up using “It depends” more often than we care to admit).
But there are at least three possibilities:
- There is evidence of an effect.
- There is evidence against the effect we predicted.
- There isn’t enough evidence to reach a reliable conclusion.
And they are not the same thing.
If your experiment is too short, your sample is too small, performance is highly variable, or the effect itself is relatively subtle, your test may simply not have had enough opportunity to detect it.
This matters particularly in SEO because our experimental clock isn’t always the same as the calendar clock: a change might take a while to get crawled and indexed, and search engines need time to respond. User behavior varies across days, weeks, and seasons.
So ditch the hard rule of running something for a strict and short time window because your team sprint dictates it. Seven days sound tidy, but doesn’t necessarily mean you’ve given the test seven meaningful days of exposure.
Bear in mind, more data will not rescue a badly designed experiment, but it will help a well-crafted one more likely to answer the question you asked if you’re struggling to get anything out of it.
This is where one of the most commonly misunderstood research principles becomes useful:
“Absence of evidence isn’t necessarily evidence of absence.”
If you didn’t detect a difference, ask yourself whether you have evidence that the change had no meaningful effect, or whether your experiment simply couldn’t tell, and you need more data to nail that down.
5. Treating Every Test As An Isolated Event
Treat your experimental program to gain compound knowledge, not for tiny little discoveries alone.
In research, we rarely expect a single study to settle a question forever: One study tells you what happened with a particular sample, using a particular methodology, under particular conditions. Confidence grows when findings replicate, when related experiments point in the same direction, and when evidence accumulates.
Your SEO tests should work in much the same way.
One experiment might tell you that changing a template produced an uplift on one group of pages, on one website, during one period, but it doesn’t mean that “Google definitely prefers this.” Nor that you should turn it into SEO law and apply the successful change indiscriminately across every market and page type you own.
This is why I am slightly obsessive about keeping experiment logs.
At minimum, document the question, hypothesis, treatment, sample, dates, relevant context, result, interpretation, limitations, and what you decided to do next – across every test, even the one that falsified your hypothesis, or the inconclusive ones.
In six months from now, all of that knowledge taken together will tell you a story and become much more valuable than it looks today: maybe you run something similar in another market, or the same pattern appears on another page type. Maybe three supposedly unrelated experiments all start pointing towards the same underlying behavior.
Science progresses because we build on what came before, including the studies that didn’t produce the result researchers expected, and SEO teams can build knowledge in exactly the same way.
One test gives you a result, but 20 properly documented tests start telling you something about your website, your audience, your markets, and which assumptions repeatedly survive contact with reality.
The value isn’t in the singular test, but in the body of evidence you’re building.
6. Testing For Testing’s Sake
Research starts with a question because we’re trying to reduce uncertainty. In business, reducing that uncertainty should also help us start an action.
Before running a test, try completing this sentence:
“Depending on what we find, we will…”
Scale the change? Revert it? Change direction? Do nothing?
That last one matters. A statistically convincing result isn’t necessarily a commercially meaningful one. If a change improves organic clicks by 0.3% but takes six months of engineering resources to roll out, the test may have “worked” while still giving you a good reason not to implement it.
So try to ask two questions:
- Can we reasonably believe there is an effect?
- Is that effect large enough to matter?
I’ve worked in environments where the number of experiments running was almost treated as a marker of maturity. But value isn’t measured in volume, but in learning and actions we take as a result.
A failed or inconclusive test can be valuable if it stops you investing in something you assumed would work. Equally, if none of the possible outcomes would change what you’re going to do, why are you testing it?
In academia, an experiment ends with a discussion: What did we learn? What are the limitations? What should we investigate next?
SEO testing should end in much the same place.
Because ultimately, the output of an experiment isn’t the chart, but a decision. And if you’re not the one in charge of the decision, this article by Dan Taylor can help you communicate what matters to the people who do.
Final Thoughts
You don’t need to turn your SEO team into a research lab, nor use complicated statistical models or methods that are longer to write than the actual run of the experiment.
Thinking like a scientist is much more about disciplined curiosity.
Start with something you genuinely need to know. Form a hypothesis that can be wrong. Design a test that gives you a reasonable chance of answering it. Try to rule out alternative explanations. Decide what evidence means before you see it. Accept the answer even when it isn’t the one you wanted. And document what you learned so the next experiment starts where the previous one finished.
Researchers spend a surprising amount of their lives discovering that their beautiful hypotheses were wrong, their experiments didn’t work, or reality was considerably messier than expected. I know I did.
And that’s not a failure of the scientific process – it IS the scientific process.
Applied to SEO, it’s what turns testing from a collection of before-and-after charts into a body of evidence you can actually use to make better decisions. Take that compound, build your knowledge about your website and audience behavior, and form a strategy that matters – for your business and your audience.
More Resources:
Featured Image: Roman Samborskyi/Shutterstock
