Not every test needs to be run until a statistical significance of 95% is reached. There, I said it.
I am fully aware that this post may be unpopular with many Conversion Rate Optimization (CRO) experts out there.
But that is ok.
From a true statistical standpoint, I recognize that killing a test too early is not the ideal. However, I am approaching this from an operational perspective.
There is a cost to every test – in most cases this includes the cost of media. You are spending your marketing dollars to drive traffic to your landing page. Every dollar being spent on traffic to a poorly performing landing page is a missed opportunity.
For this reason, if your test is a loser – and I’m not talking about a 10% decrease but rather a REAL loser – cut bait and move on, FAST!
Here is an example. A client of mine designed a completely new landing page to test against their control. We weren’t testing a modification (like copy or image testing on an existing template), but rather a completely different look and feel. This was an overall A/B test – 2 completely different designs including the form layout.
Admittedly, I liked the new design – thought it was cleaner and did a better job of highlighting the company’s value proposition. But that is why we test – what we like doesn’t matter. All that matters is the performance.
In this case, after 4 days the new landing page was converting just half as well as the control. And, this was consistent – for every hour of every day the control was performing 100% better than the test variation.
To me, this also meant that half of the marketing spend was performing half as well as it could be.
And for me, that was enough. I made the quick decision to turn off the test and have all of the remaining traffic from this campaign go to the control landing page.
I do normally want tests to run for a minimum of 1 calendar week to offset any differences based on day of the week. In a vacuum I also prefer getting 95% (or better yet 99%) significance on a test before declaring a winner. That is CRO Best Practice.
However, maximizing overall campaign, and business, performance is even more important to me.
Perhaps the results would have turned around and the new creative would end up winning over a long test – I just didn’t want to take that chance .