In an earlier post, I wrote on “Conversion Rate Optimization For Newbies”.  Since that post I have had some interesting conversations (ok, email exchanges) with folks about designing CRO experiments.

These exchanges made me realize that I should post some quick thoughts on the topic.

I believe in simplicity.  For this reason, I often prefer A/B testing to Multi-Variate testing, particularly if a site is not getting a ton of traffic.  With low traffic volume, multi-variate testing can take a very long time.  Also, the experimental design becomes far more complicated.

So, for the purpose of this post – here are the first 2 things you need to do before starting any A/B  CRO experiment:

Determine your KPIs

This is extremely important. Without determining what your true goal is, and what the KPI for measurement will be BEFORE TESTING, it can become very difficult to even determine a winner.

For example, say you are working for an Insurance company.  You are driving traffic to a landing page about homeowner’s insurance that includes a form for visitors to fill out in order to request a free homeowner’s insurance quote.  Here your KPI could be simple – Conversion Rate (# of homeowner’s quote requests / # of visits).

However, what if you are told that the company also is interested in asking these visitors if they would also like a car insurance quote.

Now, what is the KPI that should be used to determine a winner?  It could be any of the below, and I’m sure there are even more you can think of.

1)      Total Quote Requests – # of homeowner’s quote requests + # of car quote requests

2)      Conversion Rate – conversions/visits

3)      Quote Requests per Lead – (# of homeowner’s quote requests + # of car quote requests) / conversions

4)      Quote Requests per Visitor – – (# of homeowner’s quote requests + # of car quote requests) / visits

I’m sure you can now see why this step is so important.  If your KPI is #2, you may want to have the simplest form possible to move your visitors right through the process.  However, if your KPI is #1 you may then want to include a prominent cross-sell asking visitors to also request a car insurance quote.

Note that there is no true right answer here – it is really up to the business owner and it can even change over time! The key for us, is that we need to have this determined up front before we can move on to the next step and begin testing.

Determine your hypothesis

Now that you have determined what the most important metric is, it is time to create your hypothesis.  I like to use “If, Then” statements here.

For example, if we determine that #1 above (Total Quote Requests) should be the KPI here, our hypothesis could be something like “If we add an in-form cross-sell for Auto Insurance, Then we will increase our total # of quote requests”.

There it is – that is your hypothesis.  You can now get your creative set up to prove your hypothesis to be true or false.

Bonus Tip

An A/B test can really be A/B/C/D (or any number of variations that your traffic can support).  I would try multiple versions to test out this hypothesis – different wording, different placement in the form, different visual presentation of the cross-sell. . . If your site doesn’t get enough traffic to run more than 2 variations, no worries.  You can simply test 2 at a time, and have the winner stay on to test against your next variation.

In this way, you are truly testing out the hypothesis (that adding a cross-sell will increase your total quote requests) and not testing out 1 particular creative approach.

Happy Testing,


PS – This all may sound obvious, but trust me – getting your KPI and Hypothesis nailed down prior to hitting the go button on your test will save you a lot of issues down the road and make for a much cleaner testing experience.

For successful CRO, determine your hypothesis before you get started

For successful CRO, determine your hypothesis before you get started

Last updated by .

Tagged with →  

Leave a Reply