Sometimes an experiment just keeps dragging on because the difference between the winning and losing page(s) hasn’t passed Google’s criteria for significance.
Generally speaking, I think it’s a good idea to keep going until Google announces a winner, but sometimes the pattern is clear, and you really want to move along to your next test, so it’s worth it to call it done.
Here’s what I mean.
I don’t think anything’s going to change in this horse race.
The winning page isn’t a slam dunk, it but it’s good enough to call this race over and move on to the next idea.
Greg Krehbiel Google Website Optimizer
Good marketers know that their opinion on a particular color or design or choice of wording doesn’t matter. What matters is how the market responds. So marketers like to test things.
Which headline works best? Does a starburst announcing a risk-free trial help or hurt response? Is it better to offer a special report as a premium, or just keep the page and the offer simple?
That’s all well and good, and those are good things to test. But as you get into testing you have to keep your eye on testing insanity. You start to see other things you can test, …
- Does my e-mail traffic respond differently than my Adwords traffic?
- Do people respond differently on the weekend than during the week? (I’ve suspected this on some of my tests.)
- Do people respond differently during office hours?
The more you test the more variables you see. Sometimes it seems like you could go crazy coming up with new things to test.
I’ve asked Google to change its website optimizer tool so that you can see results during a given timeframe — so, for example, I could see if my weekend traffic performs differently than my weekday traffic.
But sometimes I’m glad it’s not possible to test these things. If you can keep your tests simple, you avoid testing insanity.
Greg Krehbiel Google Website Optimizer
I don’t know how Google’s website optimizer picks the winning combination on a multivariate test. It’s some complicated statistics that I don’t know and probably don’t want to know.
The trouble is that the margin for error on a complicated experiment (i.e., an experiment with a lot of options) sometimes overwhelms the results.
For example, if the “winning” combination has a conversion rate of 45% ± 10%, and the second place combination has a conversion rate of 42% ± 10%, how sure can you be that the winning combination really won?
You could let the experiment run for a long time until the margin for error decreases. The problem is that you’re continuing to run the crappy options along with the good options, so you’re hurting your overall conversion rate.
A better option is to trim out the clear losers and simplify the experiment, or run a follow-up experiment.
The “best practice” is to make the complexity of your experiment match the amount of traffic on the page — i.e., simple experiments on pages with a little traffic and complicated experiments on pages with a lot of traffic.
Greg Krehbiel Google Website Optimizer