There’s a song that often comes to mind when I try to evaluate the results of a split test. It’s Jethro Tull’s Nothing is Easy — because testing is sometimes difficult to set up, and it’s often difficult to interpret the data once the test is over.

Marketers do A-B tests because we often don’t know which version of an effort will get the best results. That applies to direct mail, a web page, an email, a telemarketing script … just about anything where you are trying to provoke a response.

It’s good to read best practices and expert guidelines to get ideas for your test, but you still have to test. You don’t know if any given expert’s approach will work for your market. (It can vary.)

You might also wonder whether that particular expert’s advice still applies. (Things do change.)

It’s lovely to hope that you’ll send an email with two different subject lines, and one version will win, and then you’ll be done. Yeah.

But what does “win” mean? It’s often not quite as clear as you want it to be.

For example, let’s say panel A gets a better open rate, but fewer emails were delivered. (Yes, your subject line can affect deliverability.) Panel B had a lower open rate, but more emails got through to the recipients. Which version won?

I’d say panel B won, as a general rule, but there may be reasons to prefer panel A. For example, if you’re experimenting with a subject line for an on-going series like a daily email, the deliverability might work itself out over time, in which case A is the better choice.

Whenever you do a test, you have to measure the right results for your bussiness. In some cases that’s relatively clear. If you’re trying to sell soap, whichever email sells more soap is the better choice, right?

Maybe. What if panel A sells $5000 worth of soap, but only get 500 customers, while panel B only sells $4500 worth of soap but gets 550 customers? Which is better? Maybe adding a new customer is worth more than a sale.

Testing requires you to think about what different metrics mean for your business.

Take the example of an email that’s meant to drive traffic to your website. Panel A and B get about the same number of clicks, but Panel A gets far more unsubscribes and Panel B gets more spam complaints. How do you choose?

I would choose Panel A. An unsubscribe is not a bad thing. If somebody doesn’t want to get your newsletter, you don’t want to send it to them. But you don’t want spam complaints. They hurt your deliverability for all your campaigns.

Also, for some reason, your Panel B is saying “spam” to your recipients, which hurts your brand reputation, and that’s more valuable than any individual email.

Testing is a necessary part of marketing, but it often raises more questions than it solves. Which leads to more testing. Which can become an obsession!

The secret is to stay focused on the numbers that drive your business. Don’t test for testing’s sake. Learn to put the right value on each metric, and adjust your valuation based on the goals of the particular campaign.