Test subject lines and learn something

Subject line testing is the most-run and least-learned-from experiment in marketing. Two variants go out, one wins, everybody notes it, and the next campaign starts from instinct again because nobody wrote down what the result meant.

Test one difference at a time

An A/B test between two subject lines that differ in length, tone and content tells you which of three things mattered. Which is to say, nothing.

Write three subject line variants for this campaign, each testing exactly
one difference against the control:
  one that changes only length
  one that changes only whether it asks a question
  one that changes only whether it names the specific benefit

Keep everything else identical. Say what each one is testing, in a sentence,
so we can write down what a win would mean.

That last clause is the actual product of this step. A variant whose hypothesis you cannot state is a variant whose result you cannot use.

Read the result honestly, including the sample size

This is where most subject line testing quietly falls apart.

The test has finished. Tell me:
  which variant won and by how much
  whether the difference is large enough to mean anything given the list
    size, or whether it is within noise
  what the result says about the hypothesis we were testing

If the difference is not meaningful, say so plainly. Do not pick a winner
to be helpful.

"Do not pick a winner to be helpful" is the line that decides whether this exercise compounds or just generates activity. On a list of two thousand, a three percent difference in opens is noise, and treating it as a finding means next quarter's decisions rest on a coin flip somebody wrote down. A test that concludes "we cannot tell" is a successful test.

Write down what you learned, not what won

The winning subject line is worth almost nothing next month. The finding — questions do better than statements with our audience — is worth something for a year.

Record it as a decision with its reasoning, so the next campaign starts from evidence and so somebody can challenge it later when the audience changes.

Retest the beliefs occasionally

An audience that has learned your pattern stops responding to it. A finding from eighteen months ago is a hypothesis again.

What good looks like

Every test has a stated hypothesis, an honest reading of whether the result means anything, and a written finding that outlives the campaign.

The failure to watch for is a growing list of "winning" subject lines with no principle connecting them. That is a record of coin flips.