How to test cold email subject lines without fooling yourself

Test subject lines on reply rate, not open rate, and do not test at all until you know both variants are reaching the inbox. Open rate stopped being a measurement when Apple's Mail Privacy Protection started pre-fetching tracking images for Apple Mail users, and corporate security proxies do the same thing. A subject line that "wins" on opens may simply have gone to an audience with more iPhones.

The second trap is quieter and worse. If a quarter of your mail is in the spam folder, the thing driving your results is placement, not phrasing. You will spend three weeks optimising words and learn nothing, because the variable you are not controlling swamps the one you are.

Before you test anything

1. Confirm authentication. SPF, DKIM and DMARC all passing, with DKIM aligned to your sending domain. Failing authentication is a placement problem dressed as a copy problem.
2. Confirm placement. Measure inbox placement per provider on the mailboxes you intend to test with. Gmail's Promotions tab counts as delivered in most tooling and reads as a clean hit while your reply rate quietly dies, so look at folder, not just delivery.
3. Check that the mailboxes are comparable. Two mailboxes with different ages, different warm-up histories and different reputations are not two arms of an experiment. They are two different experiments.
4. Hold the body constant. One variable. If the body changes, you are testing the email, not the subject.
5. Work out how long you need. Cold email reply rates are low, which means small differences need a lot of sends to be distinguishable from chance. If your segment is 200 people, you cannot measure a two-point difference. Accept that, and test bigger swings instead of wording tweaks.

Test one hypothesis at a time

A good test has a sentence behind it. "Subject lines naming the trigger beat subject lines naming the outcome" is a hypothesis. "Let's try some subject lines" is not.

Note that these are *structural* differences. Structural tests produce results you can reuse across campaigns. Swapping a comma for a dash produces a result that will not replicate.

Three methods that actually work

Split testing within one campaign. Randomise which variant each recipient gets, and randomise across your mailboxes too, so no variant is concentrated on one sender's reputation. Send both in the same window on the same days. Report replies, not opens.

Sequential testing across campaigns. When your segments are too small to split, run variant A for a full campaign and variant B for the next, keeping everything else fixed — same segment definition, same body, same mailboxes, same send times. Slower, and vulnerable to anything that changed in between, but honest about small volumes, which split testing on 150 recipients is not.

Rendering checks. Not a performance test, but necessary. Mobile Gmail shows roughly the first 35 to 40 characters of a subject line; desktop clients show more. Check how the subject and the preheader read together in Gmail, Outlook and Apple Mail, because the preheader is the second half of the subject line as far as the reader is concerned. Make sure the preheader is not pulling in "View in browser" or an unsubscribe line.

Multivariate testing is usually a mistake here

Testing tone, length and personalisation at once requires a cell for every combination, and cold email volumes rarely support even two cells cleanly. If you genuinely have the volume, test one dimension per campaign and build a picture over a quarter. If you do not, the multivariate result is noise with extra steps.

Time-of-day testing

Worth doing, but it is a separate experiment from the subject line and it interacts with your sending schedule. If your tool paces sends across a window, two variants sent at different times also hit different queue positions and different mailbox warm-up states. Test send time with one fixed subject line, and only after you have a subject line you trust.

What not to do

Do not use a "spam word" checker as your test. The word "free" in a subject line is not what gets you filtered. Authentication failures, complaint rates, bad lists and cold domains are. Writing around a word list produces stilted copy and fixes nothing.

Do not fake a reply or a forward. "Re:" and "Fwd:" on a thread that never existed is a deceptive subject line, which CAN-SPAM prohibits outright, and recipients who notice report the message. That is the most expensive possible outcome of a subject line test.

Do not declare a winner on a day of data. Weekday effects alone are larger than most subject line effects.

Do not test during a deliverability incident. If a domain was blocklisted, a mailbox got rate-limited, or you just raised volume, pause testing until placement is stable again.

Do not report open rate to anyone as if it were real. If you must track it, track it as a trend on a single segment and never compare it across audiences.

Do not test subject lines when the body is the problem. If a five-paragraph pitch with three links and an image follows, no subject line saves it.

Where MailPilot fits

MailPilot gives you the control variable a subject line test needs. It warms each mailbox over OAuth on Gmail and Outlook or over SMTP/IMAP, checks SPF, DKIM and DMARC for every sending domain, and measures real per-provider inbox placement with a seed-inbox network — including whether Gmail is filing you under Promotions rather than the inbox. Confirm placement is steady and comparable across your mailboxes first, and your subject line results start meaning something.