SMTP TLS and certificate errors, and what they cost you

SMTP's encryption is normally opportunistic: the sending server offers STARTTLS, and if the handshake fails it may fall back to plaintext or defer the message. That is why a broken certificate on a mail server often produces no obvious error — just mail that is slower, unencrypted, and increasingly filtered. The moment a receiving domain publishes MTA-STS in enforce mode, the same certificate stops mail outright.

How TLS fits into an SMTP session

220 mail.example.com ESMTP Postfix
EHLO sender.yourdomain.com
250-mail.example.com
250-STARTTLS
250 SIZE 52428800
STARTTLS
220 2.0.0 Ready to start TLS
<TLS handshake: certificate presented, verified, cipher negotiated>
EHLO sender.yourdomain.com
MAIL FROM:<you@yourdomain.com>

Two things to notice. The EHLO is repeated after the handshake, because everything before it was in the clear. And ports differ: 587 and 25 use STARTTLS (upgrade an existing plaintext connection), while 465 is implicit TLS (the handshake happens before any SMTP verb). Implicit TLS on 465 is the more robust choice for client submission; port 25 server-to-server traffic is STARTTLS in practice.

Diagnose it in one command

openssl s_client -starttls smtp -crlf -connect mail.example.com:587 -servername mail.example.com

For implicit TLS, drop -starttls smtp and use port 465. Read three things from the output:

• Verify return code: — 0 (ok) or a numbered error.
• The certificate chain, and whether the top of it is a trusted root.
• subject= and the X509v3 Subject Alternative Name list, which must contain the hostname you connected to.

Check expiry quickly:

openssl s_client -starttls smtp -connect mail.example.com:25 </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer

The errors, and what each one means

Expired certificate (certificate has expired, code 10). The single most common cause of a mail server TLS outage, and nearly always an automated renewal that stopped working — a renewal hook that no longer reloads the MTA, a DNS or HTTP challenge that broke, a certificate renewed on disk but never read back into Postfix or Exim. Fix the renewal *and* the reload, then monitor expiry externally.

Self-signed certificate (code 18 or 19). Verification fails because nothing in the chain is trusted. Acceptable only on an internal relay where you have deliberately pinned it. For anything that talks to the public internet, use a publicly trusted certificate; they are free.

Name mismatch. The certificate is valid but issued for a different hostname than the one being connected to — typically because the MX points at mail.example.com while the certificate covers only example.com, or because a hosting provider's shared certificate covers its own domain. Reissue a certificate that includes every name your MX records and SMTP banner use.

Incomplete chain. This is the error that behaves differently everywhere. Browsers often paper over a missing intermediate by fetching it; MTAs generally do not. The symptom is a certificate that validates in your browser and fails from another mail server. Serve the full chain — leaf plus intermediates — in the file your MTA loads, not just the leaf.

Untrusted issuer (code 20, unable to get local issuer certificate). Either the chain is incomplete as above, or the *sending* machine's CA trust store is stale. On a long-lived server that has not been updated in years, update the ca-certificates package before blaming the remote end.

Weak or unsupported protocol. TLS 1.0 and 1.1 are deprecated and disabled by major providers; TLS 1.2 is the working minimum and 1.3 is current. A handshake failure with no certificate error at all usually means no shared protocol or cipher. Check what your server offers:

openssl s_client -starttls smtp -tls1_2 -connect mail.example.com:25

Export-grade and anonymous ciphers should be off; so should RC4 and 3DES.

Revoked certificate. Rare, and always a security event, not a configuration one. Reissue and investigate why.

`454 4.7.0 TLS not available due to local problem`. The server advertised STARTTLS and then could not deliver it — usually a certificate file it cannot read, often after a permissions change or a renewal that wrote files as root-only.

Why this reaches deliverability

• Downgrade to plaintext. Receivers increasingly log and weight transport security. Gmail marks unencrypted messages with a broken-padlock indicator visible to the recipient.
• Deferrals that look like throttling. An MTA configured to require TLS will defer with a 4.x.x code on handshake failure. Repeated deferrals are indistinguishable from reputation throttling until you read the logs.
• MTA-STS turns soft into hard. A domain publishing an MTA-STS policy in enforce mode instructs senders to refuse delivery when the certificate does not validate or the MX name is not in the policy. Large providers publish these policies. A name mismatch that was harmless for years becomes a bounce.
• DANE. Where DNSSEC is deployed, a TLSA record pins the expected certificate. Rotating a certificate without updating the TLSA record breaks mail from DANE-validating senders.
• TLS-RPT. Publishing _smtp._tls.yourdomain.com with a v=TLSRPTv1 record gets you reports from senders about failed TLS connections to your domain — the cheapest monitoring available for this class of problem.

What not to do

• Do not disable certificate verification to make an error go away. You have kept the error and discarded the protection.
• Do not let certificate renewal be the only unmonitored automation on a mail server. Alert on expiry from outside the host.
• Do not serve only the leaf certificate because your browser is happy with it. MTAs are stricter.
• Do not use a certificate that does not cover your SMTP banner hostname. Keep the banner, the MX target, the forward DNS and the certificate names identical.
• Do not run a mail server on a stale CA trust store. Outbound verification failures are often your fault, not the recipient's.

MailPilot connects mailboxes over OAuth or SMTP/IMAP and measures real per-provider inbox placement, so a transport problem that only shows up as deferrals at one provider surfaces as a placement difference rather than as a mystery.