100 prospects, zero emails, all tests green: what our outage taught us

On October 7, a campaign of 100 prospects is launched on Phantera. Not a single email goes out. Our automated tests are all green. Here is what happened, with nothing dressed up.

First diagnosis, first fix

The launch button did all the sending during the click. Our host cuts a request off after 60 seconds: the campaign did not have time to start. We separated the two. Launching now queues the campaign in under a second, and sending happens in the background, one email at a time per mailbox.

Except that still, nothing went out.

The real cause

Phantera’s background jobs (sends, follow-ups, reading replies) run on a separate execution service. To work, that service must be able to reach every new version of our application once it is deployed. But our deployment URLs were protected. Since July 14, the sync had been failing silently: the functions added since then, including the one that sends queued emails, were simply not registered.

We found it on October 9, by testing in production with my own addresses.

Why no test saw it

Tests check the code. Here, the code was right: what production had actually loaded was not. The same day, this real test found six other defects the tests could not see, among them a campaign wizard stuck on one step and a sign-in that failed from a new device.

What we changed

• The sync is restored and runs again automatically after every deployment.
• After every new background job, we check the date of the last sync and the number of active functions.
• Every important flow is tested in production, on our own addresses, before it is announced.

On the afternoon of October 9, the same test showed five emails sent in three to four minutes, in line with the displayed estimate, one unsubscribe taken into account, and one reply detected, classified as positive, which stopped the sequence.

The rule that holds for any tool

1. Open the Sent folder of your mailbox: it is the only proof that an email went out.
2. Before a real campaign, send it to two or three addresses of your own.
3. Reply from an address other than the sending one, and check that the sequence stops.