Your signup test is red. The final line says the verification code was not found.
That does not tell you what failed.
Did the application skip the send? Did the message arrive late? Did another parallel test consume it? Did your helper select yesterday's code? Or did the email arrive correctly while a fixed sleep(10) simply wake up too soon?
Email-dependent tests often fail at this boundary. The application, mail system, inbox, parser, and test runner all move independently. A shared inbox and a timer turn those moving parts into a guess.
The reliable alternative is straightforward: give every test run its own inbox, wait on a real arrival condition, extract the code from the captured message, and retain enough evidence to explain a timeout.
This article shows that pattern with OTP verification, but the same loop applies to signup links, password resets, magic links, invitations, and other transactional email journeys.
Why the usual OTP test flakes
Most teams begin with one of four approaches. Each is useful in the right place, but each leaves a gap when the goal is to prove that a real message reached a real recipient.
A shared Gmail inbox
A shared inbox looks simple: send every test message to one account, add a tag to the address, then search for the latest matching subject.
It works until CI runs in parallel. Two jobs can search at the same time. One can read the other's code. A retry can find a stale message from the first attempt. Eventually the helper accumulates timestamps, subject filters, deletion rules, and special cases—but the underlying resource is still shared.
Isolation should be the starting point, not logic every test author has to rebuild.
A fixed sleep
Sleeping is not waiting on a condition. It is betting that the condition will become true within an arbitrary interval.
A short sleep flakes when delivery is slower than usual. A long sleep makes every successful run slower. Neither tells you whether the message never arrived or the test simply looked at the wrong time.
A homegrown polling loop
Polling is better than a fixed sleep, but it introduces another small system to maintain: retry intervals, timeouts, API quotas, MIME parsing, stale-message filters, cleanup, and diagnostic logging.
That can be a reasonable investment for a platform team. It is rarely the product work your signup feature was meant to deliver.
A mock or test-only OTP endpoint
Mocks are not wrong. A test-only endpoint is often the fastest way to verify that the application generated the correct token and state transition.
It does not prove the delivery boundary. It cannot catch a wrong recipient, duplicate send, corrupted link, broken template, missing message, or code that changed between generation and receipt.
The practical answer is usually a hybrid: keep most tests fast with mocks or internal hooks, then run a small critical suite against real received messages.
The five-step pattern for reliable email journeys
A deterministic OTP test has five parts.
1. Create one inbox for one run
Provision a fresh address for the test case. Do not pool inboxes across parallel jobs. Attach a time to live so the inbox removes itself even if the runner crashes before cleanup.
Use the CI run identity as an idempotency key. If the job retries the create request, it should receive the same result instead of creating another inbox and splitting the evidence.
2. Trigger the application
Submit the signup, reset, or login form with the generated address. This step should go through the same application path a user exercises.
Record the trigger time and the generated address in the test output. Those two facts make a later timeout much easier to investigate.
3. Wait on arrival, with a boundary
Ask for the first message after a known baseline and let the request block for a bounded period. This replaces a client-side polling loop and avoids waking early just to discover that nothing has arrived.
The boundary matters. A test must eventually return a clear timeout rather than wait forever. It should also avoid resetting the total wait budget every time a request is resumed.
4. Extract and use the received value
Read the parsed codes[] or links[] field from the received message. The test should not need to decode MIME, strip HTML, or invent a regular expression for every template.
Then use the code. A test that only asserts that an email exists proves receipt, not the journey. Submit the OTP to the application and assert the resulting state: verified account, authenticated session, completed reset, or accepted invitation.
5. Keep evidence and clean up
When the test fails, preserve the inbox ID, address, trigger time, match baseline, response status, received subject, and extracted values that are safe to log. That is the difference between “email not found” and an actionable failure.
Delete the inbox when the test completes. The TTL remains the safety net for interrupted runs.
A MailHulk example
MailHulk exposes this loop through /v1. The following shell example creates a 15-minute inbox, triggers a staging signup, waits up to 60 seconds for the first message, extracts the received OTP, and submits it back to the application.
The API wraps resources in data, so the example reads .data.address, .data.id, and .data.codes[0] explicitly.
set -euo pipefail
: "${MAILHULK_URL:?Set MAILHULK_URL}"
: "${MAILHULK_KEY:?Set MAILHULK_KEY}"
RUN_ID="${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-1}"
INBOX=$(curl --fail-with-body -sS \
-X POST "$MAILHULK_URL/v1/inboxes" \
-H "Authorization: Bearer $MAILHULK_KEY" \
-H "Idempotency-Key: $RUN_ID-signup" \
-H "Content-Type: application/json" \
-d '{"name":"signup-test","ttl_seconds":900}')
ADDRESS=$(echo "$INBOX" | jq -r '.data.address')
ID=$(echo "$INBOX" | jq -r '.data.id')
curl --fail-with-body -sS \
-X POST "https://staging.example.com/signup" \
-H "Content-Type: application/json" \
-d "$(jq -n --arg email "$ADDRESS" '{email: $email}')"
MESSAGE_FILE=$(mktemp)
trap 'rm -f "$MESSAGE_FILE"' EXIT
# A new inbox is empty, so the zero ULID is a safe first-message baseline.
STATUS=$(curl -sS -o "$MESSAGE_FILE" -w '%{http_code}' \
"$MAILHULK_URL/v1/inboxes/$ID/messages/latest?wait=60&after=00000000000000000000000000" \
-H "Authorization: Bearer $MAILHULK_KEY")
if [ "$STATUS" != "200" ]; then
echo "No OTP email arrived within 60 seconds (HTTP $STATUS)" >&2
exit 1
fi
CODE=$(jq -r '.data.codes[0] // empty' "$MESSAGE_FILE")
if [ -z "$CODE" ]; then
echo "The message arrived, but no OTP code was extracted" >&2
exit 1
fi
curl --fail-with-body -sS \
-X POST "https://staging.example.com/verify" \
-H "Content-Type: application/json" \
-d "$(jq -n --arg email "$ADDRESS" --arg code "$CODE" \
'{email: $email, code: $code}')"
The explicit after cursor closes a subtle race. If the OTP lands between the signup request and the wait request, the wait can still return it. For later messages in the same inbox, use the last observed message ID as the next baseline.
In a Playwright or Cypress suite, the orchestration is the same. Put inbox creation and cleanup in fixtures, call the bounded wait from the test, and keep the final OTP submission in the browser when you want to prove the complete user journey.
When MailHulk is not the right tool
Use the smallest test boundary that proves what you care about.
- Use a mock or test-only OTP endpoint when you only need to prove token generation or application state. It will be faster and has no external dependency.
- Use Mailpit or a similar local catcher when one developer runs tests serially on one machine. It is free, fast, and often exactly right for local development.
- Use a real isolated inbox when parallel jobs can collide, the test must cross the mail-delivery boundary, or a timeout needs evidence that the team can investigate.
You do not need to move the whole test suite. Start with the one journey whose failure would block a user: signup verification, password reset, or passwordless login. Keep the broader suite fast, and use the real-message test as a focused release signal.
OTP email testing checklist
Before calling an email journey deterministic, check the following:
- Every test run gets a fresh inbox.
- Inbox creation is idempotent across CI retries.
- The inbox expires automatically if cleanup never runs.
- The wait is tied to message arrival, not a fixed delay.
- The wait has an explicit baseline and a bounded timeout.
- Parallel jobs cannot read one another's messages.
- Codes and links are parsed before they reach test code.
- The received code or link is used to complete the application flow.
- A timeout records the address, trigger, baseline, and response status.
- The test deletes its inbox after completion.
Prove one journey first
Reliable OTP testing does not require a larger sleep or a more elaborate shared-inbox search. It requires isolation, a real wait condition, parsed results, and evidence when the journey stops.
MailHulk is being prepared for private beta as email journey testing infrastructure for developers and QA. If your team has a signup, password-reset, magic-link, or OTP test that still flakes in CI, bring that one journey. We want to help you make it deterministic—and learn where the workflow still needs more muscle.
