Exactly Once Is a Business Requirement

Designing effectively-once outcomes on top of duplicate, delayed, and out-of-order delivery

Abstract

Billing, financial-data, and email providers deliver important state through webhooks. Those deliveries can be retried, duplicated, delayed, or received out of order. The business still expects one subscription update, one account-state transition, and one delivery-status outcome.

“Exactly once” is therefore best treated as a business invariant, not a transport promise.

This paper describes an effectively-once processing model developed while integrating Stripe, Quiltt, and Amazon SES notifications into one application. The model combines authenticated intake, durable event identity, state-aware claims, idempotent projections, ordering defenses, bounded retries, reconciliation, and operational evidence.

The most important lesson came from a subtle failure in first-pass deduplication: inserting an event ID before processing prevents duplicate side effects, but if every subsequent delivery is treated as complete, a crash after the insert can turn retry safety into permanent data loss.

Deduplication is a component. It is not the processing guarantee.

1. Why the transport cannot keep the business promise

Imagine a provider sends an event, the application commits the intended database change, and the HTTP response is lost.

The provider cannot know whether the event was applied. Retrying is rational.

Now imagine the application records the event ID, starts processing, and crashes before updating the domain state.

The application knows it has seen the event. It does not know that the business effect completed.

Those two failures produce opposite pressures:

• retrying can duplicate side effects;
• suppressing retries can lose side effects.

Stripe explicitly warns that webhook events may arrive out of order and that endpoints must not depend on generation order.¹ Amazon SNS tells HTTP consumers to expect retries and provides a message ID for identifying repeat delivery.²

The requirement “apply this once” remains valid. It simply cannot be delegated to the network.

2. Define what “once” means

There are at least four different “once” claims:

1. Delivered once — the transport sends one message.
2. Received once — the endpoint accepts one HTTP request.
3. Processed once — one worker executes the handler.
4. Effective once — the business state reflects the event one time.

The fourth is what users care about.

A duplicate invoice.paid request is harmless if the entitlement projection converges to the same active state. A single request is still harmful if it charges twice, sends two emails, or overwrites a newer subscription state with an older one.

I use effectively once to describe the target:

Any number of legitimate deliveries for the same provider fact produces one intended business outcome, and failures remain recoverable until that outcome is known.

That definition shifts the design from counting requests to controlling state.

3. Authenticate the bytes before interpreting the event

Webhook processing starts with authenticity, not deduplication.

Stripe and Quiltt signatures cover the raw request payload. Parsing and reserializing JSON can change whitespace or ordering, so the server captures the raw bytes before application middleware transforms the body.

The intake sequence is:

1. capture the raw request body;
2. read provider signature and timestamp headers;
3. verify against the configured secret or certificate;
4. enforce timestamp tolerance where the protocol supports it;
5. reject unverifiable requests;
6. only then parse provider semantics.

For key rotation, the Stripe verifier can accept a bounded set of active webhook secrets. For Quiltt, the verifier uses a timing-safe comparison and rejects stale timestamps.

Signature success does not authorize every payload to change every record. It only establishes that the event came through the expected provider protocol. Subject mapping and domain validation still follow.

4. Give the event a durable identity

Each provider supplies a stable event or message identifier:

• Stripe event ID;
• Quiltt event ID;
• SNS message ID for SES notifications.

The database stores that ID under a unique constraint along with:

• provider and event type;
• receive time;
• processing status;
• processing completion time;
• relevant provider subject identifiers;
• bounded error code and message;
• payload or a privacy-appropriate representation.

The unique insert is the concurrency primitive. Two workers receiving the same event cannot both create the first claim.

Where a provider does not guarantee a stable event ID, the application needs a documented composite identity or idempotency key. A payload hash alone is often insufficient: two legitimate events can have identical bodies, while the same logical event may include non-semantic metadata differences.

The identity must match the provider’s retry semantics.

5. Model processing as a state machine

A dedupe table with event_id is not enough. It needs processing semantics.

The target model is:

Event-processing state: A received event is claimed with a lease and moves to processing. Processing can complete, fail into a retryable error, or enter dead letter after attempts are exhausted. Processed duplicates are acknowledged without reapplication; dead-lettered events require explicit repair before controlled replay.

Important fields include:

• status;
• attempt_count;
• lease_owner;
• lease_expires_at;
• next_attempt_at;
• processed_at;
• error_code;
• provider_subject_id;
• payload_hash.

The lease distinguishes a worker that is still processing from one that crashed. An expired lease can be reclaimed. A completed event returns success without reapplying the domain effect. An error can retry according to policy.

This is the difference between:

I have seen this event.

and:

I know the intended outcome completed.

6. The failure hidden inside simple deduplication

The first-pass pattern in this case study looked reasonable:

1. insert the provider event ID;
2. if the insert conflicts, return “duplicate”;
3. process the event;
4. mark it processed or error.

It prevents concurrent duplicate application.

It also has a dangerous gap. If processing fails after step 1, the row remains. If every later delivery is classified as a terminal duplicate, the provider’s retry can never repair the business state.

The dedupe succeeded. The business outcome failed.

Different adapters in the application evolved differently. The SES path checks whether the existing event actually reached processed and permits another attempt after an error. Earlier webhook paths demonstrate event identity and duplicate suppression but do not yet implement the complete lease-based, state-aware retry model described here.

I would not publish a claim of restartable exactly-once processing for those paths until that gap is closed and tested. The case study is valuable precisely because it shows why a unique event ID is necessary but insufficient.

7. Make the domain projection idempotent

Even a good event claim cannot protect every side effect.

Suppose the worker commits the subscription update but crashes before marking the event processed. A retry should be able to repeat the domain operation safely.

That favors convergent operations:

• upsert subscription by provider subscription ID;
• set entitlement to the state derived from provider truth;
• upsert payment method by provider payment-method ID;
• update connection health using a provider connection ID;
• update email delivery status using the provider message ID.

The intended effect of repeating the operation should match applying it once. This mirrors the HTTP definition of idempotence: multiple identical requests have the same intended server effect as one.³

For non-idempotent external calls, use a stable outbound idempotency key. Customer creation, charge creation, and transfer initiation should not generate a new key for each retry attempt.

If the database update and outbound call cannot share a transaction, the design needs a durable command or outbox record so completion can be reconciled rather than guessed.

8. Do not let arrival order become business order

Deduplication answers whether an event was seen. It does not answer whether the event is newer than the state already applied.

Out-of-order examples include:

• a subscription update arriving after a deletion;
• a payment-processing event arriving after payment success;
• an older connection-health event overwriting a newer reconnect result;
• an email-delivery event arriving after a bounce or complaint.

Defenses depend on the provider:

• compare provider event creation time;
• compare object version or sequence number;
• define a monotonic status precedence where the domain permits it;
• retrieve the latest provider object when an event lacks enough context;
• store the last applied provider event ID and timestamp;
• reject state regression unless an explicit transition allows it.

Arrival time is useful operational metadata. It is rarely a trustworthy business clock.

9. Separate receipt from heavy processing

Webhook providers impose response deadlines and retry on server errors or timeouts. The endpoint should therefore do the minimum synchronous work required to make receipt durable.

A mature flow is:

Durable webhook sequence: The provider sends a signed event. Intake verifies the signature and creates or inspects the event claim. A worker leases retryable work, applies an idempotent domain projection, and marks the event processed.

The current application processes much of the webhook work inline, with some background synchronization. That is appropriate at modest volume but creates a clear extraction point if latency or retry complexity grows: keep authenticated intake in the app, move durable processing behind a queue or worker, and preserve the same event ledger.

The queue is not the guarantee. The state machine and idempotent projection are.

10. Return status codes according to retry intent

HTTP status is part of the processing protocol.

Broadly:

• invalid signature: permanent client error;
• malformed unsupported request: permanent client error;
• already processed duplicate: success;
• accepted for asynchronous work: success;
• temporary database or provider failure: retryable server error;
• known permanent domain rejection: provider-specific success or client error, depending on whether retry would help.

Amazon SNS, for example, treats server errors and throttling as retryable under its delivery policy and can route exhausted deliveries to a dead-letter queue.⁴

Returning 200 to stop noisy retries can hide lost work. Returning 500 for a permanent malformed event can create a retry storm. The response should express whether another delivery can improve the outcome.

11. Reconciliation closes the last gap

No webhook design eliminates every uncertainty.

Providers can have outages. Secrets can rotate incorrectly. A handler can contain a bug. An operator can repair local data. A delivery can exhaust its retry policy.

Reconciliation compares local projections with provider truth:

• active subscriptions and entitlements;
• provider customer mapping;
• connection and account status;
• outbound email delivery state;
• unprocessed or errored event counts;
• stale processing leases;
• events with missing subject mappings.

The reconciliation process should be observable, bounded, and safe to rerun. It is not an admission that webhooks failed. It is how the system proves convergence after failures that webhooks alone cannot resolve.

12. Test the failure matrix

A useful webhook suite covers:

Failure	Expected behavior
Invalid signature	Rejected before event claim
Same event delivered twice concurrently	One claim; one business effect
Duplicate after successful processing	Acknowledged without reapplication
Failure after event claim	Event remains retryable
Crash after domain commit, before completion marker	Retry converges safely
Older event arrives after newer state	No state regression
Unknown provider subject	Recorded and bounded; no unrelated mutation
Secret rotation overlap	Accepted by one active secret
Permanent malformed payload	No endless retry
Attempts exhausted	Visible dead-letter or operator queue
Manual replay after repair	Same event identity; controlled new attempt

Testing only “send the same payload twice” proves duplicate suppression. It does not prove recovery.

13. Operational evidence

The event ledger should answer:

• How many events are received, processed, retrying, or dead-lettered?
• Which event types fail most often?
• How long does processing take?
• Which provider subjects have repeated failures?
• Are leases stuck?
• Are duplicate rates changing?
• Did reconciliation repair drift?

Logs should include event ID, type, duration, outcome, and request correlation without dumping secrets or unnecessarily retaining financial payloads.

An internal operations view can aggregate error counts and expose a bounded failure summary. That is more useful than asking an operator to search raw logs for an event ID they may not know.

14. Practical design rules

1. Treat exactly once as a business outcome, not a transport assumption.
2. Verify raw request bytes before interpreting payloads.
3. Use provider-stable event identity under a database uniqueness constraint.
4. Distinguish received, processing, processed, error, and dead-letter states.
5. Use leases so crashes do not create permanent in-flight events.
6. Make domain projections convergent and idempotent.
7. Use stable idempotency keys for outbound non-idempotent operations.
8. Defend against out-of-order delivery independently from duplicates.
9. Return HTTP status according to whether retry can help.
10. Reconcile local state with provider truth.
11. Test the gap between claim and completion.
12. Do not call duplicate suppression “exactly-once processing.”

Conclusion

Exactly once is a reasonable business demand:

• grant the entitlement once;
• apply the account update once;
• record the email outcome once;
• initiate the external action once.

The mistake is expecting the delivery channel to provide that invariant by itself.

Effectively-once behavior comes from combining identity, durable state, idempotent effects, ordering rules, retry semantics, and reconciliation. Remove any one of those and the system can still produce duplicates or lose outcomes while every individual component appears to be working.

The event ID tells us we have seen a message.

The architecture must prove that the business effect is complete.

References

Footnotes

1. Stripe Documentation, “Receive Stripe events in your webhook endpoint”. ↩

2. Amazon Web Services, “Make sure your endpoint is ready to process Amazon SNS messages”. ↩

3. Internet Engineering Task Force, RFC 9110, “HTTP Semantics,” Section 9.2.2. ↩

4. Amazon Web Services, “Amazon SNS message delivery retries”. ↩