Stage 7 · Master
Phase 9 — Payment Service
Idempotency
Make asynchronous webhooks safe by verifying raw signed payloads and deduplicating retries before any state change occurs.
Webhook Idempotency Solves a Different Problem Than Client Request Replay
Lesson 1 protected the synchronous initiation endpoint from duplicate button presses. Webhook idempotency addresses something else entirely: providers will retry the same event delivery when your endpoint times out, returns a 5xx, or briefly disappears during deploy. The business danger is similar — you do not want to capture or refund the same payment multiple times locally — but the mechanism is different because the gateway, not the resident, is the one replaying the message.
Client idempotency is keyed by your API's Idempotency-Key header. Webhook idempotency is keyed by the provider's event identity plus its signed raw payload. Treating them as the same problem usually leads to the wrong key, the wrong transaction boundary, or both.
Signature Verification Must Run on the Raw Bytes, Not Re-encoded JSON
HMAC verification is byte-for-byte. If you call Gin's JSON binding first, consume the body, and then rebuild the JSON for verification, harmless differences such as field ordering, spacing, or numeric formatting can change the bytes and make a valid provider signature look invalid. Read the raw body once, verify the signature against those exact bytes, then parse the payload. Pair that with a timestamp-tolerance check so an attacker cannot replay a previously captured signed webhook days later.
A forged payload can claim any tenant. Verify the signature, parse the event, look up the local payment by provider_reference, and then take org_id from your own payment row. Tenant identity flows from your database, not from the webhook JSON.
An Inbox Table Turns Provider Retries Into a Single Accepted Event
The inbox pattern gives you a durable deduplication boundary. Insert the signed raw payload into payment_webhook_inbox using the provider's event ID under a UNIQUE (provider, provider_event_id) constraint. If the insert affected one row, this is the first time you have seen the event and you may continue. If it affected zero rows, the provider retried something you already accepted and you can acknowledge it without reapplying business logic.
CREATE TABLE payment_webhook_inbox (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
org_id UUID NOT NULL,
payment_id UUID NOT NULL REFERENCES payments(id) ON DELETE CASCADE,
provider TEXT NOT NULL,
provider_event_id TEXT NOT NULL,
event_type TEXT NOT NULL,
payload BYTEA NOT NULL,
signature_header TEXT NOT NULL,
received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
processed_at TIMESTAMPTZ,
processing_error TEXT,
UNIQUE (provider, provider_event_id)
);
CREATE INDEX payment_webhook_inbox_org_received_idx
ON payment_webhook_inbox (org_id, received_at DESC);
CREATE INDEX payment_webhook_inbox_unprocessed_idx
ON payment_webhook_inbox (processed_at)
WHERE processed_at IS NULL;The inbox row stores the exact bytes you verified. That gives you an audit trail for finance and a replay surface for debugging missed state changes later.
DROP TABLE IF EXISTS payment_webhook_inbox;package payment
import (
"context"
"errors"
"time"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
)
type InboxStore struct{}
type ClaimResult struct {
InboxID uuid.UUID
Inserted bool
}
func NewInboxStore() *InboxStore {
return &InboxStore{}
}
func (s *InboxStore) Claim(
ctx context.Context,
tx pgx.Tx,
orgID uuid.UUID,
paymentID uuid.UUID,
event Event,
payload []byte,
signatureHeader string,
) (ClaimResult, error) {
var inboxID uuid.UUID
err := tx.QueryRow(ctx,
`INSERT INTO payment_webhook_inbox (
org_id, payment_id, provider, provider_event_id,
event_type, payload, signature_header
) VALUES ($1, $2, $3, $4, $5, $6, $7)
ON CONFLICT (provider, provider_event_id) DO NOTHING
RETURNING id`,
orgID,
paymentID,
event.Provider,
event.EventID,
string(event.EventType),
payload,
signatureHeader,
).Scan(&inboxID)
if errors.Is(err, pgx.ErrNoRows) {
return ClaimResult{Inserted: false}, nil
}
if err != nil {
return ClaimResult{}, err
}
return ClaimResult{InboxID: inboxID, Inserted: true}, nil
}
func (s *InboxStore) MarkProcessed(ctx context.Context, tx pgx.Tx, inboxID uuid.UUID, processedAt time.Time) error {
_, err := tx.Exec(ctx,
`UPDATE payment_webhook_inbox
SET processed_at = $2,
processing_error = NULL
WHERE id = $1`,
inboxID,
processedAt,
)
return err
}
func (s *InboxStore) MarkProcessingError(ctx context.Context, tx pgx.Tx, inboxID uuid.UUID, message string) error {
_, err := tx.Exec(ctx,
`UPDATE payment_webhook_inbox
SET processing_error = $2
WHERE id = $1`,
inboxID,
message,
)
return err
}INSERT ... ON CONFLICT DO NOTHING is the deduplication gate. Only the first accepted event body gets an inbox row and permission to continue to business logic.
Process the Event and Acknowledge the Inbox in One Database Transaction
The inbox does not help if you insert a dedupe row, update the payment, crash before marking the inbox processed, and then cannot tell whether the event actually changed business state. Wrap claim, payment update, and inbox acknowledgement in the same transaction. That makes the failure mode binary: either nothing committed and the provider retry can try again, or both the payment row and the inbox row committed together.
This handler must not decide whether captured may follow refunded — that rule belongs to the payment status model, and the next lesson writes it. So the handler declares the narrow interface it consumes, paymentTransitioner, exactly the way Phase 2 declared its repository interface next to the service that used it. Until the next lesson lands, a three-line stub that always returns true keeps this file compiling.
package payment
import (
"context"
"errors"
"io"
"net/http"
"time"
"github.com/gin-gonic/gin"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
// paymentTransitioner is declared here, in the package that consumes it, so
// the webhook handler depends on the rule it needs rather than on whichever
// type happens to implement it. The next lesson supplies the real
// implementation; a stub returning (true, nil) satisfies it in the meantime.
type paymentTransitioner interface {
Apply(ctx context.Context, tx pgx.Tx, orgID, paymentID uuid.UUID, next Status, eventID string) (bool, error)
}
type WebhookHandler struct {
pool *pgxpool.Pool
provider Provider
inbox *InboxStore
transitions paymentTransitioner
clock func() time.Time
maxSkew time.Duration
}
type paymentRef struct {
ID uuid.UUID
OrgID uuid.UUID
}
func NewWebhookHandler(
pool *pgxpool.Pool,
provider Provider,
inbox *InboxStore,
transitions paymentTransitioner,
clock func() time.Time,
) *WebhookHandler {
if clock == nil {
clock = time.Now
}
return &WebhookHandler{
pool: pool,
provider: provider,
inbox: inbox,
transitions: transitions,
clock: clock,
maxSkew: 5 * time.Minute,
}
}
func (h *WebhookHandler) Handle(c *gin.Context) {
rawBody, err := io.ReadAll(c.Request.Body)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "unable to read webhook body"})
return
}
signature := c.GetHeader("X-Payment-Signature")
timestampHeader := c.GetHeader("X-Payment-Timestamp")
sentAt, err := time.Parse(time.RFC3339, timestampHeader)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid webhook timestamp"})
return
}
if absDuration(h.clock().UTC().Sub(sentAt.UTC())) > h.maxSkew {
c.JSON(http.StatusBadRequest, gin.H{"error": "stale webhook timestamp"})
return
}
if !h.provider.VerifyWebhookSignature(rawBody, signature) {
c.JSON(http.StatusUnauthorized, gin.H{"error": "invalid webhook signature"})
return
}
event, err := h.provider.ParseEvent(rawBody)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid webhook payload"})
return
}
event.SignatureHeader = signature
payment, err := h.lookupPaymentByProviderReference(c.Request.Context(), event.Provider, event.PaymentReference)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
c.JSON(http.StatusNotFound, gin.H{"error": "payment not found"})
return
}
c.JSON(http.StatusInternalServerError, gin.H{"error": "payment lookup failed"})
return
}
tx, err := h.pool.BeginTx(c.Request.Context(), pgx.TxOptions{})
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "transaction start failed"})
return
}
defer tx.Rollback(c.Request.Context())
claim, err := h.inbox.Claim(c.Request.Context(), tx, payment.OrgID, payment.ID, event, rawBody, signature)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "inbox write failed"})
return
}
if !claim.Inserted {
if err := tx.Commit(c.Request.Context()); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "duplicate acknowledgement failed"})
return
}
c.JSON(http.StatusOK, gin.H{"status": "duplicate_ignored"})
return
}
nextStatus, err := transitionStatusForEvent(event)
if err != nil {
_ = h.inbox.MarkProcessingError(c.Request.Context(), tx, claim.InboxID, err.Error())
c.JSON(http.StatusBadRequest, gin.H{"error": "unsupported event type"})
return
}
accepted, err := h.transitions.Apply(c.Request.Context(), tx, payment.OrgID, payment.ID, nextStatus, event.EventID)
if err != nil {
_ = h.inbox.MarkProcessingError(c.Request.Context(), tx, claim.InboxID, err.Error())
c.JSON(http.StatusInternalServerError, gin.H{"error": "state transition failed"})
return
}
if err := h.inbox.MarkProcessed(c.Request.Context(), tx, claim.InboxID, h.clock().UTC()); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "inbox acknowledgement failed"})
return
}
if err := tx.Commit(c.Request.Context()); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "webhook commit failed"})
return
}
if !accepted {
c.JSON(http.StatusOK, gin.H{"status": "stale_transition_dropped"})
return
}
c.JSON(http.StatusOK, gin.H{"status": "processed"})
}
func (h *WebhookHandler) lookupPaymentByProviderReference(ctx context.Context, provider, reference string) (paymentRef, error) {
var ref paymentRef
err := h.pool.QueryRow(ctx,
`SELECT id, org_id
FROM payments
WHERE provider = $1 AND provider_reference = $2`,
provider,
reference,
).Scan(&ref.ID, &ref.OrgID)
return ref, err
}
func transitionStatusForEvent(event Event) (Status, error) {
switch event.EventType {
case EventInitiated:
return PaymentStatusInitiated, nil
case EventAuthorized:
return PaymentStatusAuthorized, nil
case EventCaptured:
return PaymentStatusCaptured, nil
case EventFailed:
return PaymentStatusFailed, nil
case EventRefunded:
return PaymentStatusRefunded, nil
default:
return "", errors.New("unknown event type")
}
}
func absDuration(v time.Duration) time.Duration {
if v < 0 {
return -v
}
return v
}The body is read exactly once and verified before parsing. The inbox row and the payment state change commit together, so a crash cannot leave you wondering whether the webhook was deduped but not applied.
Resend the Exact Signed Payload and Expect Only One Durable State Change
A good local proof uses the exact same bytes three times, not three logically equivalent JSON objects. When the first request succeeds, later retries should be acknowledged as duplicates because the UNIQUE (provider, provider_event_id) constraint is doing the heavy lifting. If you instead see multiple updates to the same payment row, your inbox claim is happening too late or outside the transaction boundary.
- Failure mode: calling JSON binding before signature verification can consume or normalize the body and make a valid signature fail.
- Security concern: reject webhooks whose timestamp falls outside your tolerance window, or an attacker can replay an old signed payload.
- Verification target: identical signed payloads retried three times should produce one inbox insert and one durable payment transition.
Applied exercise
Prove the inbox swallows provider retries
Your provider sandbox retries every 10 seconds whenever it sees a 500, and a teammate wants proof that replaying the same captured event will never capture the local payment twice.
- Seed one payment row with a provider_reference that matches a sandbox event.
- Post the same raw webhook body three times with the same provider_event_id and valid signature headers.
- Query payment_webhook_inbox after each delivery and confirm only one row exists for that provider/event pair.
- Record the payment status before and after the three deliveries to show only one state transition occurred.
Deliverable
A short verification log containing the curl loop, the inbox row count query, and the final payment status query.
Completion checks
- Only one inbox row exists for the repeated provider_event_id.
- The payment status changes at most once across the three requests.
- Your explanation calls out why the insert and status update must share one transaction.
Why must webhook signature verification use the exact raw request bytes instead of JSON that has been re-marshaled by the server?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.