Stage 7 · Master
Phase 16 — Kafka Integration
Producers
Build a transactional outbox so a captured payment and its Kafka event either both survive or neither does — then a separate relay is the only thing ever allowed to touch the broker.
Publishing After Commit Can Silently Lose the Event Forever
The obvious way to wire up Lesson 1's decision is to call a Kafka producer right after committing the payment's captured transition. That order has a real gap: if the process crashes between the commit and the publish call, the payment is captured, permanently, and the event is gone, permanently, with nothing in the database ever recording that it should have been sent. Publishing first and committing second only moves the same gap to the other side — a publish can succeed and the transition commit can then fail, leaving an event on the topic for a payment that, as far as the database is concerned, was never captured.
payment-service already solved the mirror-image version of this problem for inbound webhooks: payment_webhook_inbox's claim happens inside the same transaction as the state transition it authorizes (the Payment module's InboxStore). The outbound side needs the same discipline in reverse — the record of what to publish must commit in the same transaction as the state change that produced it, and only a separate process, running after that transaction has already committed, is ever allowed to touch Kafka.
payment_outbox Is Written In the Same Transaction as the Transition
CREATE TABLE payment_outbox (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
org_id UUID NOT NULL,
aggregate_id UUID NOT NULL REFERENCES payments(id) ON DELETE CASCADE,
topic TEXT NOT NULL,
event_type TEXT NOT NULL,
event_version SMALLINT NOT NULL DEFAULT 1,
partition_key TEXT NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
published_at TIMESTAMPTZ
);
CREATE INDEX payment_outbox_unpublished_idx
ON payment_outbox (created_at)
WHERE published_at IS NULL;partition_key is stored at write time rather than recomputed later, so the relay worker never needs to know payment-specific derivation rules — it forwards whatever the transition handler already decided. Note which reference survives and which does not: aggregate_id keeps its foreign key to payments because both tables sit in payment-service's own database, while org_id is a bare UUID copied from the row being published. An outbox row must remain publishable even if the owning organization is later archived somewhere else entirely, so a constraint pointing outside this database would be both impossible to create and wrong to want.
DROP TABLE IF EXISTS payment_outbox;One New Call, Inside the Same pgx.Tx the Transition Already Holds
package payment
import (
"context"
"encoding/json"
"time"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
)
type OutboxWriter struct{}
func NewOutboxWriter() *OutboxWriter {
return &OutboxWriter{}
}
type CompletedPayload struct {
AmountPaise int64 `json:"amount_paise"`
Currency string `json:"currency"`
ProviderReference string `json:"provider_reference"`
}
type envelope struct {
EventID uuid.UUID `json:"event_id"`
EventType string `json:"event_type"`
EventVersion int `json:"event_version"`
OrgID uuid.UUID `json:"org_id"`
AggregateID uuid.UUID `json:"aggregate_id"`
OccurredAt time.Time `json:"occurred_at"`
Payload CompletedPayload `json:"payload"`
}
// WritePaymentCompleted must be called with the same *pgx.Tx the payment's
// captured transition is committing through. This line only ever runs if
// that transition also commits — there is no code path where one succeeds
// without the other.
func (w *OutboxWriter) WritePaymentCompleted(ctx context.Context, tx pgx.Tx, orgID, paymentID uuid.UUID, payload CompletedPayload) error {
body, err := json.Marshal(envelope{
EventID: uuid.New(),
EventType: "payment.completed",
EventVersion: 1,
OrgID: orgID,
AggregateID: paymentID,
OccurredAt: time.Now().UTC(),
Payload: payload,
})
if err != nil {
return err
}
_, err = tx.Exec(ctx, `
INSERT INTO payment_outbox (org_id, aggregate_id, topic, event_type, event_version, partition_key, payload)
VALUES ($1, $2, $3, $4, $5, $6, $7)`,
orgID, paymentID, "payment.completed", "payment.completed", 1, paymentID.String(), body,
)
return err
}
partition_key is paymentID.String() — matching Lesson 1's ordering decision — computed once here rather than re-derived by whatever eventually reads this row.
if err := tx.updatePaymentStatus(ctx, tx.pgx, paymentID, StatusCaptured); err != nil {
return err
}
if err := outbox.WritePaymentCompleted(ctx, tx.pgx, orgID, paymentID, CompletedPayload{
AmountPaise: amountPaise,
Currency: currency,
ProviderReference: providerReference,
}); err != nil {
return err
}
return tx.pgx.Commit(ctx)
Everything above the updatePaymentStatus call — validating the transition is legal, acquiring the row lock — is unchanged from the Payment module. WritePaymentCompleted only ever runs between that update and the final commit, inside the same transaction as both.
A Separate Process Is the Only Thing Allowed to Touch Kafka
package payment
import (
"context"
"time"
kafkago "github.com/segmentio/kafka-go"
"github.com/jackc/pgx/v5/pgxpool"
)
type Relay struct {
pool *pgxpool.Pool
writer *kafkago.Writer
}
func NewRelay(pool *pgxpool.Pool, brokers []string) *Relay {
return &Relay{
pool: pool,
writer: &kafkago.Writer{
Addr: kafkago.TCP(brokers...),
Topic: "payment.completed",
Balancer: &kafkago.Hash{}, // partition_key drives placement, never round robin
RequiredAcks: kafkago.RequireAll,
},
}
}
type outboxRow struct {
id string
partitionKey string
payload []byte
}
// RelayOnce claims up to batchSize unpublished rows, writes them to Kafka,
// and only then marks them published — all inside one transaction, so a
// crash at any point leaves every claimed row either fully published or
// still unpublished, never half-done.
func (r *Relay) RelayOnce(ctx context.Context, batchSize int) (int, error) {
tx, err := r.pool.Begin(ctx)
if err != nil {
return 0, err
}
defer tx.Rollback(ctx)
rows, err := tx.Query(ctx, `
SELECT id, partition_key, payload
FROM payment_outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT $1
FOR UPDATE SKIP LOCKED`, batchSize)
if err != nil {
return 0, err
}
var claimed []outboxRow
for rows.Next() {
var o outboxRow
if err := rows.Scan(&o.id, &o.partitionKey, &o.payload); err != nil {
rows.Close()
return 0, err
}
claimed = append(claimed, o)
}
rows.Close()
if len(claimed) == 0 {
return 0, nil
}
messages := make([]kafkago.Message, len(claimed))
for i, o := range claimed {
messages[i] = kafkago.Message{Key: []byte(o.partitionKey), Value: o.payload}
}
if err := r.writer.WriteMessages(ctx, messages...); err != nil {
return 0, err // the transaction rolls back; every claimed row stays unpublished for the next call
}
ids := make([]string, len(claimed))
for i, o := range claimed {
ids[i] = o.id
}
if _, err := tx.Exec(ctx, `UPDATE payment_outbox SET published_at = $2 WHERE id = ANY($1)`, ids, time.Now().UTC()); err != nil {
return 0, err
}
return len(claimed), tx.Commit(ctx)
}
RequiredAcks: kafkago.RequireAll blocks WriteMessages until the broker itself has acknowledged every message, so the code only reaches the UPDATE that marks rows published after Kafka has actually accepted them — a broker outage leaves rows unpublished and eligible for the next RelayOnce call, never silently dropped.
If the process dies after Kafka accepts the batch but before tx.Commit runs, the whole transaction — including the SELECT ... FOR UPDATE claim — rolls back with it. The next RelayOnce call reclaims and republishes the same rows, producing a duplicate delivery on the topic. That duplicate is exactly where this module's at-least-once guarantee actually comes from on the producer side, and it is why Lesson 3's consumer needs its own idempotency guard rather than trusting the producer to send each event exactly once.
Prove One Relay Publishes Correctly, and Two Don't Duplicate
Applied exercise
Prove FOR UPDATE SKIP LOCKED is what makes two relay replicas safe together
Ops wants a second relay replica for throughput. Prove that doesn't double-publish before recommending it.
- Start a second
go run ./services/payment-service/cmd/relay &alongside the first, both pointed at the same database and broker. - Insert 20 rows directly into payment_outbox with distinct aggregate_ids (bypassing the transition code, for this exercise only).
- Use kafka-console-consumer.sh with --max-messages 20 to confirm exactly 20 messages arrive on the topic, not 40.
- Write one paragraph naming the specific line in RelayOnce responsible for that result — not a vague 'eventual consistency' explanation.
Deliverable
A short paragraph naming the exact mechanism, backed by the observed message count.
Completion checks
- The observed count is exactly 20, never double, across both replicas combined.
- The explanation names FOR UPDATE SKIP LOCKED specifically as the reason a second transaction cannot claim an already-locked row.
The relay process crashes after writer.WriteMessages returns successfully but before tx.Exec's UPDATE commits. What happens to that batch of events?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.