Stage 7 · Master
Event-Driven Architecture with Kafka
Decoupling Services with Events
Kafka is the planned boundary for cross-service side effects so task writes do not turn into orchestration spiders.
Why Direct Service Calls Did Not Scale the Design
This module is deliberately forward-looking. Fieldwork's Kafka layer is planned architecture, not something we are pretending is already humming away in production. The reason it is planned at all is simple: once the task-tracking API grows beyond one deployable, direct service-to-service calls turn ordinary writes into orchestration. A task status change should update the task row. It should not synchronously call notifications, activity feed, analytics, search indexing, and audit export services just to consider itself complete. That shape couples the caller to every current side effect and every future one.
We rejected an HTTP fan-out model because its failure mode is the write path now owns everybody else's availability. If the notifications service is slow, the task update endpoint becomes slow. If analytics is down, do we fail the task change or silently skip metrics? Neither answer is appealing inside a user-facing request path. Events give us a cleaner statement: the task service commits the state change and publishes the fact that it happened. Consumers decide what to do with that fact on their own clocks.
| Cross-service pattern | Why it looked workable | Why Fieldwork rejected or chose it |
|---|---|---|
| Inline HTTP calls from task service | Simple to understand with two services | Write latency and failure now depend on every downstream consumer |
| Shared database reads by many services | No extra event plumbing | Leaky ownership and accidental coupling to table internals |
| Kafka domain events | Needs schemas and consumer discipline | Chosen because it decouples side effects from the originating request |
They are not a substitute for command APIs inside a single synchronous workflow. The gain comes when a state change has independent downstream consequences.
The Event Boundaries Fieldwork Plans
The planned event surface is grounded in the schema the earlier modules already established: tenants, workspaces, projects, tasks, memberships, and users. We are not publishing raw table-change firehoses. Events represent domain facts that other services care about. The most obvious early topics are fieldwork.tasks, fieldwork.memberships, and fieldwork.attachments. A task event might announce creation, assignment, status transition, or archival. A membership event matters because access-related services and audit consumers need to know role changes. Attachment events matter once malware scanning, thumbnail generation, or activity feeds move out of the main API.
- Publish domain events, not every column mutation.
- Include tenant and aggregate identifiers in every payload.
- Keep event names aligned with business facts like
task.status_changed, not transport details.
Why Kafka, Not Ad-Hoc HTTP Fan-Out
Kafka was chosen for the planned event bus because the intended problems are independent consumption, replay, consumer groups, and durable decoupling between services. A direct HTTP callback system can mimic publish/subscribe at small scale, but it makes the producer responsible for subscriber discovery, retry policy, and delivery fan-out. That is exactly the coupling we were trying to remove. Kafka also gives us a path to add consumers later — analytics, notification digests, search indexing, billing if the product ever grows there — without teaching the task service about each one.
| Concern | HTTP fan-out | Kafka |
|---|---|---|
| Producer awareness of consumers | High | Low |
| Replay historical events | Custom build required | Built into log retention model |
| Independent consumer pace | Awkward | Natural with consumer groups and offsets |
| Write-path coupling | Strong | Weak once publish succeeds |
The current side effects are not the only reason to choose an event bus. The more important benefit is that future consumers do not require task-service rewrites.
Where Events Should Not Be Used
Events are not a replacement for synchronous APIs everywhere. If a request needs an immediate answer from a service it directly depends on — authenticating a token, checking current membership, reading the current task state for the response itself — an event bus is the wrong tool. We also do not plan to use Kafka as a dumping ground for internal implementation chatter. If no independent consumer cares that a cache entry was refreshed or a SQL query ran, it is not a domain event. We rejected the idea that event-driven means everything becomes asynchronous because that only relocates confusion.
The Decision
Fieldwork's planned cross-service architecture uses Kafka for domain events emitted from state changes in the task, membership, and attachment areas. We rejected direct HTTP fan-out because it would force the originating service to own downstream availability and side-effect sprawl. The decision is not that events are modern. The decision is that cross-service side effects should be decoupled from request latency, consumer count, and future feature growth. Kafka is the mechanism we chose because those are the specific problems we need the design to solve.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.