Kafka
A distributed, replayable log that decouples producers from consumers at massive scale.
Introdução
Apache Kafka is a distributed event streaming platform built around an append-only, partitioned, replicated log. Producers append events to topics; consumers read them at their own pace. Unlike a traditional queue, Kafka retains events and lets multiple independent consumers replay the same stream, which makes it the backbone of event-driven architectures.
Por que existe
Systems need to move high volumes of events between many producers and many consumers without coupling them together, losing data, or forcing everyone to process at the same speed. Point-to-point integrations become an unmaintainable mesh. Kafka exists to be a durable, high-throughput central nervous system: producers publish once, any number of consumers subscribe, and events are retained so consumers can fail, catch up, or replay history.
Analogia
Kafka is like a newspaper archive rather than a mailbox. A mailbox (traditional queue) delivers a letter once and it's gone. The newspaper is printed to an ordered archive; many readers read at their own pace, new subscribers can start from any past date, and re-reading yesterday's paper is trivial. The paper isn't removed when you read it.
Como funciona
A topic is split into partitions, each an ordered, append-only log. Every record in a partition has a monotonically increasing offset. Ordering is guaranteed only within a partition — records are routed to a partition by a key (same key → same partition → ordered).
Partitions are replicated across brokers for durability; one replica is the leader, others followers. Producers choose a durability level via acks (0/1/all). Consumers track their position by committing offsets; because Kafka retains records, a consumer can replay from an earlier offset. Consumers form consumer groups: each partition is assigned to exactly one consumer in a group, so parallelism is capped by the partition count, and a rebalance reassigns partitions when consumers join or leave.
Delivery is at-least-once by default (commit offsets after processing), so consumers must be idempotent. Kafka scales horizontally by adding partitions and brokers, and retains data by time or size regardless of consumption.
- 1
A producer serializes an event and picks a partition via the record key (hash) or round-robin.
- 2
The partition leader appends the record and replicates it to followers per the acks setting.
- 3
The record gets the next offset and is retained for the configured time/size.
- 4
A consumer in a group polls its assigned partitions and processes records in offset order.
- 5
After processing, the consumer commits its offset; on restart it resumes from the last commit (replayable).
Interativo
Arquitetura
Topic → partitions → consumer group
Topic 'orders' (ordered per partition)
Consumer group (1 partition → 1 consumer)
Records with the same key land on the same partition and stay ordered; each partition feeds exactly one consumer in the group.
No código
import { Kafka } from "kafkajs"
const kafka = new Kafka({ brokers: ["broker:9092"] })
const producer = kafka.producer()
export async function emitOrderEvent(orderId: string, event: OrderEvent) {
await producer.send({
topic: "orders",
// key = orderId → all events for one order go to the same partition → ordered.
messages: [{ key: orderId, value: JSON.stringify(event) }],
acks: -1, // 'all': wait for in-sync replicas → strongest durability
})
}await consumer.subscribe({ topic: "orders", fromBeginning: false })
await consumer.run({
eachMessage: async ({ message }) => {
const evt = JSON.parse(message.value!.toString())
// Redelivery is normal after a crash → dedupe on a stable id.
const isNew = await markProcessed(evt.eventId) // INSERT ... ON CONFLICT DO NOTHING
if (!isNew) return // duplicate → skip
await applyOrderEvent(evt)
// Offset is committed after processing → at-least-once semantics.
},
})No mundo real
Onde você já viu isso
A commerce platform publishes every domain event (OrderPlaced, PaymentSucceeded, InventoryReserved) to Kafka. Independent consumers do different jobs off the same streams: one updates the read model, one sends emails, one feeds analytics, one drives fraud detection. Adding a new consumer (say, a recommendation trainer) requires zero changes to producers — it just subscribes and can replay history.
Quando usar
Recorra a isso quando
- High-throughput event streaming to many independent consumers.
- Event-driven architectures, event sourcing, and CDC pipelines.
- When you need replay / reprocessing of historical events.
- Decoupling services so producers and consumers evolve independently.
Evite quando
- Simple task queues with a single consumer and low volume (a broker like RabbitMQ/SQS is simpler).
- Request/response or low-latency RPC (Kafka is a log, not an RPC system).
- Small apps that don't need retention, replay, or high fan-out — Kafka's operational weight isn't justified.
- Per-message priority or complex routing (RabbitMQ fits better).
Trade-offs
Vantagens
- Very high throughput and horizontal scalability via partitions.
- Durable, replayable retention decoupled from consumption.
- Multiple independent consumer groups read the same stream.
- Strong ordering within a partition and tunable durability.
Desvantagens
- Operationally heavy to run and tune (partitions, retention, rebalancing).
- Only partition-level ordering, not global.
- At-least-once by default — consumers must be idempotent.
- Rebalances can pause consumption; hot partitions skew load.
Kafka trades operational complexity and latency for durability, throughput, and replay. Partition count trades parallelism/ordering: more partitions mean more consumer parallelism but weaker cross-entity ordering and more overhead. acks trades latency for durability (0 fast/lossy, all slow/durable). Retention trades storage cost for replay ability. Compared to RabbitMQ/SQS, Kafka wins on throughput, retention, and fan-out but loses on simplicity, per-message routing, and low-latency single-consumer tasks.
| Kafka vs RabbitMQ | Kafka | RabbitMQ |
|---|---|---|
| Model | Distributed log (pull, retained) | Message broker (push, ack-and-delete) |
| Replay | Yes — re-read by offset | No — consumed messages are gone |
| Ordering | Per partition | Per queue |
| Throughput | Very high | High (lower than Kafka) |
| Best for | Streaming, event sourcing, fan-out | Task queues, complex routing, RPC-ish |
Erros comuns
Fique atento a
- Expecting global ordering across a topic (it's per-partition only).
- Too few partitions, capping consumer parallelism; or a bad key causing hot partitions.
- Assuming exactly-once for free instead of building idempotent consumers.
- Committing offsets before processing (risking data loss) — commit after.
- Using Kafka as a database or as low-latency RPC.
Pensando em falhas
O que quebra
A topic has 3 partitions and you run 5 consumers in one group. What happens?
Only 3 consumers get a partition each; the other 2 sit idle as hot standbys, because a partition is assigned to at most one consumer per group. Parallelism is capped by partition count — to use 5 consumers you need at least 5 partitions. The idle consumers still provide fast failover on a rebalance.
Kafka (or a broker) goes down mid-stream. Do you lose events?
With replication (replication.factor ≥ 3) and acks=all, a committed record survives a broker loss because it exists on in-sync replicas; a follower is promoted to leader. Producers may retry (dedupe via idempotent producer / consumer). Data loss mainly occurs if you used acks=0/1 with too few in-sync replicas — durability is a configuration choice you make explicitly.
Pense como sênior
Visão de Engenheiro(a) Sênior
The two facts that reveal understanding: ordering is per-partition (so the partition key is a design decision, not an afterthought), and delivery is at-least-once (so idempotent consumers are mandatory, not optional).
Visão de Engenheiro(a) Sênior
Kafka is a log, not a queue. That single reframing explains replay, multiple consumer groups, retention, and why you don't 'delete' a message after reading it.
Lembre-se
Ordering is guaranteed within a partition, never across a topic.
Lembre-se
Kafka is at-least-once — consumers must be idempotent.
Questões de entrevista
How does Kafka guarantee ordering, and how do you scale consumers?
Recordação ativa
Teste-se
In Kafka, what guarantees message ordering?
Desafios práticos
Ordered, idempotent order pipeline
Model an order event pipeline on Kafka with correct ordering and duplicate safety.
Requisitos
- Choose topic(s) and a partition key that preserves per-order ordering.
- Implement at least two consumer groups reading the same stream.
- Make each consumer idempotent against redelivery.
Restrições
- Commit offsets after processing.
- Assume at-least-once delivery.
Critérios de aceite
- Events for one order are always processed in order.
- A forced redelivery does not create a duplicate order or duplicate email.
- Adding a new consumer group requires no producer changes.
Casos extremos
- A rebalance mid-batch.
- A poison message that always fails.
Bônus
- Add a dead-letter topic for poison messages.
- Show replay by resetting a group's offsets.
Reflexão
- Why does the partition key double as your ordering and load-balancing decision?
Desafio de arquitetura
Design the event backbone for an order pipeline: OrderPlaced → PaymentRequested → PaymentSucceeded → InventoryReserved → OrderConfirmed, with separate email and analytics consumers. Choose topics, partition keys, and partition counts; guarantee per-order ordering; and ensure a consumer crash or redelivery can't create duplicate orders or emails.
Flashcards
Resumo
Kafka is a distributed, partitioned, replicated log that decouples producers from many independent consumers and retains events for replay. Ordering holds within a partition (chosen by key), parallelism is bounded by partition count within a consumer group, and default at-least-once delivery makes idempotent consumers mandatory. It excels at high-throughput streaming and event-driven architectures but is operationally heavy and wrong for simple queues or RPC.
Pontos-chave
- Topic → partitions → ordered logs; ordering is per partition, set by the key.
- One partition per consumer in a group ⇒ parallelism capped by partition count.
- Retention enables replay and many independent consumer groups.
- At-least-once by default ⇒ consumers must be idempotent.
Suas anotações
Salvo neste dispositivo