A Field Guide to Raft
Consensus is not a black box labeled "just use Raft." Here are the parts that actually bite — terms, commit rules, and the invariants you only learn about at 3am.
Every distributed system eventually runs into the same wall: a set of machines needs to agree on a single value, and the network refuses to cooperate. Packets drop, nodes pause for garbage collection, and clocks drift. The whole point of a consensus protocol is to let a cluster keep making progress anyway — as long as a majority of nodes are alive and can talk to each other.
The trap most engineers fall into is treating consensus as a black box labeled "just use Raft." It works, right up until it doesn't, and then you're staring at a stalled cluster at 3am with no idea which invariant you violated.
# terms are the real clock
Forget wall-clock time. In Raft, the only clock that matters is the term — a monotonically increasing integer that fences off one leader's reign from the next. Every message carries a term, and any node that sees a higher term immediately steps down.
if req.term < currentTerm {
return Reply{ term: currentTerm, success: false }
}
if req.term > currentTerm {
currentTerm = req.term
becomeFollower()
}
A leader is only a leader for the entries it can prove a majority already has. Everything else is optimism.
# commit is a majority, not an ack
The subtlest bug I've shipped came from conflating "the leader wrote it locally" with "the entry is committed." An entry is committed only once it's durably stored on a majority of the cluster — and a leader may only mark entries from its own term as committed by counting replicas.
Skip that rule and you can lose an acknowledged write during a leader change. That's the difference between a database and a very fast way to corrupt data.
# where to go next
If you only internalize one thing: consensus buys you a replicated log, not a replicated database. The state machine on top is your problem. Get the log right and everything above it becomes a much more pleasant kind of hard.
