Distributed Storage &
File Systems

Require a majority (N/2 + 1) of nodes to agree before any write or leadership decision. Only the partition with quorum continues operating. The minority partition rejects writes and waits for reconnection.

• etcd + Raft: 5-node cluster requires 3 votes. If Zone A has 3 nodes and Zone B has 2, only Zone A proceeds.
• ZooKeeper: Ensemble of servers uses leader election with quorum to prevent dual-leader scenarios.
• DynamoDB Global Tables: Uses quorum-based techniques across regions to determine the authoritative source for updates.
• Deploy clusters with odd numbers of nodes (3, 5, 7) to always guarantee a quorum majority.

Fencing forcefully isolates or shuts down the other partition to guarantee only one active node can modify data at any time. STONITH = "Shoot The Other Node In The Head."

• Power switches or management cards physically power off the presumed-dead node
• Cuts network access to shared storage from the fenced node
• Guarantees no zombie leader can continue issuing commands
• Combined with quorum in enterprise cluster managers (Pacemaker/Corosync)
• Used in high-availability database clusters to prevent dual-primary scenarios

Nodes send periodic heartbeat signals to prove they are alive. Missing heartbeats trigger failure detection — but heartbeat intervals must be carefully tuned to avoid false positives from transient network lag.

• Set heartbeat intervals based on measured network latency characteristics
• Use adaptive timeouts that account for known transient conditions (GC pauses, high CPU)
• Combine with short timeouts + robust failure detection mechanisms
• etcd: configure heartbeat-interval and election-timeout for your environment
• Avoid declaring nodes dead due to a single missed heartbeat — use a threshold (e.g., 3 consecutive misses)

For AP systems that prioritize availability, allow divergence during partition and reconcile state when the partition heals using conflict resolution mechanisms.

• CRDTs (Conflict-free Replicated Data Types) are data structures designed for automatic conflict-free merging
• Vector clocks track causality to determine which updates happened before others
• Last-Write-Wins (LWW): Simplest strategy — most recent timestamp wins on conflict
• Application-level resolution: Expose conflicts to the application for domain-specific merging
• Amazon DynamoDB, Apache Cassandra's Lightweight Transactions (LWT) use this approach

Consensus algorithms like Paxos and Raft mathematically guarantee that only one leader can be elected in any given "term" or "epoch," preventing split-brain at the protocol level.

• Raft: Simpler and more understandable than Paxos. Used in etcd, CockroachDB, TiKV. Leader holds authority for a "term"; election requires majority.
• Paxos: Academic gold standard. More complex; variations include Multi-Paxos (used in Chubby, Google Spanner).
• Term/Epoch number: Monotonically increasing — a node with a higher term automatically supersedes one with a lower term.
• Candidates campaign for votes; a vote can only be cast once per term per node.
• Mathematical proof: it is impossible for two leaders to win a majority in the same term.

// Candidate node starts election:
candidate.term += 1          // increment term
candidate.vote_for_self()
votes = broadcast_vote_request(candidate.term, candidate.log_index)

if votes >= quorum(cluster.size):
  become_leader()            // won majority → safe to lead
  broadcast_heartbeat()      // suppress other elections
else:
  revert_to_follower()       // lost → wait for next timeout