Custom Quorums in OpenRaft: Configuration and Failure Trade-Offs

Overview

Custom membership in OpenRaft lets applications define separate election and commit quorums based on node count, weight, or deployment topology. This article compares five implementations and their failure trade-offs, then walks through a weighted key-value (KV) service that keeps its primary site available during a secondary-site outage. The examples distinguish continued writes from leader replacement and explain the durability and configuration-change requirements of each design.

OpenRaft and Databend Metadata Availability

OpenRaft supports custom membership, allowing applications to choose which nodes must participate in an election or confirm a commit. A cluster of 4 nodes can commit with 2 replication confirmations instead of the usual majority of 3. A cluster of 9 nodes across 3 sites can require 2 confirmations at each of any 2 sites, committing with 4 confirmations instead of 5.

OpenRaft provides the consensus layer for Databend's Meta Service. Databend is an open-source data warehouse built in Rust. It stores table data in object storage and uses Meta Service to manage metadata, including database and table definitions. OpenRaft runs inside Meta Service, where it elects leaders and replicates metadata changes among nodes.

Databend
+-- Query: executes queries and accesses table data in object storage
+-- Meta Service: manages shared metadata
    +-- OpenRaft: leader election and replicated metadata changes

Metadata availability affects database operations even when the table data is still accessible. Creating a table or committing a metadata update requires the metadata service to make progress. Replica placement and the failures tolerated by its quorum rules therefore affect database availability.

Custom membership lets an application prioritize availability at a primary site, require copies at separate sites, or continue committing while some replicas are unavailable. Each choice has trade-offs. Waiting for fewer confirmations may reduce commit latency, but it can also mean fewer guaranteed copies at commit or stricter requirements for electing a replacement leader. Latency may still depend on a slow site whose confirmation is required.

The configurations below demonstrate OpenRaft's capabilities. They do not describe Databend's current production quorum configuration. The application programming interface (API) examples and protocol details refer to the specific OpenRaft revision used in this article.

1. Use Different Quorums for Elections and Log Commits

An application can reduce the number of replicas needed for a commit while requiring more votes in an election. Consider nodes

{1,2,3,4}

write quorum = 2

read quorum = 3

Leader 1 receives a client write

foo=1

{1,2}

A replacement leader needs votes from 3 nodes. For example, candidate 3 can win with votes from

{1,3,4}

{1,3,4}

{1,2}

append entries: write quorum {1,2}
election:       read quorum  {1,3,4}
shared voter:               {1}

Every read quorum must intersect every write quorum. An election therefore reaches at least 1 node holding each committed entry. Together with Raft's log rules and vote checks, that intersection keeps committed entries in the next leader's log. Flexible Paxos describes a similar separation between election and replication quorums.

Here, read quorum refers to the log checks performed during an election. It does not describe how an application reads a key or executes a Structured Query Language (SQL) query. A linearizable read still uses OpenRaft's leadership validation and applied-state checks. Choosing

read quorum = 3

The number of votes needed for an election also depends on the leader ID type. The examples below use OpenRaft's default advanced leader IDs. These IDs have a total order, so a read quorum is sufficient to elect a leader. With standard leader IDs, the vote set must satisfy both the read and write quorum rules. For example, a 3x3 grid needs 3 election votes with advanced leader IDs, but 5 with standard leader IDs. The election quorum documentation explains the distinction.

The remaining nodes still receive AppendEntries requests. A smaller write quorum changes which confirmations a commit waits for; it does not automatically reduce total replication traffic. It also reduces the number of copies guaranteed at commit. The examples assume crash failures, correct storage behavior, and communication among the surviving nodes. They do not cover Byzantine failures.

2. Five Membership Implementations for Different Deployment Policies

The

openraft-membership

Membership

ThresholdMembership: Quorums by Node Count

ThresholdMembership

write_size = 2

read quorum = voter count - write quorum + 1
            = 4 - 2 + 1
            = 3

Use this implementation when all voters count equally and reducing the wait for a commit matters more than being able to elect a replacement leader with fewer surviving nodes. A leader in this cluster waits for 1 follower; a majority in a 4-node cluster requires confirmations from 2 followers.

Failures expose the availability trade-off. If leader 1 and node 2 remain while nodes 3 and 4 fail, the leader can keep committing. But those 2 surviving nodes cannot complete an election. If the cluster needs a replacement leader, it must recover another voter first.

Do not use a small write quorum when the application requires more guaranteed copies before acknowledging a write. Node count alone does not account for failure domains: 2 confirmations may come from the same rack or site. Use a placement-aware rule when the requirement is to survive the loss of a whole site.

WeightedMembership: Quorums by Node Weight

WeightedMembership

{1,2,3,4}

{1:2, 2:1, 3:1, 4:1}

write_weight = 3

read weight = total weight - write weight + 1
            = 5 - 3 + 1
            = 3

Put nodes 1 and 2 at site A, and nodes 3 and 4 at site B. Site A has weight 3, so it can elect a leader and commit without site B. Site B has weight 2, so it cannot do either without site A. Nodes

{2,3,4}

Weighted membership suits deployments that deliberately favor one site or assign different voting weights to nodes. In this configuration, 4 nodes are enough to keep the primary site available during a secondary-site outage.

Do not use this configuration for symmetric site failover. Losing site A stops progress at site B. Weights also do not create physical replicas or automatically make the highest-weight node the leader. If a node's weight alone reaches the write weight, a commit may have only that node's copy. Review weight assignments for both durability and availability.

HierarchicalMembership: Quorums by Site and Rack

HierarchicalMembership

Leader 1 can commit

foo=1

Hierarchical membership suits deployments with explicit requirements for failure domains. If site A fails, sites B and C can still elect a leader and commit. The cluster also survives the loss of 1 node in each site. More levels can express datacenters, racks, and servers.

The grouping also limits which failures the cluster can tolerate. If only

{1,4,7,8,9}

A quorum of 4 has different failure properties from a majority of 5. The hierarchical rule guarantees a particular distribution of copies and tolerates specific failures, so it cannot be treated as an equivalent majority rule with fewer confirmations. It also requires confirmation from another site, so it cannot provide site-local commits when inter-site communication is unavailable.

GridMembership: Full Rows for Read Quorums, Every Row for Write Quorums

GridMembership

election: read quorum {1,2,3}
+-------------------------+
|   [1]      2       3    |
+-------------------------+
     4      [5]      6
     7       8      [9]

[n] = node in write quorum {1,5,9}

Candidate 1 can win with votes from

{1,2,3}

{1,5,9}

A grid is suitable when its layout matches the expected failure patterns and the reduction in quorum size justifies the placement constraints. With live nodes

{1,2,3,5,9}

{1,5,9}

Failures affect elections and commits differently. Losing nodes

{1,4,7}

{2,5,8}

Do not use a grid when arbitrary node failures are the main concern. This 3x3 grid tolerates any 2 node failures, while a majority of 9 tolerates any 4. Mapping each row to a site also makes every site mandatory for commits. Check that the failures a grid can tolerate match the deployment requirements before choosing it for its smaller quorums.

ExplicitMembership: List the Quorums Directly

ExplicitMembership

{1,2,3}

minimal read quorum    minimal write quorum
{1,2}                  {1,2,9}
{1,3}                  {1,3,9}
{2,3}                  {2,3,9}

The configuration has 4 voting nodes; the backup node has ID 9. It does not require a 9-node deployment. A set qualifies if it contains a listed quorum. Every listed read quorum intersects every listed write quorum, and the constructor checks those intersections.

An explicit list works well when only a few quorum combinations are needed to express the policy. In this example, every acknowledged write has a copy at node 9. The cost is synchronous backup replication: if node 9 or its network link fails, every commit stops. Two primary voters can still complete an election under advanced leader IDs, but a new leader still needs node 9 to commit its initial entry and complete linearizable reads. Standard leader IDs require node 9's vote for the election as well.

Do not use this policy if backup outages must leave writes available. It also does not make the backup site independently available after the primary site fails. Avoid explicit lists for ordinary large threshold policies: the number of minimal quorums grows combinatorially, and validation compares every listed read quorum with every listed write quorum. A count, weight, or tree rule is easier to maintain when it expresses the same requirement.

3. Keep a Primary Site Available With a Weighted KV Cluster

A metadata KV service deployed across 2 sites can prioritize availability at the primary site while accepting that failover will not work in both directions. Suppose most clients run at site A. The service should keep accepting writes when site B is unavailable, and it should tolerate any 1 node failure while the other 3 nodes can communicate. The operator accepts that site B cannot take over by itself.

Use the weighted configuration from the previous section:

127.0.0.1:37001

127.0.0.1:36001

127.0.0.1:37002

127.0.0.1:36002

127.0.0.1:37003

127.0.0.1:36003

127.0.0.1:37004

127.0.0.1:36004

Both the read weight and write weight are 3. The addresses above let the example run on 1 machine; sites A and B represent the intended deployment and failure groups. OpenRaft's weighted KV example illustrates this application design. It does not describe a Databend Meta Service deployment.

Select the Membership and Its Edit Type

The application's

TypeConfig

types_kv

use openraft_membership::{WeightedChange, WeightedMembership};

openraft::declare_raft_types!(
    pub TypeConfig:
        D = types_kv::Request,
        R = types_kv::Response,
        Node = openraft::NodeInfo,
        Membership = WeightedMembership<Self::NodeId, Self::Node>,
        MembershipEdit = WeightedChange<Self::NodeId, Self::Node>,
);

Initialize the Cluster and Promote Learners After They Catch Up

Initialize node 1 and wait for it to become the leader and apply its initial entry. Then add the other nodes as learners before assigning voting weights. In the following async setup code,

raft

TypeConfig

use std::collections::BTreeMap;
use std::time::Duration;

use openraft::{LogIdOptionExt, NodeInfo};

let node_info = |id: u64| {
    NodeInfo::new(
        format!("127.0.0.1:{}", 37000 + id),
        format!("127.0.0.1:{}", 36000 + id),
    )
};

let membership = WeightedMembership::new(
    BTreeMap::from([(1, 1)]), // Initial voter and its weight.
    1,                      // Initial write weight.
    BTreeMap::from([(1, node_info(1))]),
)?;
raft.initialize(membership).await?;

// Membership updates need the leader's initial entry to be committed.
raft.wait(Some(Duration::from_secs(10)))
    .metrics(
        |m| m.current_leader == Some(1) && m.last_applied.index() >= Some(1),
        "node 1 leads and has applied its initial entry",
    )
    .await?;

for id in [2, 3, 4] {
    // true waits for the learner to catch up before returning.
    raft.add_learner(id, node_info(id), true).await?;
}

let weights = BTreeMap::from([(1, 2), (2, 1), (3, 1), (4, 1)]);
let change = WeightedChange::weights(weights).with_write_weight(3);
raft.change_membership(change, false).await?;

The single-node bootstrap prepares the cluster but provides no fault tolerance. The final membership becomes effective through OpenRaft's membership-change protocol. Because the weights change, OpenRaft commits a joint config before moving to the target config. The

false

Commit a Write, Stop the Secondary Site, and Write Again

The process test exercises the full initialization and outage sequence:

1. Start 4 node processes, initialize node 1, add learners, and set the final weights.

2. Send the client write

   foo=1
   

   and wait until every node can read that value.

3. Stop nodes 3 and 4, representing the loss of site B.

4. Send

   foo=2
   

   to leader 1. Nodes

   {1,2}
   

   store the entry and reach write weight

   2 + 1 = 3
   

   , so the write commits.

5. Check that node 2 reads

   foo=2
   

   , and that node 1 completes a linearizable read returning the same value.

To reproduce the checked-in test, use the repository's Rust toolchain and run it at the revision used in this article:

git clone https://github.com/databendlabs/openraft.git
cd openraft
git checkout 43537dd5b7d247be49c10163cff09700915c0f60
cargo test -p raft-kv-memstore-weighted-membership \
    test_weighted_membership -- --nocapture

The test uses application ports

36001

36004

37001

37004

Understand Which Failures Stop the Service

The configured weights also determine what happens in other failure scenarios. Any replacement leader must also have a sufficiently up-to-date log and complete the normal election and initial-commit steps.

The same rule determines the durability guarantee: a commit may wait only for nodes 1 and 2 at site A. Replication to site B continues when possible, but a successful write does not guarantee that site B already holds the entry. Permanent loss of site A before replication catches up can therefore lose the latest acknowledged writes. Preserving every acknowledged write after losing either site requires a different policy, such as synchronous cross-site replication. Assigning more weight to site A cannot provide that guarantee.

Recover Replicas Within the Existing Cluster

After site B returns, let nodes 3 and 4 catch up through log replication or a snapshot. Check their replication progress before relying on these copies to survive another failure or changing their weights. Do not call

initialize()

If site A is unavailable, site B's weight of 2 does not give it the authority to form a new cluster. Restore enough nodes in the existing cluster to regain a quorum, or follow a separately planned disaster recovery procedure that accounts for possible data loss.

The executable example uses in-memory storage, so terminating a process discards its local state. Production deployments need durable logs, votes, and snapshots. The process test demonstrates quorum behavior; it does not test recovery from persistent storage.

4. How Membership Stores Quorum Rules and Node Information

A quorum policy is cluster configuration replicated across nodes. Servers cannot change it independently as a local setting. Membership contains the read and write quorum rules, the known node IDs and addresses, and the roles needed to use those nodes. OpenRaft records membership in configuration log entries so replicas receive the policy together with the node information.

Node information tells OpenRaft how to reach a replica. The application defines that metadata, which can include network addresses and site information.

NodeId

u64

Node roles distinguish the ability to receive logs from the ability to start elections. A learner normally has

NodeRole::ReceiveLogs

ReceiveLogs

Campaigner

Membership Changes: Satisfy Both Sets of Rules During Joint Consensus

Changing quorum rules requires coordination even when the set of nodes stays the same. For example, move the extra weight from node 1 to node 3. Under the old configuration, site A can commit with

{1,2}

{3,4}

OpenRaft uses a joint configuration when it cannot establish that a direct transition between the old and new rules is safe:

C_old -> joint -> C_new

joint.read  = C_old.read  AND C_new.read
joint.write = C_old.write AND C_new.write

A joint read quorum satisfies both read rules. A joint write quorum satisfies both write rules. Once the joint configuration commits, OpenRaft can move to the target configuration.

A direct transition is allowed only when every old read quorum intersects every new write quorum, and every new read quorum intersects every old write quorum. The provided implementations check whether they can establish those cross-intersections. If they cannot, OpenRaft uses a joint configuration. The provided weighted implementation uses a joint configuration for weight changes even when the set of voters stays the same.

Plan changes while enough old and new voters are available to satisfy the joint rules. Even if the target configuration could operate after an outage, the cluster must be able to reach it safely from its current configuration.

5. Implement Custom Membership When the Provided Rules Are Insufficient

A custom membership can express an application-specific policy while OpenRaft handles replication and elections. Before implementing a new type, check whether a provided implementation already expresses the policy. The primary-plus-backup example, for instance, needs only

ExplicitMembership

A custom

MyMembership

RaftMembership

read_quorum_set()

write_quorum_set()

get_node(node_id)

node_ids()

read_write_quorums_intersect()

Some(true)

supports_direct_transition_to(other)

joint_with(goal)

The implementation also defines

NodeId

Node

QuorumSet

Iter

QuorumSet::is_quorum()

ids()

QuorumIntersection::intersects_with()

For the primary-plus-backup policy, the write rule should accept

{1,2,9}

{1,2}

{1,2}

A corresponding

MyEdit

RaftMembershipEdit<MyMembership>

apply(membership, retain)

add_learner(node_id, node)

make_uniform()

TypeConfig::Membership

TypeConfig::MembershipEdit

The edit type is independent of the membership representation. An application can keep a provided membership and implement only its own edit operations, as the custom edit test demonstrates.

OpenRaft relies on the custom implementation to verify quorum intersections and transition safety. Returning

Some(true)

6. Choose the Failure Policy Before the Quorum Size

Custom membership lets an application choose which replicas must hold an entry at commit and which failures the deployment must tolerate. Thresholds choose copy counts. Weights favor selected nodes or sites. Hierarchies encode failure domains. Grids trade arbitrary-failure tolerance for smaller quorums. Explicit lists state unusual requirements directly.

For Databend and other systems with a replicated metadata service, these choices affect whether metadata updates and database operations can continue during failures. Start with a concrete requirement: continue through a secondary-site outage, preserve every acknowledged write after a site loss, or recover a leader after a particular set of node failures. Then select the simplest quorum rule that meets that requirement.

An existing leader's ability to keep committing and the cluster's ability to elect a replacement are separate availability guarantees. Evaluate both, along with the copies guaranteed at commit and the transition through membership configurations. Use a smaller quorum when its commit benefits and failure trade-offs meet the application's requirements.