DSS Raft-Based Application-Level Consensus - August 2026 update

[barroco]

Raft-Based Application-Level Consensus passes the full DSS test suite (uss_qualifier and prober included) reproducing SQL-based behaviour end to end and outperforming the distributed SQL implementation in every performance scenario, meeting throughput, latency and error-rate targets.

Introduction

This update reports on the current status of the effort to move to application-level consensus for the DSS architecture.

Background

Following Cockroach Labs' late-2024 license change, the Technical Steering Committee (TSC) investigated mitigation strategies.

At the time, the options evaluated were:

• accepting CockroachDB's commercial terms.
• migrating to a different database (rqlite, YugabyteDB, Vitess, TiDB).
• adopting a distributed key-value store.
• moving consensus into the DSS application itself.

On the application-level consensus track, CAS Paxos was rejected due to a dormant ecosystem, no reusable library and implementation-sensitive correctness. This left Raft via the etcd/raft library the only realistic algorithm. A full redesign document (InterUSS DSS 2024 redesign) was attempted but did not find common ground, notably on the storage backing the consensus engine. The proof-of-concept attempts did not succeed either, in part because the etcd/raft library was not provided as a standalone project at the time.

As a result, the TSC prioritized the switch to YugabyteDB (delivered from v0.19). However, it was agreed to reconsider this decision if a contributor could provide a working DSS prototype using etcd/raft library.

Goals

The goals remain unchanged from the investigation - the Must-Haves remain the review criteria for any consensus design, especially:

• Respond to a USS only after the transaction is recorded in the DAR (DSS Airspace Representation) as required by F3548-21 - DSS0215 and F3411-22a - DSS0070.
• Maintainable by available InterUSS contributors.
• Room to grow to roughly country-scale traffic.
• A pool of three or more instances survives the failure of one without substantial latency impact.
• operators control pool membership.
• No cloud-provider lock-in.
• Every DSS instance is an equal peer (temporary Raft leadership is acceptable).
• Explainable to semi-technical regulators.
• USSs can meet their own performance requirements.

Desirable in addition: no restricted-license dependencies, minority-failure tolerance, low deployment and maintenance effort, resistance to accidental DAR corruption, and the ability for a participant to improve its own performance unilaterally.

Load testing scenarios based on production experience have added concrete targets since 2024 for country-scale traffic: SCD same-cell throughput is expected at ~2 QPS (#1541), multi-cells around ~45 QPS, with limited latency (< 5s) and error rates (< 1%), in a 9 node environment with various latencies (see this comment). Since then, the assumption on the number of nodes has been updated to meet users feedback (>24).

What did the thesis prototypes unlock?

Mariem Baccari's EPFL master thesis (Distributed transactions ensuring deconfliction of long range drone flights, Feb 2026, in the investigation folder) met the TSC's revisit condition on a working toy DSS:

• A working single-Raft DSS prototype on the intended etcd raft library, with its WAL (Write-Ahead Log) and snapshotting: writes applied only after a majority accepts, ~300-400 QPS sustained on three nodes, leader crash absorbed with a ~20% throughput dip and ~25-second recovery, and stable, linear latency under injected network delay.
• A working multi-Raft prototype answering the scalability question that the 2024 conversation left open: S2-cell-bounded geographic shards, each its own Raft group, with cross-shard transactions via two-phase commit through a transaction-coordination Raft group. Three isolated shards delivered the expected three time single-Raft throughput. However, the overhead from the two-phase commit and other multi-Raft induced coordination efforts affect performance scalability during cross-shard transactions based workloads. Therefore, the multi-Raft extension still requires more effort into the design in order to achieve the performance goals.
• Resolution of the old blocker. The library packaging problem that sank the 2024 toy DSS is gone: the implementation now underway imports etcd/raft and reuses etcd's rafthttp transport, WAL, and snapshotter directly, so InterUSS only implements a simple wrapper (~1,100 lines at the current snapshot).

The design relies on etcd/raft as the underlying consensus engine. The library implements the Raft consensus algorithm and is actively maintained and used by systems such as CockroachDB and etcd itself.

Execution Plan

The thesis findings are moving to the repository: the incremental raftstore implementation is landing on master as a distinct datastore. The design decisions are captured in the package’s README and discussed in issues / pull requests. The architecture and progress is tracked in the issue tracker (#1463) and the measurements in #1594.

As part of the move and build up on the existing code base, an approach to the storage question raised in the redesign document is proposed and implemented (#1526): in-memory per-service state, made durable via the Raft log and snapshots.

The work is organised into four stages, each gated on a measured result. At the time of writing this post, gates 0 (functional) and 1 (performance) passed test criteria and code is under review.

• Gate 0: Functional practicability: does application-level Raft comply with the DSS test suites?
• Gate 1: Performance fitness: does it meet country-scale traffic requirements?
• Gate 2: Production readiness: can users run it in production ?

Gate 0: Proof of Functional Practicability

Goal: the Raft implementation passes the full DSS test suite, including uss_qualifier and prober.
Status: Tests passed on code now under review

Gate 0 reproduces the behaviour of the SQL implementation, including its legacy cases, on top of application-level Raft. It passes the full DSS test suite.

This stage addressed multiple aspects:

• Consensus layer: wraps etcd/raft's consensus implementation, including cluster configuration with the various base options. The DSS consensus layer only makes high-level calls: proposing values, stepping the clock, and consuming Raft updates. The architecture overview is documented in #1463.
• Transport: etcd/raft is transport agnostic. Our implementation relies on rafthttp, etcd's own peer-to-peer transport implementation which is already optimized for raft. The alternative was to implement a grpc transport from scratch. With the thesis prototype naive implementation, the overall performance was similar so the more practical choice of rafthttp was preferred for the moment and can be revisited later if needed (#1464).
• In-memory data store as projected view: etcd/raft leaves storage to the application. The options for holding state in memory were evaluated in #1526; following that evaluation, in-memory data stores were implemented for every DSS component (RID, SCD, AUX).
• Consistency guarantees: Both reads and writes go through the same path as a Raft request (#1474) and are replicated via a consensus round, ensuring linearizability. In the future, using ReadIndex for read-only requests will maintain the same guarantees without the need for a consensus round.
• Store update: #1525 applies committed proposals to the in-memory store, completing the write path: a write is confirmed to the USS only once the proposal has been committed and applied locally, as DSS0215 requires.

The existing test baseline passes against the initial implementation, as shown by a77b08d in #1538 (uss_qualifier and prober results).
Code is currently reviewed and progress is tracked in #1463.

Gate 1: Performance Fitness

Goal: the Raft implementation meets the throughput, latency and error-rate targets recorded in #1541.
Status: Tests passed on code now under review

Results are recorded in #1594. The Raft implementation outperforms the distributed SQL implementation in every test scenario: SCD same-cell throughput is at ~95 QPS (45x requirement), multi-cells around ~80QPS (2x requirement), with limited latency (<300ms below <5s requirement) and error rates (0% below 1% requirement), in a 9-node environment with various latencies.

Payload size was identified as an immediate limiting factor: when simulating network latency, bandwidth becomes the main constraint. #1587 addresses the low-hanging fruits with:

• better encoding of proposals
• compression of proposals
• OVN format optimisations
• removal of unused keys

Analysis of the remaining bottlenecks continues and is expected to yield further improvements.

Gate 2: Production Readiness

Goal: a DSS user can deploy, upgrade, observe and change the membership of a Raft-backed pool in production.
Status: Work in progress

This stage will cover what a pool needs over its lifetime and optimizations required for a large number of DSS instances (>24), especially:

• Fault injection: partitions, clock skew and node restart during apply need to be tested.
• Membership changes: cluster membership is fixed at Gate 1. Adding and removing nodes needs tooling, and membership changes must remain safe while the pool is serving traffic.
• Updates: the SQL migration system relies on a shared datastore and does not carry over. Its replacement must handle snapshots already on disk, proposals in multiple version formats, and enabling new behaviour coherently across a mixed-version pool.
• Non-voting nodes: as the number of participants grows, not all have to be voters. Voting membership must be decoupled from pool membership, and the voting set balanced, automatically or by configuration.
• Pool authentication: TLS is the core of authentication of a DSS pool today, and tooling, procedures and related code need to be adjusted for the raft pooling before being production-ready.
• Monitoring/Diagnostics: #1545 adds a first set of metrics to the OpenTelemetry interface. The set will be extended to support operational diagnostics of the pool.
• User documentation: deployment, membership changes, upgrades, monitoring and TLS all need documented procedures.

Future-Level Scale

If future demand requires it, the single-leader bottleneck may be removed and the single raft group may be split into multiple independent Raft groups. The thesis prototype partitions the airspace into static S2-cell-bounded shards, one Raft group each, with cross-shard transactions coordinated by two-phase commit through a dedicated group, the protocol CockroachDB and Google Spanner use for the same purpose.

What will change for users?

The Raft datastore will ship as a selectable datastore alongside the SQL ones. Current deployments are unaffected. When enabled, the DSS node becomes a single container: consensus moves inside the DSS process, and no external datastore needs to be deployed.

Next steps

Following the successful demonstration of functional practicability and performance fitness, design and implementation efforts will now move toward production readiness (Gate 2). Future work may focus on scaling to sustain very-high-usage traffic, if required.

[vasilisnasopoulos]

Thanks for the detailed write-up — the Must-Haves list is unusually clear, and one line in it caught
my attention: "Every DSS instance is an equal peer (temporary Raft leadership is acceptable)."

I have been building a leaderless replicated log (every node an equal peer, no election, no shared
database), and I wanted to know whether it can carry DSS deconfliction at all before saying anything
here. So I put the F3548 operational-intent rule on top of it — accept an intent only if its key
names every accepted intent it intersects, otherwise 409 — applied by every instance in the same
log order, and ran a busy single-cell load modelled on FlightsInSub.py (8 and 32 writers, 10 s
intents, same altitude model). Three instances, Frankfurt / Tokyo / Los Angeles, each USS talking to its own:

	your busy-cell target (#1541)	8 writers, 1 s seal	32 writers, 250 ms seal
write throughput	≥ 2 QPS	3.27 accepted/s	48.7 accepted/s
latency	p99 < 5 s	p99 3.09 s	p50 0.47 s · p95 0.53 s · p99 1.55 s
5xx	< 1 %	0 / 152	0 / 1,779

Every instance made the same decision on every intent, and no two accepted intents intersect
without the later one's key naming the earlier. Then I stopped the Los Angeles instance halfway:
Frankfurt and Tokyo answered again after 0.55 s, longest gap 2.13 s (same as before), 0 / 111
errors — nothing to elect.

(the recording is one machine on loopback so it fits in a GIF; the numbers are from the three servers)

All runs, raw per-request data, each instance's decisions and a verify.py that recomputes every
number: https://github.com/vasilisnasopoulos/vortex-dss-demo

What this is not, to be clear up front:

• It is not faster than your Raft datastore (~95 QPS, < 300 ms, 9 nodes with simulated latency).
  Latency here is the sealing period, set per deployment; on three real continents 250 ms held; at
  150 / 100 ms (p50 0.27 / 0.21 s) nodes stopped agreeing on every intent. Deciding only on a
  prefix every instance has confirmed fixes agreement at every setting, but today costs p50 ~0.9 s
  (one new connection per confirmation across continents). The argument is "no
  leader", not speed.
• Only PUT operational_intent_references, bounding boxes instead of polygons, no subscriptions or
  notifications; DSS0215 not checked; three instances, not > 24.
• My own load generator with your test's semantics — not uss_qualifier or your locust files.
• The engine is not open source; I can run your tests on it and post the raw outputs here.

If equal peers without a temporary leader matters for Gate 2 (partitions, clock skew, > 24 nodes),
I would be glad to run your own load tests and uss_qualifier against it and report the numbers
here, good or bad.