Repository navigation
One-shot persist: unified atomic persistence for ChannelManager and ChannelMonitors #4457
Description
LDK's current persistence model requires the ChannelManager and each ChannelMonitor to be persisted independently, at different times, by different callers. This creates a fundamental consistency problem: on restart, the ChannelManager and ChannelMonitor states may not agree, which is the root cause of unnecessary force closes after crashes or unclean shutdowns. To mitigate this, a significant amount of reconciliation logic runs on startup to detect and resolve inconsistencies, adding complexity and still not covering all edge cases.
The lack of atomic persistence also forces the existence of channel freezing via ChannelMonitorUpdateStatus::InProgress, where channels are paused while persistence catches up. This machinery is difficult to reason about and still has edge cases.
Proposed approach
Instead of persisting the ChannelManager and ChannelMonitors as independent operations, persist them together atomically by extending the KV store interface with atomic multi-key writes and buffering all writes through a queuing layer.
1. ChannelManager persists itself at the right moments
Rather than relying on the background processor to periodically call an external persist function, the ChannelManager holds a reference to the KV store and writes its own state. The key insight is that state changes only need to be persisted before they become externally observable. As long as no events or messages have been handed to the caller, the system can safely restart from the last persisted state and re-derive the same changes.
To avoid blocking event and message consumers on KV store I/O, persistence is decoupled from consumption through a persist_and_stage_events() method. This method serializes and commits under the write lock, then moves events and messages into staged queues. Consumers (get_and_clear_pending_msg_events, process_pending_events) simply drain from these staged queues without any I/O. persist_and_stage_events is called at the top of the background processor loop (both sync and async), and can be made into an async fn to run as part of a more parallelized background processor loop.
The current persistence flag mechanism triggers full re-serialization even when nothing recoverable has changed. This approach eliminates those redundant writes entirely.
2. Per-channel keys for granular ChannelManager updates
The ChannelManager currently serializes all of its channel state into a single blob. Instead, each channel's data (as stored within the ChannelManager, not the ChannelMonitor) is written to its own KV store key. Combined with change detection (e.g. dirty flags set by mutating operations), updating one channel out of thousands writes only that channel's key plus the small manager metadata, not a re-serialization of the entire ChannelManager.
3. Atomic multi-key commits
A QueuedKVStoreSync wrapper buffers all writes (monitor updates, manager updates) in memory. On commit(), it flushes all queued changes to the underlying KV store as a single atomic multi-key write. This guarantees that either all changes from a commit are persisted or none are. There is never a window where one is persisted without the other, eliminating the force close problem and the need for startup reconciliation.
The KV store interface is extended with an atomic multi-key write method. Backends that support transactions (e.g. databases) can implement this directly. For filesystem-based backends that cannot support true atomic multi-key writes, the implementation writes keys individually. A crash mid-write can leave the on-disk state partially updated. This includes not only monitors being ahead of the CM (as in the current model), but also inconsistency within the CM state itself, since per-channel keys mean that some channel keys may be written while others are not. Startup filtering can handle the monitor-ahead-of-CM case by skipping updates beyond what the CM recorded, but internal CM desync is a new failure mode that does not exist today. Filesystem users therefore do not gain atomicity and need additional reconciliation logic to handle partial CM writes.
Downgrade note: this approach uses the same key paths for monitors (monitors/{name}) but splits the ChannelManager across per-channel keys. Old code expects a single manager blob and would not know how to reassemble it. To support downgrade, a combined manager key must also be written on every commit, at the cost of extra I/O.
4. Async monitor persistence can be removed
The InProgress variant of ChannelMonitorUpdateStatus was originally added for performance on high-latency storage backends, where blocking on each individual monitor write would be too slow. With the queuing layer, all monitor updates are buffered in memory (no I/O) and flushed on commit. The commit can happen asynchronously as long as we hold off on sending messages and handling events until it completes. This eliminates the channel freezing machinery and the associated edge cases around fund loss.
5. Extensible to application state
Higher-level application state (e.g. a payment store in ldk-node) can piggyback on the same atomic commit, keeping application data consistent with LDK's internal state without additional coordination.
Trade-offs
Batching all updates into a single atomic commit means that channels which could in theory operate completely independently (e.g. two unrelated forwards touching four different channels) are funneled through one commit. At very large scale, this could become a bottleneck compared to a model where each channel or channel pair persists independently. In practice, we are likely far from the point where this matters, and there are other scaling paths such as running multiple nodes (though that is less capital efficient). The simplicity gained by having a single, consistent persistence model is worth a lot: it eliminates entire classes of bugs and makes the system much easier to reason about.
Note that the global persist model does not necessarily block unrelated channels from making progress. The write lock only needs to be held during snapshotting. After that, I/O happens in the background with the channel manager fully unlocked for further operations, including another snapshot and I/O that can potentially run in parallel with the first.
Variants
CM-as-commit-point (no queuing layer)
A lighter variant uses the ChannelManager persist as the commit point instead of batching all writes through a queuing layer. Monitor updates are written directly to the store as they happen today. The CM persists itself at two chokepoints: before returning events from process_pending_events and before returning messages from get_and_clear_pending_msg_events. On startup, the CM is loaded first (benefiting from the recent split of channel manager data reading and instantiation) and any monitor update files with an update_id beyond what the CM recorded are skipped during replay.
This shares the core insight (persist before externally observable side effects) but avoids changes to the KV store interface and the queuing/commit infrastructure. The trade-offs are that orphaned monitor update files can exist on disk after a crash and need to be filtered on load, rather than being prevented by construction. Incremental CM persistence (per-channel keys) also does not apply here, because the CM must land atomically in a single write to serve as the consistency boundary.
Downgrade note: this variant uses the same on-disk format as today, so downgrade works without special handling. The worst case after downgrade is a monitor ahead of the CM due to a crash, which is the same risk as the current model.
Single-key delta concatenation
Instead of extending the KV store with atomic multi-key writes, all queued changes can be concatenated into a single value and written under a unique sequenced delta key. Because any KV store can write a single key atomically (e.g. FilesystemStore uses write-to-temp + rename), this provides atomicity without backend support for multi-key transactions. On startup, the base snapshot plus any unconsolidated delta keys (ordered by sequence number) are replayed to reconstruct the current state. Reads during normal operation check the in-memory queue first and fall back to the inner store.
The trade-off is that this variant changes the on-disk format fundamentally, and adds the overhead of concatenating all changes into a single value on write and splitting them back out on read.
Downgrade note: this variant changes the on-disk format fundamentally (sequenced delta keys instead of the current paths), so there is no simple downgrade path. Either every commit would also need to write the data in the old format, or a standalone downgrade tool would need to be provided to convert back before reverting to an older LDK version.
Proof of concept
An early proof of concept is available at https://github.com/joostjager/rust-lightning/pull/new/one-shot-persist. It is incomplete and not production-ready, but demonstrates the core ideas described above.
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
TheBlueMatt commentedon Mar 3, 2026
This would block
get_and_clear_pending_msg_eventson persistence, which we really shouldn't do IMO.joostjager commentedon Mar 3, 2026
It is intentionally blocking on persistence, to gather as many writes as possible before committing. What downside do you see?
TheBlueMatt commentedon Mar 3, 2026
Probably the biggest goal of our async persistence logic is to avoid blocking latency-sensitive actions on persistence. That includes the P2P logic as well as user-facing methods like fetching balances or listing pending transactions. If we didn't have that as a goal we could implement async persist by just doing native rust async await.
joostjager commentedon Mar 3, 2026
How are these scenarios impacted by one-shot persist exactly? P2P messages are blocked on persistence already. And something like fetching balances isn't blocked because the actual disk writing (commit) is happening outside of the lock?
TheBlueMatt commentedon Mar 3, 2026
P2P messages (for specific channels) don't go out, but the thread(s) doing P2P management aren't blocked waiting on persistence.
joostjager commentedon Mar 3, 2026
You're right that with one-shot persist, all outgoing messages are blocked until the commit completes, not just the ones for the channel that changed.
That said, in the current model safety-critical messages are also blocked on an fsync completing before they go out. One-shot persist doesn't change that, the difference is scope. Today only the specific channel's messages are held, while with one-shot persist all messages returned from
get_and_clear_pending_msg_eventswait for a single fsync. But it is a single fsync, replacing what would otherwise be multiple independent fsyncs for the individual monitor updates. The total time spent waiting on disk may actually decrease.Even if there is some regression in message dispatch latency for unrelated channels, we're talking about the duration of a single fsync per event cycle. It seems unlikely this would be a practical problem, and the reduction in complexity is substantial: no more channel freezing, no more
InProgressstate machine, no more force close edge cases, no more startup reconciliation.If the latency does turn out to matter, one optimization would be to split messages into two buckets: safety-critical ones that must wait for the fsync (e.g.
commitment_signed,revoke_and_ack) and non-critical ones that can be dispatched immediately.I think it would be a good idea to come up with a concrete benchmark scenario to measure the real impact on message dispatch latency, comparing the current per-channel persistence model against one-shot persist under realistic load.
joostjager commentedon Mar 5, 2026
A lighter variant of this: use the ChannelManager persist as the commit point instead of batching all writes through a queuing layer. Monitor updates are written directly to the store as they happen today. The CM persists itself at the same two chokepoints. On startup, the CM is loaded first (benefiting from the recent split of channel manager data reading and instantiation) and any monitor update files with an
update_idbeyond what the CM recorded are skipped during replay.This shares the core insight (persist before externally observable side effects) but avoids changes to the KV store interface and the batching/delta infrastructure. The trade-offs are that orphaned monitor update files can exist on disk after a crash and need to be filtered on load, rather than being prevented by construction, and that it loses the single-fsync advantage since monitor updates still hit disk individually between chokepoints.
Incremental CM persistence is orthogonal and applies equally to both approaches.
joostjager commentedon Mar 9, 2026
Added another commit to the PoC branch that addresses the remaining blockage in the event/message fetching methods.
Previously,
get_and_clear_pending_msg_eventsand event processing still performed persistence inline, blocking callers on KV store I/O. The new commit introduces apersist_and_stage_events()method that decouples persistence from consumption: it serializes and commits under the write lock, then moves events and messages into staged queues. Consumers (get_and_clear_pending_msg_events,process_pending_events) now simply drain from these staged queues without any I/O. This is called at the top of the background processor loop (both sync and async).persist_and_stage_eventscan be made into an async fn, allowing it to run in the background as part of a more parallelized background processor loop.joostjager commentedon Mar 10, 2026
Offline we discussed whether a global persist model blocks persistence of unrelated channels. But does it? The write lock only needs to be held during snapshotting. After that, I/O happens in the background with the channel manager fully unlocked for further operations, including another snapshot and I/O that can potentially run in parallel with the first.
joostjager commentedon Mar 18, 2026
To put some numbers behind the performance discussion, I ran two fio benchmarks that model the current per-channel persistence vs. one-shot persist on an NVMe-backed system (macOS, APFS).
Current model: 20 channels persisting independently, each writing 4KB with its own fdatasync:
Write latency median: 6 µs. fdatasync median: 155 µs, 99.99th percentile: 44 ms.
One-shot model: single writer, 80KB (20 × 4KB) per write, one fdatasync:
Write latency median: 9 µs. fdatasync median: 26 µs, 99.99th percentile: 2 ms.
Write latencies are single-digit microseconds in both cases, so the cost is entirely dominated by fdatasync. Since fdatasync flushes the disk's write cache to stable storage, its cost is largely fixed per call rather than proportional to data size for small writes. The 20-job case pays that cost 20 times and the concurrent flushes contend with each other, pushing latency up across the board. One-shot pays it once.
In terms of useful data committed to stable storage per second, one-shot achieves 1,708 MiB/s vs. 387 MiB/s for the per-channel model, a ~4.4x improvement. The disk spends far less time on flush overhead and more time on actual data transfer.
So even though one-shot persist makes all channels wait for one fsync instead of each channel waiting for its own, that one fsync finishes faster in absolute wall-clock time than any of the competing per-channel fsyncs do today.
This is a rough model and I welcome refinements, for example varying the number of channels, block sizes, or running on different OS/filesystem/storage combinations. But if these results hold, it means one-shot persist is not a simplicity-for-performance trade-off. It improves both.
joostjager commentedon Mar 19, 2026
Tested on linux where I get a similar result.
TheBlueMatt commentedon Mar 19, 2026
Went and tested on my ceph cluster and 80kb was about the best number to pick 😂 . If I go up from there, the total bandwidth doesn't really change, ie the latency only goes up.
joostjager commentedon Mar 19, 2026
I swear I didn't cherry-pick 😅 20 parallel channels seemed reasonable to me.
Can you share exact commands and results?
When I 10x the sizes I get:
So the independent monitor throughput stays roughly the same (387 -> 439 MiB/s), while batch writes go up quite a bit (1708 -> 5127 MiB/s). The gap actually widens at scale.
It makes sense that independent monitors can be faster when each monitor gets its own disk (ceph), since the per-call fdatasync cost is then truly parallel across separate physical devices. But in setups where everything lands on the same disk, the contention from concurrent fsyncs dominates and the batched model wins by a larger margin as channels grow.
That said, I highly doubt any of this is relevant in practice. Both options deliver throughput that is orders of magnitude beyond what a Lightning node needs. The real bottlenecks are going to be elsewhere: network round trips, HTLC processing, peer message handling. We're benchmarking a part of the system that is unlikely to be the constraint.
Given that performance is a non-issue either way, I think simplicity should be the deciding factor. Both in the end result (no channel freezing, no InProgress state machine, no startup reconciliation) and in the migration path to get there. One-shot persist is a simpler model that also happens to perform well, not a trade-off between the two.
tnull commentedon Mar 24, 2026
While I have to say I like the simplicity of the general concept in general (and doing atomic multi-key writes in particular), I have to say I'm also a bit concerned whether we won't end up blocking concurrent access to
ChannelManagermethods, or further delaying event processing. The latter would ofc. be particularily annoying in cases where we have head-of-line blocked events that themselves wouldn't need to wait for persistence at all. If we consider going with one-shot persistence, it might be worth revisiting splitting the event queue into events that need to be held back and events that are free to make progress?joostjager commentedon Mar 24, 2026
Blocking concurrent access to ChannelManager methods: the write lock is only held during in-memory snapshotting/serialization. The actual disk I/O happens outside the lock, so methods like balance queries or listing channels, or independent channel ops, are not blocked during the fsync.
Event processing is never literally blocked, because events are consumed from a staged queue without any I/O. They are only staged after persistence completes, but most events need to wait for persistence anyway. The set that would benefit from bypassing the persistence gate is I think small and infrequent? If it turns out to matter, we could indeed split the queue.
joostjager commentedon May 20, 2026
I repeated the same fio comparison on an EC2 instance backed by EBS gp3, using the same parameters as above: 20 jobs writing 4 KiB each with independent
fdatasync, versus one job writing 80 KiB with onefdatasync.Results on gp3:
So on what is arguably a likely deployment setup, a single gp3-backed EBS volume, batching monitor writes does not appear to hurt performance. It is actually faster, both in useful throughput and median sync latency. The independent-write case also lands very close to the default gp3 3,000 IOPS baseline, which suggests it is spending the IOPS budget on tiny durability barriers rather than useful data.
There may be storage topologies that reap more benefit from independent monitor writes, for example setups where those writes truly land on independent physical devices and can flush in parallel. But preserving that optimization means accepting worse performance on arguably more common single-volume/cloud-volume deployments, while also keeping the more complex persistence model.
tnull commentedon May 20, 2026
Would be very interested in benchmarks based on a realistic Postgres deployment (post lightningdevkit/ldk-node#863), as this is what most users will likely run and are currently looking to improve upon when migrating. Does the god commit improve or worsen the story there?
joostjager commentedon May 20, 2026
I expect Postgres to come down to the same underlying bottleneck. Each committed write still has to pass through the fsync barrier.
My opinion remains unchanged that we are over-optimizing this part of the system: all of these persistence options appear more than sufficient for realistic Lightning node throughput. Simplicity and correctness of the persistence model are far more important imo, since they translate directly into better funds safety and faster development velocity.
joostjager commentedon May 27, 2026
@tnull, not a real deployment, but a postgres datapoint nonetheless: #4633 (comment)
joostjager commentedon Jun 12, 2026
The dev sync (summarized in #4633) named failover with locking as a near-term ask. Worth noting how that interacts with the persistence model, because it goes beyond a lock row.
Conditioning each write on a lease is cheap in any model: a WHERE clause in a transactional KVStore impl. The difference is what the check buys. With independent per-object writes, the lease can transfer mid logical operation, so the successor takes over from an arbitrary cut and failover safety rests on restart reconciliation being correct for every interleaving. The loser must also treat a rejected write as terminal in every async persist pipeline, the same unsolved area as panic-on-persistence-failure. With atomic commits both problems collapse: takeover is just loading the last consistent snapshot, and the loser is contained because nothing externally observable is released except after a successful fenced commit.
So the distance from "lock row in Postgres" to "safe failover" is near zero under atomic commits, and a reconciliation-plus-error-semantics project under independent writes. Suggest we make failover safety an explicit criterion for evaluating persistence proposals.
joostjager commentedon Jun 23, 2026
I (AI) put together a PoC branch here as a concrete preview of what a consistent commit boundary could look like:
main...joostjager:rust-lightning:atomic-commit-poc
The branch assumes ChannelMonitor updates are durably included in the same atomic state commit as the ChannelManager, before any corresponding messages or events are released. Under that assumption, a lot of the machinery for independent monitor persistence becomes unreachable and can be removed rather than carried forward as in-flight state.
ldk-reviews-bot commentedon Sep 3, 2026
@joostjagercommented on Forgejo:With persistence split-brain issues becoming particularly prominent in ldk-node, I think it is time for another take on one-shot persistence, this time including ldk-node state in the same commit as the
ChannelManagerand monitors.The sketch assumes atomic batched KV writes and keeps transaction handling centralized in the store and background processor. Operations such as
send_paymentuse a guard and wait for the next atomic commit before returning. Messages and events are gated until that commit succeeds, so external effects cannot become visible before the corresponding LDK and ldk-node state is durable. No database transaction is passed through the API call graph.The focus here is the resulting API and commit coordination. It deliberately does not implement deltas or partition the
ChannelManageryet.ldk-node: lightningdevkit/ldk-node@main...joostjager:poc-consistent-commit
rust-lightning: https://git.rust-bitcoin.org/lightningdevkit/rust-lightning/compare/main...joostjager:poc-consistent-commit