Speed up bulk NFT offer creation - #877
Draft
judeallred wants to merge 2 commits into
Draft
Conversation
Add make_offers to decrypt the signing key once per batch instead of once per offer, and delegate make_offer to it as the single-item case. Remove a dead NFT data fetch in import_offer whose result was never read. Update the bulk offer-creation UI to call make_offers once instead of looping make_offer per NFT, and drop the now-inaccurate per-offer progress counter.
make_offers previously called import_offer once per offer, each opening and committing its own SQLite write transaction. In a 50-offer batch this fires 50 write transactions back-to-back with no pacing, which collides with background writers (sync manager, transaction queue) under SQLite's single-writer lock and causes escalating connection-acquire delays (confirmed via a repro test: up to 8.5s per acquire, 190+s total for 50 offers). Split import_offer into a thin wrapper (opens/commits its own transaction, unchanged behavior for standalone imports) and import_offer_into, which writes into a caller-supplied transaction. make_offers now builds and signs all offers first, then imports every auto-imported offer through one shared transaction. Confirmed via the same repro test: no more slow-acquire warnings, 50 offers in ~13s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Speeds up the bulk "one offer per NFT" flow (Offers → New Offer, NFT-splitting enabled), which previously created offers one at a time from a sequential frontend loop, each paying two avoidable costs on the backend.
make_offercalledKeychain::extract_secrets(Argon2, deliberately slow/memory-hard) once per offer. Addedmake_offers, which decrypts the key once and reuses it for every offer in the batch — mirrors the existingbulk_mint_nftspattern.make_offeris now a one-line special case ofmake_offers(single item in, single item out), so there's no duplicated key-extraction logic between them.import_offerfetched each NFT's image/metadata over the network on every import, computed anExtractedNftDatafrom it, and threw the result away (git log -Lconfirms it's been discarded, unused, since this code was introduced — not a regression). Removed both occurrences (offered and requested NFTs) along with the now-unused imports.useOfferProcessor.ts) now callscommands.makeOffersonce instead of loopingcommands.makeOfferper NFT. The progress dialog's creation-phase message dropped the live "N of 50" counter (now a plain spinner) since creation is a single batched call; the marketplace-upload phase's counter is unchanged.Profiling
Built a standalone benchmark exercising the actual production code (
sage-keychain::Keychain::extract_secretswith real Argon2 params, andsage-assets::fetch_uris_with_hash— the exact function the deleted code called, including the real MintGarden testnet thumbnail lookup) to measure the two removed costs directly, for a 50-offer batch:import_offerTwo runs, consistent within normal network variance. This isolates the waste removed by this PR — it doesn't include the unchanged per-offer cost (coin selection, spend-bundle construction, signing, DB writes), which still runs once per offer either way. So this is a floor, not the full before/after wall-clock time for the whole dialog, but it's a substantial one: for a 50-offer batch, roughly 22 of however-many-seconds it took before was pure overhead that's now gone.
Regression found and fixed during manual testing
Manual testing against a real testnet wallet (50 NFTs, split on, each offer requesting 123 XCH, fee 0, 1-day expiry) surfaced a real bug in the initial version of this PR:
make_offerscalledimport_offeronce per offer (auto_importdefaults to true), and eachimport_offeropened and committed its own SQLite write transaction. Firing 50 of these back-to-back with no pacing collides with background writers (the sync manager processing recent coin updates, the transaction queue, etc.) under SQLite's single-writer lock, causing escalating connection-acquire delays.Reproduced with a test against the real
Sage/RPC layer (real DID + 50 minted NFTs + the exact scenario above): connection-acquire times climbed from 2.0s to 8.5s over the course of the batch, and the whole call took 190+ seconds. Bumping the SQLite pool'smax_connectionsmade it worse, confirming the bottleneck is SQLite's single-writer serialization, not pool size.Fix: split
import_offerinto a thin wrapper (opens/commits its own transaction — unchanged behavior for standalone offer imports) andimport_offer_into, which writes into a caller-supplied transaction.make_offersnow builds and signs all offers first, then imports every auto-imported offer through one shared transaction instead of N separate ones. Re-ran the same repro: no more slow-acquire warnings, 50 offers in ~13s.Test plan
cargo check/cargo clippy --workspace --all-targets(clean)tsc -b/eslinton touched frontend files (clean)commands.makeOfferspresent insrc/bindings.ts)make_offerstill behaves identically after theimport_offerrefactor