Skip to content

Speed up bulk NFT offer creation - #877

Draft
judeallred wants to merge 2 commits into
xch-dev:mainfrom
judeallred:worktree-bulk-offer-creation-perf
Draft

Speed up bulk NFT offer creation#877
judeallred wants to merge 2 commits into
xch-dev:mainfrom
judeallred:worktree-bulk-offer-creation-perf

Conversation

@judeallred

@judeallred judeallred commented Sep 10, 2026

Copy link
Copy Markdown

Summary

Speeds up the bulk "one offer per NFT" flow (Offers → New Offer, NFT-splitting enabled), which previously created offers one at a time from a sequential frontend loop, each paying two avoidable costs on the backend.

  • Batch the signing key decryption. make_offer called Keychain::extract_secrets (Argon2, deliberately slow/memory-hard) once per offer. Added make_offers, which decrypts the key once and reuses it for every offer in the batch — mirrors the existing bulk_mint_nfts pattern. make_offer is now a one-line special case of make_offers (single item in, single item out), so there's no duplicated key-extraction logic between them.
  • Delete a dead network fetch. import_offer fetched each NFT's image/metadata over the network on every import, computed an ExtractedNftData from it, and threw the result away (git log -L confirms it's been discarded, unused, since this code was introduced — not a regression). Removed both occurrences (offered and requested NFTs) along with the now-unused imports.
  • Frontend (useOfferProcessor.ts) now calls commands.makeOffers once instead of looping commands.makeOffer per NFT. The progress dialog's creation-phase message dropped the live "N of 50" counter (now a plain spinner) since creation is a single batched call; the marketplace-upload phase's counter is unchanged.

Profiling

Built a standalone benchmark exercising the actual production code (sage-keychain::Keychain::extract_secrets with real Argon2 params, and sage-assets::fetch_uris_with_hash — the exact function the deleted code called, including the real MintGarden testnet thumbnail lookup) to measure the two removed costs directly, for a 50-offer batch:

Before (50 offers) After (50 offers) Speedup
Argon2 key decryption ~0.8–1.0s (50× Argon2) ~16–31ms (1× Argon2) ~30–48x
Dead NFT-data fetch in import_offer ~21.3–21.6s (100 wasted fetches, ~430ms/NFT) 0s (code deleted)
Combined overhead removed ~22–22.6s ~0.02–0.03s

Two runs, consistent within normal network variance. This isolates the waste removed by this PR — it doesn't include the unchanged per-offer cost (coin selection, spend-bundle construction, signing, DB writes), which still runs once per offer either way. So this is a floor, not the full before/after wall-clock time for the whole dialog, but it's a substantial one: for a 50-offer batch, roughly 22 of however-many-seconds it took before was pure overhead that's now gone.

Regression found and fixed during manual testing

Manual testing against a real testnet wallet (50 NFTs, split on, each offer requesting 123 XCH, fee 0, 1-day expiry) surfaced a real bug in the initial version of this PR: make_offers called import_offer once per offer (auto_import defaults to true), and each import_offer opened and committed its own SQLite write transaction. Firing 50 of these back-to-back with no pacing collides with background writers (the sync manager processing recent coin updates, the transaction queue, etc.) under SQLite's single-writer lock, causing escalating connection-acquire delays.

Reproduced with a test against the real Sage/RPC layer (real DID + 50 minted NFTs + the exact scenario above): connection-acquire times climbed from 2.0s to 8.5s over the course of the batch, and the whole call took 190+ seconds. Bumping the SQLite pool's max_connections made it worse, confirming the bottleneck is SQLite's single-writer serialization, not pool size.

Fix: split import_offer into a thin wrapper (opens/commits its own transaction — unchanged behavior for standalone offer imports) and import_offer_into, which writes into a caller-supplied transaction. make_offers now builds and signs all offers first, then imports every auto-imported offer through one shared transaction instead of N separate ones. Re-ran the same repro: no more slow-acquire warnings, 50 offers in ~13s.

Test plan

  • cargo check/cargo clippy --workspace --all-targets (clean)
  • tsc -b / eslint on touched frontend files (clean)
  • Bindings regenerated (commands.makeOffers present in src/bindings.ts)
  • Manual test: bulk-create 50 offers on testnet with NFT-splitting on, each requesting XCH — reproduced a DB write-contention regression, fixed it, and confirmed the fix (see above)
  • Exercise the single-offer (non-split) path once more to confirm make_offer still behaves identically after the import_offer refactor

Add make_offers to decrypt the signing key once per batch instead of
once per offer, and delegate make_offer to it as the single-item case.
Remove a dead NFT data fetch in import_offer whose result was never
read. Update the bulk offer-creation UI to call make_offers once
instead of looping make_offer per NFT, and drop the now-inaccurate
per-offer progress counter.
make_offers previously called import_offer once per offer, each
opening and committing its own SQLite write transaction. In a 50-offer
batch this fires 50 write transactions back-to-back with no pacing,
which collides with background writers (sync manager, transaction
queue) under SQLite's single-writer lock and causes escalating
connection-acquire delays (confirmed via a repro test: up to 8.5s per
acquire, 190+s total for 50 offers).

Split import_offer into a thin wrapper (opens/commits its own
transaction, unchanged behavior for standalone imports) and
import_offer_into, which writes into a caller-supplied transaction.
make_offers now builds and signs all offers first, then imports every
auto-imported offer through one shared transaction. Confirmed via the
same repro test: no more slow-acquire warnings, 50 offers in ~13s.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant