feat: keep retrying cancels until the chain confirms them by MoonBoi9001 · Pull Request #716 · graphprotocol/dipper

MoonBoi9001 · 2026-10-02T15:34:15Z

Dipper now marks an agreement it wants ended as Cancelling before sending its on-chain cancel, and the chain listener retries the cancel until the chain shows it ended, alerting after 10 failed attempts. Only then is it marked cancelled and its end announced. It also fixes the remaining findings from the review of #715.

Warning

Before rolling back below this version, move every Cancelling agreement (status 9) to another status: older builds can't read it.

Offers and cancels share 1 wallet, so marking an agreement Cancelling before its cancel goes out lets a late offer withdraw itself. The chain listener retries the cancel until the chain shows the end, with an ERROR after 10 failed attempts; only then is the end announced.

The offer job's withdraw and the on-chain cancel job each read the chain and cancelled only a live agreement, with their own copy of that logic; both now use the helper the retry uses.

After its offer lands, the offer job rereads the agreement to withdraw the offer if it was cancelled meanwhile. A failed reread finished the job, leaving such an offer open; it retries.

Accepts dipper missed were announced only if it cancelled within 1 hour of the offer deadline, so a lagging listener dropped them. Only agreements created before events existed are skipped.

6 test modules each spelled out the whole agreement config, so every new setting meant editing 6 copies. They now start from a shared test helper and override what they need.

The subgraph reports a withdrawn offer as cancelled by the payer with an accept time of 0. Recording that as an accept announced an agreement that was never live.

The cancel retry reads the chain ahead of the listener, so marking an ended agreement there lost who ended it and when; it now waits for the listener, which records both from the chain.

Each sweep took the 100 oldest cancelling agreements, so a backlog stalled the listener and hid newer ones, and could resend a cancel still being mined. It now takes 10, least recently checked first, and skips any marked in the last 2 minutes.

A misconfigured or paused manager failed every agreement's cancel, using up their attempts so none was retried once fixed. Only a cancel mined without ending the agreement now counts, and an agreement with no stored terms hash, which can never be cancelled, is given up at once.

Retrying it sent the offer again. The chain listener's cancel retry already withdraws the offer of an agreement left cancelling, so the job now logs the failed read and finishes.

Agreements still cancelling kept the listener at its fast poll rate so their retry ran often, and one given up on kept it there for good. The retry now runs every 5 minutes at any rate.

An agreement dipper is cancelling is paid until the cancel lands, so it now stays in the fee estimates the indexer selection service uses to compare indexers.

A lagging listener can mark an agreement expired that was in fact accepted. Such an agreement couldn't be marked cancelling, so its replacement's acceptance no longer cancelled it.

The sweep for accepted agreements left behind when a request was cancelled resent their cancel on every run with no limit. It now marks each one cancelling, and the cancel retry finishes it.

The production registry trait gave the retry's 2 queries do-nothing defaults, so a registry that forgot them would silently never retry a cancel; only the test stub keeps defaults now.

The cancelling agreements query returned each row's failed cancel count, which no caller reads.

A failed read was treated like a clean check, so an offer past its deadline was marked ended without knowing it wasn't live, as an accept the listener hadn't yet recorded would be.

Agreements created before dipper announced lifecycle events are never announced, but one being cancelled had its accept recorded from the chain, which then announced it.

Every replaced agreement marked expired was relabelled cancelled, losing its expired event and the record that the indexer let the offer lapse. Only one the chain shows live is cancelled.

A mined cancel that reverted was retried every 5 minutes without limit, costing gas each time. It now counts towards the limit; one that reverts before sending is retried, logged as an ERROR.

The retry left an accepted agreement that already ended for the chain listener to confirm, so one whose end the listener never read stayed cancelling for good. After an hour it marks it.

Each retried cancel can wait 15 seconds to be mined, and the sweep holds up the chain listener, so 10 slow ones stalled it for minutes. A sweep now leaves what it hasn't reached to the next.

A reassessment no longer queues it when its own cancel fails; the cancel retry handles that.

MoonBoi9001 added 24 commits October 2, 2026 18:29

refactor(worker): share the read-then-cancel step between cancel paths

443641e

The offer job's withdraw and the on-chain cancel job each read the chain and cancelled only a live agreement, with their own copy of that logic; both now use the helper the retry uses.

fix(worker): retry an offer job that can't recheck its agreement

3456774

After its offer lands, the offer job rereads the agreement to withdraw the offer if it was cancelled meanwhile. A failed reread finished the job, leaving such an offer open; it retries.

fix(listener): announce a missed accept however late it is read

e2a82b9

Accepts dipper missed were announced only if it cancelled within 1 hour of the offer deadline, so a lagging listener dropped them. Only agreements created before events existed are skipped.

test(config): build every test's agreement config from one helper

103a974

6 test modules each spelled out the whole agreement config, so every new setting meant editing 6 copies. They now start from a shared test helper and override what they need.

docs(worker): write job counts as numerals in the cancel job's comments

075f8d8

fix(listener): record no accept for a withdrawn offer being cancelled

3374a09

The subgraph reports a withdrawn offer as cancelled by the payer with an accept time of 0. Recording that as an accept announced an agreement that was never live.

fix(listener): let the listener confirm an accepted agreement that ended

ac6470f

The cancel retry reads the chain ahead of the listener, so marking an ended agreement there lost who ended it and when; it now waits for the listener, which records both from the chain.

fix(worker): finish an offer job whose recheck fails without resending

340e399

Retrying it sent the offer again. The chain listener's cancel retry already withdraws the offer of an agreement left cancelling, so the job now logs the failed read and finishes.

fix(listener): retry cancels on a timer, not by keeping polls fast

b8daeac

Agreements still cancelling kept the listener at its fast poll rate so their retry ran often, and one given up on kept it there for good. The retry now runs every 5 minutes at any rate.

fix(registry): count fees of agreements still being cancelled

556d300

An agreement dipper is cancelling is paid until the cancel lands, so it now stays in the fee estimates the indexer selection service uses to compare indexers.

fix(registry): cancel a replaced agreement marked expired as well

0de1b2b

A lagging listener can mark an agreement expired that was in fact accepted. Such an agreement couldn't be marked cancelling, so its replacement's acceptance no longer cancelled it.

fix(listener): start orphan cancels like any other, with a retry limit

f106463

The sweep for accepted agreements left behind when a request was cancelled resent their cancel on every run with no limit. It now marks each one cancelling, and the cancel retry finishes it.

refactor(registry): require every registry to implement the cancel retry

ddb651c

The production registry trait gave the retry's 2 queries do-nothing defaults, so a registry that forgot them would silently never retry a cancel; only the test stub keeps defaults now.

refactor(registry): stop reading a cancel count nothing uses

2774efb

The cancelling agreements query returned each row's failed cancel count, which no caller reads.

fix(listener): never confirm a cancel the chain could not be read for

f7632df

A failed read was treated like a clean check, so an offer past its deadline was marked ended without knowing it wasn't live, as an accept the listener hadn't yet recorded would be.

fix(listener): record no accept of an old agreement being cancelled

8c8c78a

Agreements created before dipper announced lifecycle events are never announced, but one being cancelled had its accept recorded from the chain, which then announced it.

fix(listener): cancel a replaced expired agreement only if it is live

3ad96af

Every replaced agreement marked expired was relabelled cancelled, losing its expired event and the record that the indexer let the offer lapse. Only one the chain shows live is cancelled.

fix(listener): limit retries of a cancel that is mined and reverts

735ea06

A mined cancel that reverted was retried every 5 minutes without limit, costing gas each time. It now counts towards the limit; one that reverts before sending is retried, logged as an ERROR.

fix(listener): mark an ended accepted agreement after an hour's wait

280f0c1

The retry left an accepted agreement that already ended for the chain listener to confirm, so one whose end the listener never read stayed cancelling for good. After an hour it marks it.

fix(listener): stop a cancel retry sweep after 30 seconds

c3e5bda

Each retried cancel can wait 15 seconds to be mined, and the sweep holds up the chain listener, so 10 slow ones stalled it for minutes. A sweep now leaves what it hasn't reached to the next.

docs(worker): say only the chain listener queues the on-chain cancel job

7d662d0

A reassessment no longer queues it when its own cancel fails; the cancel retry handles that.

MoonBoi9001 added this pull request to stack #717 October 2, 2026 17:05

MoonBoi9001 marked this pull request as ready for review October 2, 2026 17:56

MoonBoi9001 removed this pull request from stack #717 October 2, 2026 17:57

MoonBoi9001 merged commit 884e33d into mb9/review Oct 2, 2026
12 checks passed

Navigation Menu

feat: keep retrying cancels until the chain confirms them - #716

feat: keep retrying cancels until the chain confirms them#716
MoonBoi9001 merged 24 commits into
mb9/reviewfrom
mb9/keep-retrying-cancels-until-the-chain-confirms

MoonBoi9001 commented Oct 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

MoonBoi9001 commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

MoonBoi9001 commented Oct 2, 2026 •

edited

Loading