Skip to content

fix(agents): run a schedule that fell behind once, not once per missed occurrence - #334

Open
Lob26 wants to merge 2 commits into
theam:mainfrom
Lob26:fix/scheduler-coalesce-backlog
Open

Lob26 wants to merge 2 commits into
theam:mainfrom
Lob26:fix/scheduler-coalesce-backlog

Conversation

@Lob26

@Lob26 Lob26 commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

What changes

A schedule that fell behind runs once when the worker comes back, instead of
once per missed occurrence.

AgentScheduler.claim advanced a due schedule by exactly one cron step from its
own stored nextRunAt. The row therefore stayed due after the claim, and tick
— which the worker runs on * * * * * — dispatched again the next minute, and
the minute after, until the backlog drained. The next occurrence is now computed
from now.

tick also returns coalesced: the number of claims that absorbed at least one
further occurrence.

Why

Every one of those replays is a real stories.start — a workspace provisioned
and an engine run charged to the project budget. An hourly schedule after a day
of worker downtime is 24 paid runs in 24 minutes, against a budget sized for
one an hour, in a window narrow enough that each sees the last as still in
flight. A five-minute schedule is 288. Nothing bounded it: the replay was as long
as the outage. Downtime is not exotic — tsx watch restarts the worker on every
file change.

The replayed occurrences are also stale by construction. A security audit due
05:00 Monday and run 14:32 Tuesday is not the audit anyone asked for; it is the
same audit against a repository that has moved on, and the work it was meant to
precede already happened.

Computing from now makes the loop level-triggered: it converges on "this
schedule is due" and satisfies it once, however long the gap, rather than
replaying every edge it missed. A schedule claimed on time is unaffected, because
the next occurrence after now and the next after its own due instant are the
same one.

due counts rows, so a 24-occurrence catch-up and an ordinary minute were
indistinguishable in the worker log. coalesced costs one extra cron step per
due schedule — never a walk over the backlog — so an outage of any length is the
same work.

What is deliberately untouched. The claim stays a compare-and-swap on
(nextRunAt, lastScheduledAt), so two workers still cannot take the same
occurrence. The dispatch still identifies itself by the occurrence it satisfies:
lastScheduledAt, the message dedupe key and trigger.scheduledFor all keep the
observed due instant rather than the wall clock, which is what keeps the dispatch
idempotent.

#330 offered three shapes — coalesce, bound the catch-up the way a Kubernetes
CronJob uses startingDeadlineSeconds, or make it the manifest's choice. This
implements coalescing, which is what every agent kickstart ships wants. It is
here as a concrete proposal, cheap to reject: if bounded catch-up is the intended
contract instead, the semantics live in one pure function and the test matrix
flips with it.

Verification

The decision is extracted as a pure function, scheduleAdvance, so the semantics
are testable against a fixed clock with no database.

Unit — 3 of the 5 new cases are red against the previous behaviour, and the
on-time and clock-skew cases are green both ways because they must not regress:

× collapses a backlog of any size into a single claim
  AssertionError: expected 2026-09-06T07:00:00.000Z to deeply equal 2026-09-07T07:00:00.000Z
× reports a single missed occurrence, not only a long outage
  AssertionError: expected 2026-09-07T06:05:00.000Z to deeply equal 2026-09-07T06:10:00.000Z
× keeps the trigger timezone when catching up across a daylight-saving shift
  AssertionError: expected 2026-03-07T07:00:00.000Z to deeply equal 2026-03-10T06:00:00.000Z

Integration — security-audit:nightly put a week in arrears, then three ticks at
the same instant. Without the fix:

× runs a schedule that fell behind once, not once per missed occurrence
  AssertionError: expected 3 to be 1

Three ticks, three paid runs. With the fix, one run, one new story message, and
the schedule left ahead of the clock at 2026-01-11T02:00:00Z.

vitest run test/agent-scheduler.test.ts test/agent-automation.integration.test.ts
                                                        21 passed
tsc --noEmit (@facility/api)                            clean
biome check (changed files)                             clean
node guards/run.mjs                                     2 guards, 0 failed

The integration test asserts this project's own rows, not the tick counters:
agent_schedules is global and the shared test database accumulates rows across
suites, so a counter assertion there would pass or fail on history. I hit exactly
that while writing it. scheduled is safe because the fixture's catalog throws
for any other project, and the run was repeated three times consecutively to
confirm it.

  • pnpm verify passes locally
  • Behaviour verified beyond the test suite (say how)
  • Documentation updated, or no user-facing change

pnpm verify does not pass on this machine and not because of this change:
test:dev reports 116 tests, 92 pass, 24 fail here, and I measured the
identical 92/24 on a clean main — Windows noise in the patched image-size
cases, tar failing to resolve C:, and the registry publication tests. Beyond
the suite, the failure was reproduced end to end against a real PostgreSQL: three
ticks producing three dispatches before the change and one after.

Closes #330.

🤖 Claude Code helped

…d occurrence

`AgentScheduler.claim` advanced a due schedule by exactly one cron step
from its own stored `nextRunAt`. The row therefore stayed due after the
claim, and `tick` — which the worker runs on `* * * * *` — dispatched
again on the next minute, and the minute after that, until the backlog
drained. Every one of those dispatches is a real `stories.start`: a
workspace provisioned and an engine run charged to the project budget.

An hourly schedule after a day of worker downtime is 24 paid runs in 24
minutes, against a budget sized for one an hour, in a window narrow
enough that each one sees the last as still in flight. A five-minute
schedule is 288. Nothing bounded it; the replay was as long as the outage.
Downtime is not exotic: `tsx watch` restarts the worker on every file
change.

The occurrences being replayed are stale by construction. A security
audit due 05:00 Monday and run 14:32 Tuesday is not the audit anyone
asked for — it is the same audit against a repository that has moved on,
and the work it was meant to precede already happened.

Compute the next occurrence from `now` instead. That makes the loop
level-triggered: it converges on "this schedule is due" and satisfies it
once, however long the gap, rather than replaying every edge it missed. A
schedule claimed on time is unaffected, because the next occurrence after
`now` and the next after its own due instant are the same one.

The decision is a pure function, `scheduleAdvance`, so the semantics are
testable against a fixed clock with no database: on-time claims land
exactly where they used to, a week-long backlog collapses to one claim, a
single missed period is still reported as coalesced, catching up across a
daylight-saving shift keeps the trigger timezone, and a clock behind the
due instant never moves a schedule backwards.

What was already right is untouched: the claim remains a compare-and-swap
on `(nextRunAt, lastScheduledAt)`, so two workers still cannot take the
same occurrence, and the dispatch still identifies itself by the
occurrence it satisfies — `lastScheduledAt`, the message dedupe key and
`trigger.scheduledFor` all keep the observed due instant rather than the
wall clock.

`tick` now also returns `coalesced`, the number of claims that absorbed at
least one further occurrence. `due` counts rows, so a 24-occurrence
catch-up and an ordinary minute were indistinguishable in the worker log;
they no longer are. It costs one extra cron step per due schedule, never a
walk over the backlog, so an outage of any length is the same work.

Closes theam#330.

@adrian-lorenzo adrian-lorenzo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A scheduled agent replays every occurrence it missed while the worker was down, one paid run per tick

2 participants