Skip to content

Make pr_labeler safe for concurrent runs - #25

Merged
bryanbeverly merged 2 commits into
mainfrom
fix/pr-labeler-sticky-comment-idempotent
Aug 3, 2026
Merged

bryanbeverly merged 2 commits into
mainfrom
fix/pr-labeler-sticky-comment-idempotent

Conversation

@bryanbeverly

Copy link
Copy Markdown
Contributor

Why

Two labeler runs can overlap on the same PR today. The caller's concurrency group is:

group: pr-labeler-${{ github.event.pull_request.number || github.run_id }}

A workflow_dispatch backfill has no pull_request.number, so it falls back to run_id — a group of one. The documented way to label a stacked PR (gh workflow run pr-labeler.yml -f pr_number=N) can therefore run alongside an event-driven run on the same PR right now. Both bugs below are reachable without changing anything.

They also block a fix we need. Cancelling a run leaves a permanent cancelled check run on the head commit; Renovate reads every check run without discarding superseded ones (renovatebot/renovate#36837), computes a red branch status, and silently declines to automerge. Measured on one unchanged commit in truffle-release-bot#29 — five edited events from Renovate rewriting the PR body, four cancelled:

00:42:39 cancelled   00:42:43 cancelled   00:42:47 cancelled
00:42:51 cancelled   00:42:55 success (ran 22s)

cancel-in-progress: false doesn't help (only one run may be pending per group, and triggers arrive every ~4s against a ~22s job), and dropping edited isn't available (it's what applies the risk label after Bugbot edits the description).

Two fixes

1. The sticky comment converges on one copy. It's created only when no copy is found and updated thereafter, but find-then-create isn't atomic: two runs can both see "no comment yet" and both create one. Only the lowest-id copy is ever found again, so the other is orphaned permanently, showing reviewers the CODEOWNERS team list twice. The survivor is now chosen by lowest comment id and the rest are deleted — once from the listing already fetched (free, and it cleans up copies left behind before this change) and again on fresh data after a create. Choosing by id rather than by who arrived first is what makes it lock-free: concurrent runs pick the same survivor and converge instead of deleting each other's. The loser gets a 404, so deletes pass check=False.

2. A run whose inputs moved stands down. A run snapshots the PR at start and applies ~20s later. Task-list checkboxes save on click, so ticking Urgent then High complexity fires two edited events seconds apart. Run A reads the unticked body and plans a removal; run B reads it ticked but skips its add because the label is still present; A then lands the removal, and nothing re-runs to correct it. The fix re-reads the body and head sha immediately before applying and skips when either moved.

Only inputs are compared — body for risk and template labels, head sha for size and domain. Applying labels or the comment changes neither, so a run whose inputs held still always proceeds and the newest run can't be starved.

What this unlocks

Fix 2 is the ordering cancel-in-progress was providing, moved into the script. The workflow achieved it by killing the older run and leaving a red check run behind; the script now reaches the same outcome with the run exiting green. The concurrency group stops being load-bearing, so a follow-up can delete it from the callers and unblock Renovate automerge. No caller changes here.

Risk

The common single-run path is unchanged: the prune is a no-op with no API calls, and the guard costs one gh pr view per PR before applying. Dry runs skip the guard entirely (nothing to stand down from) and count duplicates without deleting.

Test plan

  • python -m pytest .github/scripts — 148 passed (11 new)
  • ruff check and ruff format --check clean
  • Dedupe coverage: survivor stable regardless of listing order, all-but-survivor deleted, single copy untouched, delete tolerates an already-removed copy, dry run counts without deleting
  • Guard coverage: unchanged inputs proceed, edited body stands down, new head commit stands down, null vs empty body is not a change
  • After merge: confirm on a live PR with CODEOWNERS (topo) that the comment still renders once and labels still apply
  • Then delete the concurrency block in the three callers and confirm Renovate automerges

Made with Cursor

bryanbeverly and others added 2 commits August 2, 2026 22:06
The domain-reasons comment is created only when no copy is found and updated
thereafter, but that find-then-create pair is not atomic. Two labeler runs
overlapping on the same PR can each observe "no comment yet" and both create
one. Only the first is ever found again, so the second is orphaned on the PR
permanently, showing reviewers a duplicate list of CODEOWNERS-derived teams.

Choose the surviving copy by lowest comment id and delete the rest: once from
the listing already fetched, which also cleans up duplicates left behind
before this change, and again on a fresh listing after a create. Picking the
survivor deterministically rather than by who arrives first means concurrent
runs select the same copy and converge instead of deleting each other's, so
no lock is needed; deletes tolerate the 404 from losing that race.

This clears the way to drop cancel-in-progress from the caller workflows,
where cancelling a run leaves a permanent cancelled check run on the head
commit that Renovate reads as a red branch status and silently refuses to
automerge (renovatebot/renovate#36837).

Co-authored-by: Cursor <cursoragent@cursor.com>
Every run snapshots the PR when it starts and applies its decisions ~20s
later, so two runs overlapping on one PR can apply stale conclusions. The
reachable case is ordinary: task-list checkboxes save on click, so ticking
"Urgent" and then "High complexity" fires two edited events seconds apart.
The first run read the unticked body and plans to remove the label; the
second reads it ticked but skips its add because the label is still present;
the first then lands its removal and the label is wrong, with no further
event to correct it.

Re-read the body and head sha immediately before applying and skip the run
when either moved. Only inputs are compared -- the body decides risk and
template labels, the head sha decides size and domain -- and writing labels
or the sticky comment changes neither, so a run whose inputs held still
always proceeds and the newest run is never starved.

This is the ordering that cancel-in-progress was providing, moved into the
script. The workflow got it by killing the older run, which leaves a
cancelled check run on the head commit that Renovate reads as a red branch
status and refuses to automerge behind (renovatebot/renovate#36837);
standing down here reaches the same outcome with the run finishing green,
so the concurrency group is no longer load-bearing.

Co-authored-by: Cursor <cursoragent@cursor.com>
@bryanbeverly
bryanbeverly requested a review from a team August 3, 2026 05:37
@bryanbeverly
bryanbeverly merged commit 6461ff0 into main Aug 3, 2026
4 checks passed
@bryanbeverly
bryanbeverly deleted the fix/pr-labeler-sticky-comment-idempotent branch August 3, 2026 19:00
bryanbeverly added a commit to trufflesecurity/helm-charts that referenced this pull request Sep 14, 2026
#39)

## Why

`cancel-in-progress: true` in this workflow makes Renovate automerge
fail silently. Renovate rewrites a PR body several times in the seconds
after opening it, each rewrite fires `edited`, and the cancellation
leaves a **permanent** `cancelled` check run named `label / label` on
the head commit. Renovate reads every check run on the head without
discarding superseded ones
([renovatebot/renovate#36837](renovatebot/renovate#36837)),
so it computes a red branch status and declines to merge. GitHub still
reports `mergeStateStatus: CLEAN`, because `label / label` is not a
required check, so the PR looks mergeable to a human and only Renovate
disagrees.

## What changed

Deleted the `concurrency` block. The "latest run wins" ordering it
provided now lives in `pr_labeler.py`
([trufflesecurity/.github#25](trufflesecurity/.github#25)),
which dedupes the sticky comment deterministically and stands down when
the PR body or head sha moved under it. Same ordering guarantee, no red
check left behind. That script change is already live and shared by
every caller.

The two explanatory comments are added as well, so this file is now
byte-identical to the caller in topo, hoglet-hub and
truffle-release-bot.

## Evidence

The same change merged in topo, hoglet-hub and truffle-release-bot in
early August. Since 4 Aug, Renovate has merged its own PRs **50 times**
across those three: 23 of 29 in topo, 23 of 25 in hoglet-hub, 4 of 4 in
truffle-release-bot. Before the change, routine automerge had never once
fired in any of them.

The live counter-example is `thog#7394`, a `[SECURITY]` update the
preset marks `automerge: true`. It is `mergeStateStatus: CLEAN`, it has
an approval, all 30 real checks pass, its body reads `🚦 Automerge:
Enabled.` with no schedule restriction, and a manual Renovate run
declined to merge it. The only anomaly on that head commit is three
cancelled `label / label` runs.

## Note on existing PRs

Check runs are immutable and bound to a commit, so merging this does not
clear cancelled runs already sitting on open PR heads. Those PRs need a
rebase to pick up a clean check-run set.

## Risk

Low, and CI-only. Worst case is two labeler runs overlapping, which
`.github#25` already handles: it picks the lowest-id sticky comment as
the survivor so concurrent runs converge on the same choice, and it
skips label application when its inputs are stale.

Made with [Cursor](https://cursor.com)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants