Harden PR labeler against transient GitHub API failures

[martinlocklear]

Human note - written by Martin, verbatim, not AI-generated:

I've run into this transient failure a couple of times, so I decided to go ahead and harden this to transient network effects.

---

[authored by Cursor]

Summary

• Add classified retry with backoff to gh() so transient GitHub API errors (502/503/504, timeouts, connection resets) retry instead of failing the run.
• Floor the failure-rate gate with MIN_FAILURES_TO_FAIL = 2 so a single blip never fails event-mode labeling, while backfill still catches systemic breakage.
• Cover both with unit tests: retry classification, retry-then-success, non-retryable fast-fail, exhausted retries, and the threshold floor.

Motivation

A pull_request labeler run failed on a one-off HTTP 504. gh() ran each API call once, and the 10% failure gate has zero tolerance with a single target (1/1 = 100%), so a momentary hiccup failed an otherwise-healthy PR check. Retry is the real fix; the floor is the backstop. Merging to main propagates to every repo that calls the reusable labeler workflow at @main.

Tracked in INT-802.

Background: labeler transient failure hardening plan

Test plan

• python -m pytest .github/scripts -v (134 passed)
• ruff check .github/scripts and ruff format --check .github/scripts
• CI checks green

[martinlocklear]

[authored by Cursor]

Conversation summary

This PR came from investigating a PR-labeler run that failed on a transient GitHub HTTP 504.

Plan doc: labeler transient failure hardening

Key decisions

• Retry in the gh() wrapper, classified by stderr signature. One chokepoint covers every API call; a factored _is_retryable_gh_error helper keeps the transient-vs-real distinction unit-testable and re-raises real errors (404, auth, bad input) immediately.
• Floor the failure-rate gate rather than replace it. MIN_FAILURES_TO_FAIL = 2 stops event mode from failing on a single exhausted-retry blip while backfill still fails on 2+ failures over 10%.

Alternatives considered

• Retry only - leaves event mode zero-tolerance if retries ever exhaust.
• Floor only - stops this specific case but leaves every API call one blip from a per-PR error, with no backfill protection.
• Rely on gh's built-in retry - less explicit and not unit-testable; the classified wrapper keeps the logic in code.

[bryanbeverly]

[authored by Cursor]

Conversation summary

This PR came from investigating a PR-labeler run that failed on a transient GitHub HTTP 504.

Plan doc: labeler transient failure hardening

....

I really like this whole block comment. Useful info.

[bryanbeverly]

Never mind, I had a 🤖-driven comment here, but when I wrapped my head about it, I decided it was irrelevant. Ignore me.

[bryanbeverly]

Non-blocking: these are matched as bare substrings anywhere in stderr, so codes like 502/504 and especially eof can false-positive on unrelated text (e.g. a numeric ID or SHA that happens to contain 504). Anchoring to HTTP 50x would prevent the numeric one. For eof, I'm a little more shaky. Maybe "unexpected eof", ": eof".

[bryanbeverly]

Non-blocking: "rate limit" also matches primary rate limits, which reset minutes out. 3 quick retries with a 1-2s backoff won't recover, they'll just delay the inevitable failure. Restricting to "secondary rate limit" (plus the existing "abuse detection") would avoid retrying an unrecoverable condition.

[bryanbeverly]

👍

[martinlocklear]

Should fix the lint failure: #23