feat(consumers): Auto-restart on single-partition lag - #8386
Conversation
Add an option-gated watchdog in the Snuba rust healthcheck that tracks submit vs commit progress per partition. When a partition has in-flight work without commit progress past the timeout, stop touching the health file so the k8s liveness probe restarts the pod and Kafka rebalances. Co-Authored-By: PDPM <pierre.massat@sentry.io>
Extend the stall watchdog so it also fails health when one assigned partition's commit rate falls far below the median sibling rate while still receiving work. That covers the single-partition lag case where offsets still move slowly and a hard commit stall never fires. Co-Authored-By: PDPM <pierre.massat@sentry.io> Co-Authored-By: PDPM <pierre.massat@sentry.io>
sentry-junior
Bot
changed the title
Rename experimental_healthcheck to consumer.commit_progress_healthcheck so the name matches consumer-level commit/idle progress checks. Keep the old key as a legacy alias. Document strategy and progress modes in the consumer architecture docs. Co-Authored-By: PDPM <pierre.massat@sentry.io>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Compare each partition to the median of the other active partitions on the assignment. An inclusive median diluted the slow partition on small assignments, so the default 0.25 ratio missed the two-partition collapse case. Co-Authored-By: PDPM <pierre.massat@sentry.io>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 9abdb3a. Configure here.
Keep leave-one-out sibling medians for the ratio comparison, but gate the whole assignment on an inclusive median quiet floor so one hot partition cannot fail mostly-quiet peers. Also satisfy clippy::question_mark on the slow-ratio option helper. Co-Authored-By: PDPM <pierre.massat@sentry.io>
There was a problem hiding this comment.
I think this might be a bad idea without more safeguards. It has potential to cause a negative feedback loop.
We've seen in backlog situations before that when a consumer is successfully processing multiple partitions of a topic (after a period of not-processing) where all partitions are quite behind that librdkafa and the broker will queue messages for processing based on partition ordering, meaning some partitions will not make progress for a while until the "front-most" partition drops below a certain threshold where the buffer will receive messages from a wider set. If that happens with this change enabled, you'll force more rebalancing and slow down recovery; a similar pattern to OOMing when we hit a backlog
The key is unused in sentry-options-automator, so keep only consumer.commit_progress_healthcheck.
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
Select one file owner with --health-check (arroyo, commit-progress, partition-stall). Drop option-gated modes and relative slowdown. consumer.partition_stall_timeout_secs is only the stall timeout.
Nothing in ops or tests passed --health-check snuba; keep only arroyo, commit-progress, and partition-stall.
Drop redundant health-file state, saturate idle-poll counters, and use checked SystemTime adds so long-running consumers cannot panic on overflow.
Track stable there for CI, Docker, and local cargo. Drop the extra rustup install step and Makefile +stable fmt override. Satisfy current-stable clippy.
The CLI already advertised --health-check and --health-check-file, but the Rust factory ignored them. Wrap the pipeline with the same strategies as rust-consumer so Kubernetes probes actually work.
|
@onewland relative throughput comparison is gone for exactly that reason.
During backlog recovery, librdkafka starving siblings should not trip this: those partitions never get submits, so they are not in-flight. A restart still only helps if the owning member is the problem.
|
Touching ci.yml runs validate-devservices-config, which currently fails in teardown (empty CODEROOT). Keep the toolchain pin in rust-toolchain.toml instead.
validate-devservices-config is gated on ci.yml changes and currently fails in teardown. Drop the workflow edit so that job does not run.
3 participants
--health-check partition-stallfails the Kubernetes health file when any assigned partition has in-flight work with no commit progress forconsumer.partition_stall_timeout_secs(default 300s;0disables detection). The pod restarts and Kafka rebalances.--health-checkselects one file owner:arroyo(default): touch on every successful pollcommit-progress: touch on commit or idle (consumer-level; one stuck partition can stay healthy if siblings still commit)partition-stall: per-partition stall as aboveRelative throughput comparison was dropped. During backlog recovery, librdkafka can starve sibling partitions, which would look like slowdown and force extra rebalances. This check only trips on stuck in-flight work, not on partitions that are not being delivered.
experimental_healthcheckand--health-check snubaare gone. Python consumers are unchanged (arroyo poll-only file touch). Requires--health-check-file.Requested by PDPM.
--
View Junior Session in Sentry