Context
The generic compute reconcile-storm alerts currently flag consumer-serviceconsumer because the provider returns a five-minute RequeueAfter for every watched ServiceConsumer. With roughly 540 active consumers, the aggregate rate is about 1.8 reconciles/sec even when the queue stays drained and every reconcile succeeds.
The upstream fix is milo-os/service-catalog#107, which removes that per-object periodic resync.
Follow-up
After #107 merges and a service-catalog release is available:
- bump the service-catalog dependency in compute;
- deploy to staging;
- verify
consumer-serviceconsumer reconcile and queue-add rates fall to event-driven levels;
- remove the temporary alert noise and close the associated alert cleanup work.
Do not exclude the controller from the generic alert before the upstream behavior change is verified; a real event-driven hot loop should remain visible.
Context
The generic compute reconcile-storm alerts currently flag
consumer-serviceconsumerbecause the provider returns a five-minuteRequeueAfterfor every watchedServiceConsumer. With roughly 540 active consumers, the aggregate rate is about 1.8 reconciles/sec even when the queue stays drained and every reconcile succeeds.The upstream fix is milo-os/service-catalog#107, which removes that per-object periodic resync.
Follow-up
After #107 merges and a service-catalog release is available:
consumer-serviceconsumerreconcile and queue-add rates fall to event-driven levels;Do not exclude the controller from the generic alert before the upstream behavior change is verified; a real event-driven hot loop should remain visible.