Let salt-proxy finish its graceful shutdown (#70217) - #70218
Open
ggiesen wants to merge 1 commit into
Open
Conversation
ProxyMinion._handle_signals asked MinionManager to stop and then immediately called the parent signal handler as well. stop() does not shut anything down; it schedules stop_async on the io_loop and hands it the parent handler to invoke once the graceful shutdown has finished. So the process exited before the io_loop could run the task, and Python reported "coroutine 'MinionManager.stop_async' was never awaited" on every SIGTERM. The 5 second grace period that lets the minion flush its final return messages to the master lives in stop_async, so those were lost too. Minion._handle_signals already only calls the parent handler when there is no stop method to defer to; do the same here. _terminate_subprocess_list also read proc.pid inside an except OSError. With multiprocessing disabled the entries are threading.Thread objects, which have no pid, and AttributeError is not an OSError, so it escaped and aborted the teardown before destroy() ran. Skip entries that cannot be signalled and carry on with the rest.
twangboy
approved these changes
Sep 2, 2026
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Lets
salt-proxyactually finish the graceful shutdown it schedules, and stops a thread entry from aborting the teardown.What issues does this PR fix?
Fixes #70217
Root cause
The proxy daemon exited before its own graceful stop could run.
MinionManager.stop()does not shut anything down -- it callsself.io_loop.create_task(self.stop_async(signum, parent_sig_handler))and returns, andstop_asyncinvokes the parent handler once it has finished.ProxyMinion._handle_signalscalledstop()and then called the parent handler itself, exiting the process before the io_loop could run the task. Python reports this on every SIGTERM:The 5 second grace period that lets the minion flush its final
returnmessages to the master lives insidestop_async, so those were lost as well.Minion._handle_signalsin the same file already only calls the parent handler in itselsebranch, when there is nostopto defer to; this makes the proxy daemon match.A
threading.Threadentry aborted the teardown._terminate_subprocess_listreadproc.pidinside anexcept OSError. Withmultiprocessing: FalsetheSubprocessListentries arethreading.Threadobjects, which have nopid;AttributeErroris not anOSError, so it escaped and aborted the teardown beforedestroy()ran._is_process_alivedoes not filter them out becausethreading.Threadhas a workingis_alive(). Entries that cannot be signalled are now skipped and the rest are still handled.multiprocessing: Falseis the normal configuration for a proxy driving a real device, since a live NETCONF/SSH session cannot be forked.Previous Behavior
Every
salt-proxySIGTERM abandonedstop_async, so nothing was torn down and final job returns were never flushed. Withmultiprocessingdisabled, the teardown additionally died onAttributeErrorbefore reachingdestroy().New Behavior
The scheduled graceful shutdown runs to completion and then exits the process, and a pid-less entry no longer stops the rest of the teardown.
Merge requirements satisfied?
Tests written?
Yes, four, each with its inverse. In
tests/pytests/unit/cli/test_daemons_signals.py: the proxy daemon asks for the graceful stop and does not tear the process down underneath it, and it still exits when there is nostopmethod to defer to. Intests/pytests/unit/test_minion.py:_terminate_subprocess_listtolerates a livethreading.Threadentry, and still signals an entry that really is a process, so skipping thread entries cannot turn the helper into a no-op.Against unmodified 3008.x the thread test fails with exactly
AttributeError: 'Thread' object has no attribute 'pid'atsalt/minion.py:468.The signal-handling fix was also verified against a live deltaproxy on 3008.2 (one control proxy, three
dummysub-proxies): before, a SIGTERM produced onenever awaitedwarning; after, zero.One scoping note for reviewers:
_terminate_subprocess_listdoes not exist in 3008.2, so that half has not shipped in a release yet -- it is a branch-only fix. The signal-handling half is present and reproducible in 3008.2.Commits signed with GPG?
No