Skip to content

Let salt-proxy finish its graceful shutdown (#70217) - #70218

Open
ggiesen wants to merge 1 commit into
saltstack:3008.xfrom
ggiesen:fix-proxy-graceful-shutdown
Open

Let salt-proxy finish its graceful shutdown (#70217)#70218
ggiesen wants to merge 1 commit into
saltstack:3008.xfrom
ggiesen:fix-proxy-graceful-shutdown

Conversation

@ggiesen

@ggiesen ggiesen commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Lets salt-proxy actually finish the graceful shutdown it schedules, and stops a thread entry from aborting the teardown.

What issues does this PR fix?

Fixes #70217

Root cause

The proxy daemon exited before its own graceful stop could run. MinionManager.stop() does not shut anything down -- it calls self.io_loop.create_task(self.stop_async(signum, parent_sig_handler)) and returns, and stop_async invokes the parent handler once it has finished. ProxyMinion._handle_signals called stop() and then called the parent handler itself, exiting the process before the io_loop could run the task. Python reports this on every SIGTERM:

RuntimeWarning: coroutine 'MinionManager.stop_async' was never awaited

The 5 second grace period that lets the minion flush its final return messages to the master lives inside stop_async, so those were lost as well. Minion._handle_signals in the same file already only calls the parent handler in its else branch, when there is no stop to defer to; this makes the proxy daemon match.

A threading.Thread entry aborted the teardown. _terminate_subprocess_list read proc.pid inside an except OSError. With multiprocessing: False the SubprocessList entries are threading.Thread objects, which have no pid; AttributeError is not an OSError, so it escaped and aborted the teardown before destroy() ran. _is_process_alive does not filter them out because threading.Thread has a working is_alive(). Entries that cannot be signalled are now skipped and the rest are still handled.

multiprocessing: False is the normal configuration for a proxy driving a real device, since a live NETCONF/SSH session cannot be forked.

Previous Behavior

Every salt-proxy SIGTERM abandoned stop_async, so nothing was torn down and final job returns were never flushed. With multiprocessing disabled, the teardown additionally died on AttributeError before reaching destroy().

New Behavior

The scheduled graceful shutdown runs to completion and then exits the process, and a pid-less entry no longer stops the rest of the teardown.

Merge requirements satisfied?

  • Docs
  • Changelog
  • Tests written/updated

Tests written?

Yes, four, each with its inverse. In tests/pytests/unit/cli/test_daemons_signals.py: the proxy daemon asks for the graceful stop and does not tear the process down underneath it, and it still exits when there is no stop method to defer to. In tests/pytests/unit/test_minion.py: _terminate_subprocess_list tolerates a live threading.Thread entry, and still signals an entry that really is a process, so skipping thread entries cannot turn the helper into a no-op.

Against unmodified 3008.x the thread test fails with exactly AttributeError: 'Thread' object has no attribute 'pid' at salt/minion.py:468.

The signal-handling fix was also verified against a live deltaproxy on 3008.2 (one control proxy, three dummy sub-proxies): before, a SIGTERM produced one never awaited warning; after, zero.

One scoping note for reviewers: _terminate_subprocess_list does not exist in 3008.2, so that half has not shipped in a release yet -- it is a branch-only fix. The signal-handling half is present and reproducible in 3008.2.

Commits signed with GPG?

No

ProxyMinion._handle_signals asked MinionManager to stop and then
immediately called the parent signal handler as well. stop() does not
shut anything down; it schedules stop_async on the io_loop and hands it
the parent handler to invoke once the graceful shutdown has finished. So
the process exited before the io_loop could run the task, and Python
reported "coroutine 'MinionManager.stop_async' was never awaited" on
every SIGTERM. The 5 second grace period that lets the minion flush its
final return messages to the master lives in stop_async, so those were
lost too. Minion._handle_signals already only calls the parent handler
when there is no stop method to defer to; do the same here.

_terminate_subprocess_list also read proc.pid inside an except OSError.
With multiprocessing disabled the entries are threading.Thread objects,
which have no pid, and AttributeError is not an OSError, so it escaped
and aborted the teardown before destroy() ran. Skip entries that cannot
be signalled and carry on with the rest.
@twangboy twangboy added the test:full Run the full test suite label Sep 2, 2026
@twangboy twangboy added this to the Argon v3008.3 milestone Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:full Run the full test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: salt-proxy never completes a graceful shutdown: it exits before stop_async runs, and a thread entry aborts the teardown

2 participants