`bundle install` step is assumed to have succeeded when OOM killed

[schnittchen]

I have a Rails app that when deployed, runs the command

cd /home/app/staging/releases/20250304094541 && /usr/bin/env bundle install --jobs 4 --quiet

on the server. Due to --jobs 4 and the amount of memory available, it happens to be OOM-killed.

SSHKit happily assumes it has succeeded and continues, erroring on a follow-up step.

My suspicion is that this commented-out code points into the direction of a solution: https://github.com/capistrano/sshkit/blob/master/lib/sshkit/backends/netssh.rb#L160-L166

[mattbrictson]

Thanks for reporting this! It looks like this is an edge case that sshkit is not handling.

Ideally, what would you like sshkit to do in this scenario? Just logging that the command was killed is probably insufficient, because that would mean execution would just continue to the next step. I'm guessing you'd like an exception raised in this scenario?

Right now, when running a command, sshkit accepts an option :raise_on_non_zero_exit, which defaults to true. If the command is killed with a signal, we could treat this as "non-zero exit", which then cause an exception to be raised by default.

Or, we could introduce a new option, :raise_on_killed_by_signal, default that to true. That would allow us to treat non-zero exit and killed as separate scenarios, which seems more correct to me.

[schnittchen]

I think a new option is more clear. And I think for 99.999% of all use cases, the option makes sense to be true, so that should be the default.

[braun2morrow]

Posix mandates for the shell that
"The exit status of a command that terminated because it received a signal shall be reported as greater than 128."
While sshkit is not a Posix shell and thus not bound by that, I'd argue that treating "killed by signal" any differently from "exited nonzero" still violates the Law of Least Astonishment.

In case the user desires detailed information about the death of a process, the best thing to do is to hand them two integers: both the exit code and the signal number (or 0 if the process didn't die to a signal).