Example as given in README doesn't work

[deets]

I tried to get the simple ls-example working, but failed. Starting/Waiting/Closing all works, but not getting a buffer.

I had to resort to

https://github.com/deets/brombeerquark/blob/master/cpp/subprocess/src/cpp-subprocess.cpp

(taken from the tests) to get things to work. The command used btw. is not the culprit, I did in fact start with your ls-example. Just had no reason to commit that.

I have no idea why this is not working. I'm running on a raspberry pi with a YOCTO linux, but glibc and GCC 8.3.0.

[tsaarni]

Sorry, I'm not able to reproduce it.

I don't have access to Yocto though, but I tried on Raspbian, by modifying your example with rdbuf()

$ cat main.cpp
#include "subprocess.hpp"
#include <chrono>
#include <thread>

int main(int argc, char *argv[])
{
  using namespace std::chrono_literals;
  while(true)
  {
    std::string buf;
    subprocess::popen cmd("vcgencmd", {"measure_temp"});
    std::cout << cmd.stdout().rdbuf();
    cmd.wait();
    std::this_thread::sleep_for(1s);
  }
  return 0;
}
$ g++ --version
g++ (Raspbian 8.3.0-6+rpi1) 8.3.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$ g++ -std=c++17 -Iinclude main.cpp -o main
$ sudo ./main
temp=40.0'C
temp=40.0'C
...

[dir-ableton]

I agree that it is weird, I did try it several times. I could - if you were willing to look a bit deeper - hand you an image to flash, as well as a cross compilation SDK, if that helps. Let me know.

[tsaarni]

I must admit I don't have a clue what could be wrong. Since you have the development environment there already existing, maybe you can try couple of random things:

Maybe the problem could be related to buffering. You could try calling out_stream->rdbuf()->pubsetbuf(0,0) to disable buffering after this line but I don't think that necessarily does anything in this case since it is said "The way to do this is just to turn off the buffering before any I/O operations at all have been done (note that opening counts as an I/O operation)".

Secondly, the call to stdout() returns a reference to out_stream here. I wonder, is there any possibility that it interferes with the iostream object? Maybe you could try to return pointer instead and see if there is any effect.

edit:
Thirdly, could it be that call to rdbuf() in relation to the forked & executed program would become timing critical? You could try experiment by adding sleep after launching the program and taking the buffer and consuming it.

[dir-ableton]

Ok, a short update: I was running the following code, and the first of your above suggestions applied:

#include "subprocess.hpp"
#include <chrono>
#include <thread>

int main(int argc, char *argv[])
{
  using namespace std::chrono_literals;
  while(true)
  {
    std::cout << "before\n";
    subprocess::popen cmd("ls", {});
    std::cout << "after\n";
    std::cout << cmd.stdout().rdbuf();
    std::cout << "waiting..\n";
    std::this_thread::sleep_for(1s);
  }
  return 0;
}

No output, as before.

The following is the session:

root@move-dir:~# /tmp/cpp-subprocess &
[1] 900
root@move-dir:~# before
after

root@move-dir:~# 
root@move-dir:~# 
root@move-dir:~# ps |grep ls
  276 root         0 SW   [irq/169-ablspi_]
  901 root         0 Z    [ls]
  902 root         0 Z    [ls]
  903 root         0 Z    [ls]
  904 root         0 Z    [ls]
  905 root         0 Z    [ls]
  906 root         0 Z    [ls]
  907 root         0 Z    [ls]
  908 root         0 Z    [ls]
  909 root         0 Z    [ls]
  911 root      2200 S    grep ls
root@move-dir:~# fg
/tmp/cpp-subprocess
^C
root@move-dir:~# 

So I

• started the process in the background
• it did not print out the lines from the moment I reached for the stdout buffer.
• but it DID start the process several times!

To me this indicates that the issue here is somehow todo with meddling with the stdout of the invoking process. Or more general iostreams, because the following example showed the exact same issue:

#include <chrono>
#include <thread>

int main(int argc, char *argv[])
{
  using namespace std::chrono_literals;
  std::ofstream of("/tmp/output");
  while(true)
  {
    of << "before\n";
    subprocess::popen cmd("ls", {});
    of << "after\n";
    of << cmd.stdout().rdbuf();
    of << "waiting..\n";
    of.flush();
    std::this_thread::sleep_for(1s);
  }
  return 0;
}

Adding a std::this_thread::sleep_for(1s); as by your suggestion before trying to capture didn't do anything good, either.

I don't think there is a difference between a reference and a pointer, but I will investigate.

[dir-ableton]

As I feared, changing the reference to a pointer didn't affect the outcome :(

[tsaarni]

Thanks for troubleshooting, and sorry that I could not help.

One more random idea: Does it make any difference if you change std::ios_base::in to std::ios_base::in | std::ios_base::out | std::ios_base::binary on line:

cpp-subprocess/include/subprocess.hpp

Line 119 in 2e91c39

out_filebuf = new __gnu_cxx::stdio_filebuf<char>(out_pipe[READ], std::ios_base::in, 1);

[tsaarni]

And one more test:

subprocess::popen cmd("ls", {});
cmd.wait();
std::cout << cmd.stdout().rdbuf();

that is, wait for process to terminate before call to rdbuf().

[dir-ableton]

Neither the wait nor the file mode did change a thing :( It's really weird. It did solve the issue of accumulating processes though. Don't you think it would be a good idea to implicitly invoke wait() in ~popen()?

[dir-ableton]

Ok, one more bit of information: it works by using printf for printing. I don't get the output printed yet, there is probably something more happening when invoking operator<<, but the before/after are coming! I got this idea after staring at the strace log. I wonder if somehow the C++ stream runtime state is corrupted, and why.

[tsaarni]

Could it be somehow distantly related to this https://stackoverflow.com/questions/35261389/stringstream-rdbuf-causing-cout-to-fail, that is: "std::cout is unable to read any characters ... sets failbit on std::cout, and until this state is cleared any further output will fail too". I'm not that familiar with iostreams library to understand this really, and in your case rdbuf() is read only once so it should not be at the end - this still would be unexplained. But it seems consistent with your observation of cout getting in failed state.

Don't you think it would be a good idea to implicitly invoke wait() in ~popen()?

Yeah, I think you are right. There is potential problem though, since wait() will block until the executed command exits, but I guess that may be expected by user, at least in most cases.
Other alternatives for preventing zombies would be ignoring SIGCHLD or installing signal handler for it that collects the status by calling wait() but those are something that user should decide on.

[dir-ableton]

Just regarding the last point (will research your first suggestion later): I think the current behavior is the most suprising one. From a RIAA-perspective (and this is how I thought about this as a pattern) it follows for me that the lifetime of the object should govern the lifetime of it's resources. One of the few really bright sides of C++ ;) I agree though that the behavior is not ideal. So from my POV we can do these things:

• wait
• kill
• throw an exception if hitting ~popen, but the process is still alive.
• do wait/kill by default, but allow control at construction time.

I'm slightly in favor of killing, because it doesn't make the program hang, and exceptions are a choice that many C++-devs don't want to be forced to use. What do you think?

[tsaarni]

Yes, makes sense to me. Maybe the option between wait and kill should be provided anyways even if kill would be default.

Killing brings up yet another question: should it be SIGKILL or SIGHUP, where latter would give the child an option for graceful termination.

Furthermore, if the process has already terminated earlier kill() would not do anything, so kill would probably anyway imply waitpid(child, nullptr, WNOHANG) too, to cover that specific scenario.

[dir-ableton]

I think only SIGKILL makes sense, as SIGHUP can be ignored, and then we are in the same space as just doing an implicit wait: the program hangs, and the user has to find out why. Regarding the waitpid I have to admit I'm not deep enough in the details, but it sounds as if you are ;)

[MLutt]

Been over a month since the last reply - ran into the same issue and had to resort to using Python for now ( https://gist.github.com/MLutt/4b50cdd1644564307bda0d276e235ab4 )... Any progress on it?

[tsaarni]

I have not had time to work with this, sorry for trouble.