Hijacking a process’s i/o streams using gdb
The Story
I recently had a very annoying problem - some daemon failed, but ran
fine when told to run at foreground (not daemonize). Running at
foreground is the easiest way to debug processes, because that way you
get their input / output / error streams in your terminal.
Said daemon had no “log to file” option as well, so the only indication
I had that something was wrong is that the daemon didn’t do what it’s
supposed to do.
When processes daemonize, they create a sub process that isn’t attached
to anything (so it won’t be affected by the terminal closing, for
instance).
The originating process usually exists after creating the sub process,
and so you can’t easily capture the output of the sub process.
I eventually realized that I need to “hijack” the stderr
stream from the
sub process. I did some stupid attempts, like this (NOT WORKING):
tail -f /proc/$(pidof $DAEMON)/ld/2
Eventually I wrote something using strace, which was OK:
sudo strace -ff -p $(pidof $DAEMON) -e write=1,2 -s 1024 2>&1 | grep "^ |"
It gave me the output I wanted, and I solved my issue (which was me
passing relative file locations, inaccessible to the sub process created
because it doesn’t inherit the parent’s working directory). However, I
wanted something more elegant. I found the commands in this
post,
which did something better - given a pid
, they redirect its input,
output and error streams to some tty
(terminal), giving you control over
the process.
However, it wasn’t a fire-and forget script. I tried to fix that :-)
The Solution
My script will hook the process to your current terminal.
Please note: I don’t think it’s a good idea to leave the hijacked
daemon running after finishing troubleshooting. You should probably
restart it.
DAEMON=afuse
MYT=$(tty)
sudo gdb -p $(pidof $DAEMON) <<EOF
call close(0)
call close(1)
call close(2)
call open("$MYT", 2, 0)
call dup(0)
call dup(0)
detach
EOF
Notice the output - much better:
The way the script works is this:
First, it uses tty
to find the path to the current terminal.
gdb is then called to close streams 0,1,2 (which are almost always
stdin
, stdout
, stderr
) and open a new stream to the tty
found before.
The new stream opens at index 0 (because it’s the lowest index
available, since we just closed 0,1,2) and copies it to 1 and 2 as well.
Now stdin
, stdout
and stderr
are all mapped to the current terminal -
success!