Sometimes you need to continously monitor a systemd service and respond quickly to warning or error messages. Here we show how to use a tiny bash script and a named pipe to facilitate this kind of auditing. You can easily adapt this to monitor other services using bash.

Full example in this github gist.

Set up a named pipe

This step couldn't be easier:

# mkfifo mypipe

This just sets up a fifo (pipe) under the name "mypipe" in the current directory.

Redirect output of a running log file to the pipe

This part will set up the source of your log messages. Since I'm interested in a systemd service (hostapd), I use journalctl:

# journalctl -x -b -f -u hostapd >mypipe &

It's important that whatever you're continuously monitoring, the command you use to feed data into the pipe should be put in "follow mode" (here -f). This is so that the command doesn't just print the current contents of the log and exit. We want it to wait indefinitely for log messages and print them as they become available.

Also, don't forget to run the command in the background using & so that your script can continue doing the line-reading parts described below.

Read from the pipe and optionally take action

This part uses a neat bash syntax for reading lines indefinitely from a file or pipe:

while read line; do
  # ... do interesting things with $line here
done < mypipe

In my case, I'd check the line against a regexp, and take action accordingly:

if [[ "$line" =~ "Could not add STA to kernel driver" ]]; then
  sleep 3
  /usr/bin/systemctl restart hostapd
fi

That's it! This technique is so useful for watching troublesome daemons, kicking them when they are down, or for creating more assertive email notifications to administrators in response to long-running service events.

Again, my full example is here.

© 2021 michaeljoya.com