Sometimes you need to continously monitor a systemd service and respond quickly to warning or error messages. Here we show how to use a tiny bash script and a named pipe to facilitate this kind of auditing. You can easily adapt this to monitor other services using bash.
Full example in this github gist.
This step couldn't be easier:
# mkfifo mypipe
This just sets up a fifo (pipe) under the name "mypipe" in the current directory.
This part will set up the source of your log messages. Since I'm interested in a systemd service (hostapd), I use journalctl:
# journalctl -x -b -f -u hostapd >mypipe &
It's important that whatever you're continuously monitoring, the command you use to feed data into the pipe should be put in "follow mode" (here -f). This is so that the command doesn't just print the current contents of the log and exit. We want it to wait indefinitely for log messages and print them as they become available.
Also, don't forget to run the command in the background using & so that your script can continue doing the line-reading parts described below.
This part uses a neat bash syntax for reading lines indefinitely from a file or pipe:
while read line; do # ... do interesting things with $line here done < mypipe
In my case, I'd check the line against a regexp, and take action accordingly:
if [[ "$line" =~ "Could not add STA to kernel driver" ]]; then sleep 3 /usr/bin/systemctl restart hostapd fi
That's it! This technique is so useful for watching troublesome daemons, kicking them when they are down, or for creating more assertive email notifications to administrators in response to long-running service events.
Again, my full example is here.