In this post I am going to use existing systemd components to build a
centralised logging server with minimal configuration of systemd and journald.
It’s lightweight and I can use tools I am already familiar with such as
journalctl to read and parse the logs.
Setup
I run a small cluster of Raspberry Pis, a mix of version 3 and version 4. One of them has started powering off when it pleases, without warning and uncerimoniously dumping any system logs that might help diagnose an issue.
It looks like the default configuration of journald on the Raspberry Pi sets
Storage=auto. When the persistent store directory at /var/log/journal doesn’t
exist journal messages are stored in memory only. We could change
this setting but sending logs from the rest of the cluster to a central server
sounds more fun and I expect will be more useful if the cluster grows.
I have considered third-party services, running an ELK stack, rsyslogd but all are more effort than I am willing to put in for a £30 computer terminating itself.
The Raspberry Pis I have are all at Debian 10 and run systemd. I discovered that
amongst (many) other things systemd will send journal messages to another host.
I can use systemd-journal-remote to receive and ship messages!
I’m going to use my desktop computer that runs Arch Linux to receive
messages. It already has systemd-journal-remote installed as part of the
systemd package.
Receiving
Create an override configuration for the systemd-journal-remote.service unit
file.
1sudo systemctl edit systemd-journal-remote.service
The remote listener by default expects TLS configuration but I’m going to use plain HTTP. The following configuration switches the listen argument from HTTPS to HTTP.
1[Service]
2ExecStart=
3ExecStart=/usr/lib/systemd/systemd-journal-remote --listen-http=-3 --output=/var/log/journal/remote/
We can reload unit file changes and start the service.
1sudo systemctl daemon-reload
2sudo systemctl enable --now systemd-journal-remote.service
By default the listener uses port 19532 so we can check it’s listening.
1$ sudo ss -tlnp | grep 19532
2LISTEN 0 4096 *:19532 *:* users:(("systemd-journal",pid=3568731,fd=3),("systemd",pid=1,fd=152))
Great, receiver is started and listening! You might need to open the port on
your firewall, I’m using nftables and have a rule like this in an input chain.
1tcp dport 19532 counter accept comment "systemd-journal-remote"
Sending
The raspberry pis have systemd already but not the systemd-journal-remote
package
1sudo apt-get update && \
2 sudo apt-get install systemd-journal-remote -y
Edit the file /etc/systemd/journal-upload.conf that contains the configuration
for the upload service. Uncomment and set the URL= key to the IP of your
receiving host and the default port 19532. Leave the TLS and certificate
settings commented out.
1[Upload]
2URL=http://192.168.0.100:19532
Now start the upload service
1sudo systemctl enable --now systemd-journal-upload.service
If that has worked the directory /var/log/journal/remote should be populated
with journal files from your remote hosts! You can read them with journalctl
like you would system logs.
1journalctl --follow --directory=/var/log/journal/remote _HOSTNAME=raspberry-pi-3
Find more fields to filter by with journalctl --output=json-pretty.