In this post I am going to use existing systemd components to build a
centralised logging server with minimal configuration of systemd and journald.
It’s lightweight and I can use tools I am already familiar with such as
journalctl to read and parse the logs.
I run a small cluster of Raspberry Pis, a mix of version 3 and version 4. One of them has started powering off when it pleases, without warning and uncerimoniously dumping any system logs that might help diagnose an issue.
It looks like the default configuration of
journald on the Raspberry Pi sets
Storage=auto. When the persistent store directory at
exist journal messages are stored in memory only. We could change
this setting but sending logs from the rest of the cluster to a central server
sounds more fun and I expect will be more useful if the cluster grows.
I have considered third-party services, running an ELK stack, rsyslogd but all are more effort than I am willing to put in for a £30 computer terminating itself.
The Raspberry Pis I have are all at Debian 10 and run systemd. I discovered that
amongst (many) other things systemd will send journal messages to another host.
I can use
systemd-journal-remote to receive and ship messages!
I’m going to use my desktop computer that runs Arch Linux to receive
messages. It already has
systemd-journal-remote installed as part of the
Create an override configuration for the
sudo systemctl edit systemd-journal-remote.service
The remote listener by default expects TLS configuration but I’m going to use plain HTTP. The following configuration switches the listen argument from HTTPS to HTTP.
[Service] ExecStart= ExecStart=/usr/lib/systemd/systemd-journal-remote --listen-http=-3 --output=/var/log/journal/remote/
We can reload unit file changes and start the service.
sudo systemctl daemon-reload sudo systemctl enable --now systemd-journal-remote.service
By default the listener uses port
19532 so we can check it’s listening.
$ sudo ss -tlnp | grep 19532 LISTEN 0 4096 *:19532 *:* users:(("systemd-journal",pid=3568731,fd=3),("systemd",pid=1,fd=152))
Great, receiver is started and listening! You might need to open the port on
your firewall, I’m using
nftables and have a rule like this in an input chain.
tcp dport 19532 counter accept comment "systemd-journal-remote"
The raspberry pis have
systemd already but not the
sudo apt-get update && \ sudo apt-get install systemd-journal-remote -y
Edit the file
/etc/systemd/journal-upload.conf that contains the configuration
for the upload service. Uncomment and set the
URL= key to the IP of your
receiving host and the default port
19532. Leave the TLS and certificate
settings commented out.
Now start the upload service
sudo systemctl enable --now systemd-journal-upload.service
If that has worked the directory
/var/log/journal/remote should be populated
with journal files from your remote hosts! You can read them with
like you would system logs.
journalctl --follow --directory=/var/log/journal/remote _HOSTNAME=raspberry-pi-3
Find more fields to filter by with