Centralised logging with systemd and journald

In this post I am going to use existing systemd components to build a centralised logging server with minimal configuration of systemd and journald. It’s lightweight and I can use tools I am already familiar with such as journalctl to read and parse the logs.

Setup

I run a small cluster of Raspberry Pis, a mix of version 3 and version 4. One of them has started powering off when it pleases, without warning and uncerimoniously dumping any system logs that might help diagnose an issue.

It looks like the default configuration of journald on the Raspberry Pi sets Storage=auto. When the persistent store directory at /var/log/journal doesn’t exist journal messages are stored in memory only. We could change this setting but sending logs from the rest of the cluster to a central server sounds more fun and I expect will be more useful if the cluster grows.

I have considered third-party services, running an ELK stack, rsyslogd but all are more effort than I am willing to put in for a £30 computer terminating itself.

The Raspberry Pis I have are all at Debian 10 and run systemd. I discovered that amongst (many) other things systemd will send journal messages to another host. I can use systemd-journal-remote to receive and ship messages!

I’m going to use my desktop computer that runs Arch Linux to receive messages. It already has systemd-journal-remote installed as part of the systemd package.

Receiving

Create an override configuration for the systemd-journal-remote.service unit file.

1sudo systemctl edit systemd-journal-remote.service

The remote listener by default expects TLS configuration but I’m going to use plain HTTP. The following configuration switches the listen argument from HTTPS to HTTP.

1[Service]
2ExecStart=
3ExecStart=/usr/lib/systemd/systemd-journal-remote --listen-http=-3 --output=/var/log/journal/remote/

We can reload unit file changes and start the service.

1sudo systemctl daemon-reload
2sudo systemctl enable --now systemd-journal-remote.service

By default the listener uses port 19532 so we can check it’s listening.

1$ sudo ss -tlnp | grep 19532
2LISTEN  0  4096  *:19532  *:*  users:(("systemd-journal",pid=3568731,fd=3),("systemd",pid=1,fd=152))

Great, receiver is started and listening! You might need to open the port on your firewall, I’m using nftables and have a rule like this in an input chain.

1tcp dport 19532 counter accept comment "systemd-journal-remote"

Sending

The raspberry pis have systemd already but not the systemd-journal-remote package

1sudo apt-get update && \
2    sudo apt-get install systemd-journal-remote -y

Edit the file /etc/systemd/journal-upload.conf that contains the configuration for the upload service. Uncomment and set the URL= key to the IP of your receiving host and the default port 19532. Leave the TLS and certificate settings commented out.

1[Upload]
2URL=http://192.168.0.100:19532

Now start the upload service

1sudo systemctl enable --now systemd-journal-upload.service

If that has worked the directory /var/log/journal/remote should be populated with journal files from your remote hosts! You can read them with journalctl like you would system logs.

1journalctl --follow --directory=/var/log/journal/remote _HOSTNAME=raspberry-pi-3

Find more fields to filter by with journalctl --output=json-pretty.