systemd socket-activated service times out after moving /var to separate partition

Environment

Ubuntu 22.04 server running systemd 249 with /var moved to a dedicated ext4 LVM logical volume (fstab updated). A critical internal service uses systemd socket activation (mydaemon.socket → mydaemon.service).

Symptoms

After reboot the machine reaches multi-user.target but mydaemon never starts; systemctl status mydaemon shows 'loaded, inactive (dead)' and 'Socket unit mydaemon.socket is listening' with 'Startup finished in Xs (kernel)'. Attempts to start the service manually hang and eventually fail with a 90s timeout.

Observations

journalctl -b shows unit startup ordering: systemd starts mydaemon.socket before /var is mounted; logs include 'Failed to listen on socket: No such file or directory' pointing to /var/run/mydaemon.sock; fstab entry for /var uses UUID=... defaults, 0 2; 'systemctl list-dependencies multi-user.target' shows no explicit dependency from socket unit to local-fs.target; mounting /var manually and restarting the socket unit makes the service work immediately.

Already attempted

Edited mydaemon.service and mydaemon.socket to add After=local-fs.target and Wants=/var; reloaded systemd daemon and rebooted — no effect because the socket unit still activated too early. Tried adding x-systemd.automount to fstab for /var — caused different timing issues. Restored /var content to root filesystem (temporary) and service started normally.

Question

What is the minimal, reversible systemd unit or fstab change to ensure a socket‑activated service waits for /var to be mounted before systemd creates/listens on the socket? How can I avoid race conditions without moving /var back to the root filesystem?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

mydaemon.socket lacks an explicit dependency on local-fs.target or the specific mount unit for /var, so socket activation occurs before /var is mounted 100%

The socket path is inside /var; when /var is a late-mounted filesystem the directory doesn't exist at activation time 95%

Incorrect fstab options (noauto or improper systemd automount settings) change mount ordering and cause race conditions 20%

Summary

Socket activation can fail if systemd creates the listening socket path in /var/run before /var is mounted; systemd evaluates unit dependencies and by default may activate sockets early. The observed 'No such file or directory' indicates the directory under /var does not exist at socket activation time. Adding ordering to the socket unit or ensuring the socket path is on a filesystem available early are typical remedies. Care must be taken to not create dependency loops between mount and socket units.

Suggested Fixes

Diagnostic step 1

Modify mydaemon.socket: add 'After=var.mount' and 'Requires=var.mount' (replace var.mount with the actual mount unit name from systemctl list-units --type=mount) so socket activation waits for the /var mount — this is reversible (systemctl daemon-reload after edits)

Diagnostic step 2

Alternatively move the socket path to /run (tmpfs mounted early) and create a small drop-in for mydaemon.service specifying ListenStream=/run/mydaemon.sock; after confirming, create a symlink or tmpfiles.d entry to establish the directory on boot

Diagnostic step 3

Test changes without reboot by manually mounting /var and restarting units; keep a rescue method (single-user mode or live USB) before making changes on remote/critical systems