systemd-journald grows log files until root partition fills after logrotate failed on overlay root filesystem

Environment

Ubuntu 22.04 container host running as kiosk appliance; root filesystem is an overlayfs on a small 8GB eMMC, /var/log journal stored on / (no separate /var). systemd-journald configured with SystemMaxUse=200M. Recent package update changed rsyslog config and logrotate cron job failed with exit status 1.

Symptoms

System becomes unresponsive after overnight operation; services fail to start with 'No space left on device' and new SSH sessions cannot write to journal. Journal logs show continuous writes: a local application emits repeated warnings and each iteration is logged. Disk usage shows /dev/mmcblk0p1 at 100% and journal files under /var/log/journal occupy >90% of disk.

Observations

journalctl --disk-usage reports 5.8G used. systemctl status rsyslog.service shows 'inactive (dead)' with last logrotate execution timestamp and a 'logrotate: error reading state' message in the journal. /etc/logrotate.d/ contains a file with incorrect permissions preventing rotation. Running journalctl -f shows the same repeated warning messages every 2 seconds from application 'app-daemon[1234]'. dmesg shows 'EXT4-fs (mmcblk0p1): volume is full' around time of failure.

Already attempted

Attempted to truncate largest journal files (journalctl --vacuum-size=200M) but vacuum failed with 'No space left on device' when trying to create temporary files. Remounted root as read‑write to attempt deletion; deleted several old rotated logs from /var/log to free ~200MB and then successfully ran journalctl --vacuum-size=300M which reduced usage. Restarted services; issue reproduced within hours. Increased SystemMaxUse to 500M and configured rate-limiting for journald to suppress repeated identical messages, but underlying app continues to flood logs.

Question

How can I safely stop the immediate outage risk, ensure journald and logrotate work reliably on a constrained overlay root, and find the root cause of the repeating application warnings so disk doesn't fill again?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Application repeatedly logging the same warning without backoff, overwhelming journal and consuming free space 95%

Logrotate configuration file with bad permissions caused rotation to fail leaving older logs retained 70%

Overlayfs behavior with small writable layer causing tmp file allocation failures during vacuum operations 60%

Summary

The immediate cause of full disk is runaway log/journal growth driven by an application emitting frequent warnings and a failed logrotate job that left rotated logs undeleted. On an overlay or small eMMC root, journald and logrotate must be configured with tight caps and guaranteed rotation. Vacuums failed initially because temporary operations also require spare space; temporary deletion freed enough to perform sensible cleanup. Reproduced rapid growth indicates the application issue must be fixed or its log verbosity reduced and rotation enforced to prevent recurrence.

Suggested Fixes

Diagnostic step 1

Immediate mitigation: schedule a maintenance window and delete or move nonessential large logs to external storage to free >10% of disk before further operations; avoid deleting system journals until a backup exists.

Diagnostic step 2

Configure systemd-journald with lower SystemMaxUse and SystemKeepFree (e.g., keep free at least 10% of eMMC) and enable RateLimitIntervalSec/RateLimitBurst to limit floods.

Diagnostic step 3

Fix logrotate permission error: verify ownership and permissions of files in /etc/logrotate.d, test logrotate -d to confirm rotate path, then run a manual rotation to ensure it can rotate and delete older logs.