1
Diagnostic step 1
Immediate mitigation: schedule a maintenance window and delete or move nonessential large logs to external storage to free >10% of disk before further operations; avoid deleting system journals until a backup exists.
FULL AI ANALYSIS
systemd-journald grows log files until root partition fills after logrotate failed on overlay root filesystem
Generated Oct 3, 2026, 12:35:03 AM
Analysis summary
The immediate cause of full disk is runaway log/journal growth driven by an application emitting frequent warnings and a failed logrotate job that left rotated logs undeleted. On an overlay or small eMMC root, journald and logrotate must be configured with tight caps and guaranteed rotation. Vacuums failed initially because temporary operations also require spare space; temporary deletion freed enough to perform sensible cleanup. Reproduced rapid growth indicates the application issue must be fixed or its log verbosity reduced and rotation enforced to prevent recurrence.
Possible causes
Application repeatedly logging the same warning without backoff, overwhelming journal and consuming free space
95%Logrotate configuration file with bad permissions caused rotation to fail leaving older logs retained
70%Overlayfs behavior with small writable layer causing tmp file allocation failures during vacuum operations
60%journald SystemMaxUse not low enough for the small eMMC and rate-limits not configured, allowing bursts to fill disk
50%Suggested fixes
2
Diagnostic step 2
Configure systemd-journald with lower SystemMaxUse and SystemKeepFree (e.g., keep free at least 10% of eMMC) and enable RateLimitIntervalSec/RateLimitBurst to limit floods.
3
Diagnostic step 3
Fix logrotate permission error: verify ownership and permissions of files in /etc/logrotate.d, test logrotate -d to confirm rotate path, then run a manual rotation to ensure it can rotate and delete older logs.
4
Diagnostic step 4
Investigate the application flooding logs: enable debug logging to capture context or temporarily reduce its logging level; add application-side backoff to reduce repetition. Consider redirecting verbose app logs to a rotating file with size limits rather than journal.
5
Diagnostic step 5
If constrained storage is a limiting factor, move /var/log to an external persistent partition or increase root storage; plan such changes during a maintenance window and backup before moving logs.