After a kernel upgrade Docker containers get 'permission denied' writing to bind-mounted volumes (overlay2/overlayfs change)

Environment

Ubuntu 22.04 LTS host upgraded from kernel 5.15.x to 6.1.x, Docker Engine 23.0 using overlay2 storage driver, containers run as rootless user namespaces in some environments, host filesystem is ext4, volumes are host bind mounts under /srv/appdata.

Symptoms

Existing containers that previously wrote to bind-mounted directories now fail with EACCES / permission denied when trying to create files or change ownership in those mounts. Container processes run and respond to liveness probes, but application logs show repeated permission errors and I/O failures. No changes to application code or container images were made at the same time.

Observations

Changing permissions on the host to 777 on the target directory did not resolve writes from inside the container. Running an interactive shell inside the container and attempting 'touch /srv/appdata/test' returns permission denied; strace shows open( ... , O_CREAT) failing with EACCES. dmesg shows kernel messages referencing overlayfs and 'permission denied' or 'revalidate failed' entries. AppArmor audit logs show no denials. Running the container with --privileged makes the write succeed. Reverting to the previous kernel (booting old kernel) restores write functionality.

Already attempted

Restarted Docker daemon, recreated containers, recreated bind mount directories, verified host permissions and ownership, tested writing from a new scratch container, reboots between kernel versions, tried changing storage driver to vfs for one test (container recreated manually) and observed write success with vfs. Did not alter SELinux or AppArmor profiles on the host.

Question

What kernel or Docker interactions can cause overlay2-mounted containers to see permission denied on host bind mounts after a kernel update, and what safe diagnostics and remediation steps should I take so I can restore write access without risking data loss?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Kernel regression or behavior change in overlayfs requiring different handling for user namespace remapping (shiftfs) leading to UID/GID mismatch enforcement. 88%

A change in how the kernel enforces DAC/permissions for overlay-mounted bind mounts (stricter checks introduced). 85%

Docker storage driver incompatibility with the new kernel (overlay2 expecting an older overlayfs ABI). 70%

Summary

A kernel update introduced a change in overlayfs or in user namespace / shiftfs handling that makes Docker's overlay2 with bind-mounted host directories fail to map UIDs correctly or enforce different permission checks. The fact that --privileged allows writes and that reverting the kernel fixes the problem points to a kernel-side regression/behavioral change rather than an application bug. Another possibility is a change in required mount options for overlayfs, or that user namespace remapping (userns) now requires shiftfs/explicit mapping which the current Docker version doesn't provide in your configuration.

Suggested Fixes

Diagnostic step 1

Collect diagnostic logs before making changes: capture dmesg, journalctl -u docker, and the exact overlayfs kernel messages (dmesg -T | grep overlay) to document the failure.

Diagnostic step 2

Test with a minimal reproduce case: run an official lightweight image (alpine) and bind mount a new host directory to reproduce the EACCES; keep the test directory separate from production data.

Diagnostic step 3

If a regression is suspected, temporarily boot the previous working kernel to restore production and avoid data loss while performing deeper diagnostics.