Self-hosted GitHub Actions runners produce incomplete multi-stage Docker images after kernel update (overlay2: missing build artifacts)

Environment

CI: GitHub Actions self-hosted Ubuntu 20.04 runners with Docker Engine 20.10 using overlay2 on ext4; runners are VMs on on-prem KVM cluster; CI builds multi-stage Dockerfiles that copy artifacts from builder stage to final stage; recent host kernel updated from 5.4.0-xx to 5.11.0-xx.

Symptoms

Several CI jobs started failing after kernel update: final Docker image missing binary artifacts that were present in builder stage; container unpacks appear successful but runtime errors 'executable not found' or 0-byte files in /app/bin; rebuilding the same Dockerfile locally on developer machines reproduces clean image.

Observations

Build logs show successful 'COPY --from=builder /app/dist /app/dist' steps with no errors; running intermediate 'builder' image in CI runner shows artifacts present inside builder stage; final image filesystem lacks files or contains zero-length placeholders. docker build uses BuildKit disabled; daemon logs show overlay2 'link delete' warnings at times; host kernel logs contain overlayfs WARN messages referencing missing dentry and delayed lowerdir cleanup around build time.

Already attempted

Downgraded Docker daemon to previous minor release (no effect); rebuilt images with --no-cache (same result); enabled BuildKit on CI runner (changed behavior: artifacts present but image larger and a different failure later); moved runner to another host without kernel update — builds succeed there; preserved failing host journalctl and /var/log/docker logs.

Question

What kernel/overlay2 interactions or race conditions can cause copied files to vanish from final multi-stage images on self-hosted runners, and what reversible checks and mitigations should I perform to confirm overlayfs corruption vs Docker bug vs CI runner host config?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 2d ago• Trouble ID

Most likely causes

Kernel overlayfs regression in the updated kernel causing incorrect lowerdir/upperdir merges under heavy layer operations 90%

Overlay2 driver interacting badly with ext4 options (e.g., ftype=0) or with concurrent containerfs operations causing dentry cleanup races 75%

Docker daemon bug triggered by specific sequence of COPY in multi-stage build on overlay2 that BuildKit avoids 30%

Summary

Symptoms point to an overlayfs/overlay2-related race where files visible in a source layer aren't reliably propagated to the merged final image. The correlation with a kernel update and host-specific success/failure suggests a kernel/overlayfs regression or incompatible overlay driver behavior. BuildKit changing the result implies different layering semantics hide the problem. Warnings in kernel logs and docker daemon overlay2 warnings strengthen the overlayfs hypothesis; filesystem metadata races or whiteout/link handling could cause zero-length placeholders or missing files in the merged view.

Suggested Fixes

Diagnostic step 1

Reproduce with minimal Dockerfile: create small multi-stage Dockerfile that copies a generated file and run build repeatedly on failing host to confirm reproducible failure; collect daemon logs and host kernel ring buffer during reproduce (journalctl -k and dockerd logs).

Diagnostic step 2

Check ext4 ftype and storage health: verify filesystem supports overlay by running 'stat -f -c %T /var/lib/docker' and ensure ftype=1 on backing fs (tune if misconfigured); run fsck on unmounted partition during maintenance window, or clone runner to test host if fsck not possible now.

Diagnostic step 3

Test with BuildKit: enable BuildKit in CI as a temporary workaround (export DOCKER_BUILDKIT=1) to see if it reliably avoids the bug while diagnosing; note this is reversible.