FULL AI ANALYSIS

Self-hosted GitHub Actions runners produce incomplete multi-stage Docker images after kernel update (overlay2: missing build artifacts)

Generated Oct 13, 2026, 9:45:52 AM

Analysis summary

Symptoms point to an overlayfs/overlay2-related race where files visible in a source layer aren't reliably propagated to the merged final image. The correlation with a kernel update and host-specific success/failure suggests a kernel/overlayfs regression or incompatible overlay driver behavior. BuildKit changing the result implies different layering semantics hide the problem. Warnings in kernel logs and docker daemon overlay2 warnings strengthen the overlayfs hypothesis; filesystem metadata races or whiteout/link handling could cause zero-length placeholders or missing files in the merged view.

Possible causes

Kernel overlayfs regression in the updated kernel causing incorrect lowerdir/upperdir merges under heavy layer operations

90%

Overlay2 driver interacting badly with ext4 options (e.g., ftype=0) or with concurrent containerfs operations causing dentry cleanup races

75%

Docker daemon bug triggered by specific sequence of COPY in multi-stage build on overlay2 that BuildKit avoids

30%

Corrupted overlay metadata on host due to previous unclean shutdowns or storage bugs on the VM's backing storage

25%

Suggested fixes

Diagnostic step 1

Reproduce with minimal Dockerfile: create small multi-stage Dockerfile that copies a generated file and run build repeatedly on failing host to confirm reproducible failure; collect daemon logs and host kernel ring buffer during reproduce (journalctl -k and dockerd logs).

Diagnostic step 2

Check ext4 ftype and storage health: verify filesystem supports overlay by running 'stat -f -c %T /var/lib/docker' and ensure ftype=1 on backing fs (tune if misconfigured); run fsck on unmounted partition during maintenance window, or clone runner to test host if fsck not possible now.

Diagnostic step 3

Test with BuildKit: enable BuildKit in CI as a temporary workaround (export DOCKER_BUILDKIT=1) to see if it reliably avoids the bug while diagnosing; note this is reversible.

Diagnostic step 4

Rollback kernel or move runner: temporarily run the runner on a host with the previous kernel or a known-good host (you already tested this) while collecting all logs from the failing host for vendor/kernel bug reporting.

Diagnostic step 5

Collect and preserve artifacts: capture and upload (to secure storage) tarballs of the builder stage filesystem and final image layers for analysis without including secrets; this helps triage whether files are omitted at layer commit or lost later.

Contact • Help • API Docs • Careers • Terms • Privacy • Cookies • Ads