mdadm RAID1 Array Shows 'wrong UUID' and Refuses to Re‑add Disk After Transient USB Enclosure Disconnect

Environment

Debian 12 server with mdadm RAID1 composed of two 2TB SATA disks: one internal (sda) and one in an external USB hot-swap enclosure (sdb). mdadm 4.2, kernel 6.1. After accidental power loss to the enclosure while OS running, the external disk reconnected and now shows mdadm error on assembly.

Symptoms

mdadm --detail /dev/md0 reports only one active device (sda) and the other device shows 'spare' with event count mismatch and 'wrong metadata' when attempting mdadm --add /dev/md0 /dev/sdb. /var/log/syslog shows repeated mdadm messages: 'Device … has wrong UUID 0000: not in array' and kernel messages about 'SCSI disk removed/added'. Filesystem (ext4) mounted read-only occasionally due to clean unclean flag concerns.

Observations

smartctl -a on the external disk shows no SMART errors. mdadm --examine /dev/sdb reports a valid raid superblock but with an older event count than sda. The external enclosure exposes the drive as /dev/sdb but with slightly different serial ordering occasionally. There is no apparent hardware damage; the array had been healthy prior to the disconnect. dmesg shows USB reset messages and slight device path changes when reconnected.

Already attempted

Tried mdadm --add, mdadm --assemble --update=resync, recreated superblock on spare (sketchy), and ran fsck on replica (did not write changes). Did not attempt zeroing superblocks because of data safety concerns. Rebooted server once (array assembled degraded on boot), then reconnected enclosure before boot — same result. No backups of entire disk image available beyond regular file backups.

Question

What safe diagnostics and recovery steps can I take to restore the RAID1 mirror without risking data loss, and how can I avoid permanently recreating array metadata that might make recovery impossible?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

External disk did not record the latest array events due to sudden power loss while enclosue was disconnected, causing an event-count gap. 95%

Human attempt to rewrite metadata in earlier recovery attempts left inconsistent state between disks. 80%

USB enclosure firmware or path change altered how the kernel presents the drive (different WWN/ID), making mdadm treat it as a different device with mismatched UUID. 50%

Summary

The external drive appears to have a raid superblock with an event count older than the internal disk; mdadm treats it as out-of-date and refuses to add it to prevent data regression. The transient USB disconnect likely caused the external device to miss some array events; reconnecting under a different device path or with slightly different metadata can trigger UUID mismatches. Safe recovery requires careful reconciliation of raid superblock event counts, not blind metadata recreation.

Suggested Fixes

Diagnostic step 1

Make a sector-level read-only image of the external disk (dd if=/dev/sdb of=/path/to/storage/drive_sdb.img bs=1M conv=sync, status=progress) to preserve current state before any metadata changes. If insufficient local space, attach another drive or use a remote block-transfer—do not proceed without a backup image.

Diagnostic step 2

Use mdadm --examine on both devices and record the raid-superblock event counts, UUIDs, and layout. Save the outputs for vendor/advanced support.

Diagnostic step 3

If event count on sdb is lower than sda (expected), avoid zeroing metadata. Instead, attempt to add sdb as a spare with --backup-file and --write‑most‑recent‑superblock options if mdadm supports them; test first on the disk image with mdadm --build in a safe environment.