NFS client mounts switch to read-only with 'stale file handle' after server cluster failover

Environment

Ubuntu 20.04 clients mounting NFSv4 exports from an HA storage cluster (two-node active/passive). Mounts configured in /etc/fstab using server virtual IP and automount via autofs on some hosts. Server cluster performed a planned failover which moved storage to the passive node.

Symptoms

Applications on the clients that write to the NFS share begin to receive I/O errors and the mount is switched to read-only by the kernel. Subsequent operations return 'Stale file handle' and processes attempting writes block or error out.

Observations

Kernel logs on clients show repeated messages like 'NFS: server 10.0.0.5 not responding, timed out' followed by 'NFS: Unexpected reply from server: Stale file handle'. showmount -e 10.0.0.5 returns export list only after remount. Remounting the NFS export (sudo mount -o remount /mnt/share) restores read/write access temporarily until the next failover. The server's exportfs configuration appears unchanged, but the underlying filesystem UUIDs on the storage may have changed during the failover.

Already attempted

Restarted nfs-common, rpcbind and autofs on clients, then remounted exports which restored access until the next failover. Checked /etc/fstab options and confirmed NFSv4 is used with default caching. On the server side, verified exports were present but noticed that the fsid values reported differ between nodes in the cluster after failover.

Question

What causes 'stale file handle' and mounts being forced read-only after an NFS server failover, and how can I make NFS client mounts resilient to server-side failover without risking data corruption? What reversible configuration/operational steps should I try next?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

The HA failover changed underlying filesystem IDs (fsid) so file handles on clients are invalid after failover. 95%

Client caching combined with NFSv4 stateful locking/cached handles leads to inconsistency when server reexports the filesystem. 75%

The passive node exported a different mountpoint or lacked proper bind/export options, causing clients to see a different filesystem namespace. 40%

Summary

Stale file handle indicates that the file handle (which encodes inode / fsid information) held by the client no longer matches what the server exports — often because the server's representation of the filesystem changed (different fsid or inode numbering) after a relocation or re-export. The kernel protects against silent corruption by switching the mount read-only when server replies are inconsistent. The temporary remount works because the client obtains new handles, but any failover that changes the exported filesystem identity will break clients until remounted. Using unstable export identifiers or a failover mechanism that does not preserve filesystem identity causes exactly these symptoms.

Suggested Fixes

Diagnostic step 1

Avoid online changes to exports during peak operations. Document and backup /etc/exports on both nodes before making changes.

Diagnostic step 2

On the storage/cluster side: ensure that exported filesystem identities (fsid) remain stable across failover. Configure exports to include an explicit fsid= option if appropriate so clients see the same fsid regardless of active node.

Diagnostic step 3

Test client behaviour in a maintenance window: perform a controlled failover and capture kernel logs (dmesg), rpc debug logs, and showmount outputs to correlate the change.