She logged in. Heart sank. The 12-TB VMFS volume—hosting a real-time fraud detection system—wasn’t just offline. It was gone. ls -la /vmfs/volumes/ showed only the local datastore. Someone (an intern following an outdated runbook) had accidentally zapped the LUN mapping from the SAN side, then re-presented it—but as a new device signature.
Within 90 minutes, 34 of 37 VMs booted cleanly. Three had corrupted swap files—recreated them. The fraud detection system was back online at 4:15 AM, just before the London trading desk opened.
Step 3: Deeper scan. She ran vmfs6-recover (part of vmfs-tools ). It parsed backup VMFS metadata—the first copy of the file system descriptor had been overwritten when the host re-scanned the "new" LUN, but VMware stores a second copy at offset 512 MB. recover vmfs datastore
Success: Found backup VMFS6 superblock at 0x20000000 .
Step 5: Mount attempt on ESXi: # esxcli storage vmfs snapshot mount -l Prod-HighSpeed She logged in
Step 2: Use vmfs-fuse to try a read-only mount. # vmfs-fuse /dev/sde1 /mnt/recover → failed: "Unsupported VMFS version or corrupted heartbeat region" .
Step 1: Identify the device. fdisk -l showed /dev/sde as 12 TB, with partition 1 (VMFS) starting at sector 2048. Good—partition still there. It was gone
Maya stared at the now-green dashboard. Somewhere in the datacenter, a disk blinked its steady heartbeat. She smiled. Another VMFS ghost story, beaten by knowing exactly where VMware hides its backup superblocks.