Nutanix – Updating CE2 from LCM, server not coming back up

If you have updated the Community Edition 2.0, from the LCM (Life Cycle Manager), you may experience that the server is not coming back online. You may experience that the AHV is printing out the following, continuously:

dracut-initqueue[682]: Warning: dracut-initqueue timeout - starting timeout scripts.

After many outputs of this line, you will end up in the Dracut Emergency Shell, this, with a warning of /dev/disk/by-uuid/{UUID} not existing. This is also a driver issue, like it was with the Realtek network driver.

In order to solve this, you should load the kernel modules.

From the Dracut Emergency Shell, type in the following if you’re using a USB (which may be the disk not being read):

insmod /lib/modules/5.10.139-2.el7.nutanix.20220304.385.x86_64/kernel/drivers/usb/storage/usb-storage.ko

Make sure to press TAB to auto complete the version as it may vary for you.

If you’re also using SATA (AHCI), type in the following, still in Dracut Emergency Shell, then exit the shell.

modprobe ahci
exit

Damage control after update

As settings were overwritten during the upgrade, we need to disable these changed settings:

Open the file /etc/modprobe.d/pci-passthru-nx2k.conf and comment out the settings to match below:

# Intel SATA AHCI (NX-2000)
#blacklist ahci
#blacklist lpc_ich
#blacklist i2c_i801

# Allow AHCI controller of the multifunction device to be passed through.
#options vfio_iommu_type1 allow_unsafe_interrupts=1

Additionally we must comment out the settings below, from /etc/modprobe.d/stig-reqs-usbstorage.conf:

#install usb-storage /bin/true
#blacklist usb-storage

Backup and update initramfs

Navigate to /boot and backup the original initramfs file, type dracut -f to force the creation of a new initramfs image. Lastly, reboot.

cd /boot
cp initramfs-5.10.139-2.el7.nutanix.20220304.385.x86_64.img initramfs-5.10.139-2.el7.nutanix.20220304.385.x86_64.img.bak

dracut -f

reboot

Finally, restart the CVM, and the cluster should get back online, with Nutanix showing the correct, updated versions.