I spent a fair bit of time this evening trying to get NixOS installed in a VM on my ESXi server, and this note just captures the silly problem Adam and I had.

When setting up a new VM in ESXi, you get prompted fairly early on for what type of guest OS the VM is going to be. Historically, all of my VMs were just Ubuntu, so I had only ever used Ubuntu (64-bit) as my selection here. But with NixOS, that felt like the wrong thing, so I selected Other 4.x or later Linux (64-bit) and went on my way. VMXNET3 adapter with a manually set mac address so I can hook it up to my IP block from OVH. Boot from a Datastore ISO file that contains the NixOS minimal installer. We’re off to the races.

Or so I thought.

The boot failed from the CDROM drive. I’d not had this happen before, because I always selected Ubuntu and always subsequently installed Ubuntu.

We tried again and again to boot it. Made sure the hash of the ISO was the same to make sure it didn’t get corrupted while being uploaded. Nothing was wrong, it just wouldn’t boot.

Back to the beginning, I made a new VM, but this time selected “Other Linux (64-bit).” This option annoyed me because I didn’t know that I couldn’t use VMXNET3 on this guest OS, and using the E1000 driver has performance impact. But I moved on, set the boot from the ISO, and started it up.

And we were booted.

Weird. Why did simply changing this Guest OS setting during setup change the ability of the installer to run? It’s not like it was starting up the installer and then failing partway through, it was just failing immediately.

I went back to the first VM, shut it down, and tried to change the guest OS type to “Other Linux (64-bit).” Booted it up, and it failed again. The same way. What the heck.

Comparing the VMX files

The next thing I tried was to take the .vmx files from both VMs and comparing them to see what the difference could be. I downloaded the .vmx file for each VM from the datastore and tossed them into diffchecker.

Not to my surprise, at this point, there were numerous differences. The ones that I thought might play a part are included below.

Functional VMX

ide0:0.deviceType = "cdrom-image"
ide0:0.fileName = "/vmfs/volumes/5edb1eff-36987c5f-a604-002590893b4a/ISOs/nixos-minimal-23.11.4976.79baff8812a0-x86_64-linux.iso"
ide0:0.present = "TRUE"
displayName = "adam-nixos"
guestOS = "otherlinux-64"

Non-functional VMX

sata0:0.deviceType = "cdrom-image"
sata0:0.fileName = "/vmfs/volumes/5edb1eff-36987c5f-a604-002590893b4a/ISOs/nixos-minimal-23.11.4976.79baff8812a0-x86_64-linux.iso"
sata0:0.present = "TRUE"
displayName = "dade-nixos"
guestOS = "other4xlinux-64"
uefi.secureBoot.enabled = "TRUE"

I wondered if the NixOS installer didn’t work correctly on a SATA CDROM device. This seemed like a dumb thing to wonder, in hindsight. I was on the right track, just looking at the wrong setting.

The important bit here was that last line.

uefi.secureBoot.enabled = "TRUE"

Secure Boot

It turns out, NixOS doesn’t support secure boot out of the box. NixOS Wiki has a page about this, and recommends the Lanzaboote project if you’re interested in enabling Secure Boot for your NixOS installation.

That’s great and all, but what I’m interested in is a functional virtual machine live installer experience. Maybe we can turn secure boot on later, who knows. But for now, let’s change that setting and see if it works. I powered off the broken machine, went into the edit menu, selected “VM Options” at the top, and then went into “Boot Options.” From here, I toggled off the “Enable UEFI secure boot” option.

I saved my changes and fired up the VM again. The live installer works, but now with a VMXNET3 adapter. Huzzah!

Now to face the bigger problem: How do I install NixOS?