depot/web/lukegbcom/posts/2024-01-07-nixos-stuck-boot.md

1.6 KiB

title date layout
NixOS: Stuck Boot (bad systemd unit) 2024-01-07 Post

One of my systems at home runs NixOS and receives some (encrypted) backups via zfs send/zfs recv shenanigans. I don't want to actually decrypt these at boot, but I forgot to set boot.zfs.requestEncryptionCredentials appropriate, so I got dropped into a systemd recovery prompt.


To fix this enough that the system would boot, I manually made sure that I had imported and loaded the necessary keys for the ZFS pool in question (named "tank") manually:

zpool import tank
zfs load-key tank/enc

Because zfs-import-tank is configured as Type=oneshot and RemainAfterExit=true, the unit only needs to be marked as successful once, then we can reload back to the "broken" config, but the fact that the unit ran will be remembered.

As such, to make the system finish booting enough that I could reliably switch to a new system with a fixed config, I copied the systemd unit to /tmp, and sneakily edited it to replace the ExecStart with /nix/var/nix/profiles/system/sw/bin/true, then bind-mounted the unit over the one in /etc:

cat /etc/systemd/system/zfs-import-tank.service > /tmp/zfs-import-tank.service
vim /tmp/zfs-import-tank.service
mount --bind /tmp/zfs-import-tank.service /etc/systemd/system/zfs-import-tank.service

# Start our hacked-up zfs-import-tank service
systemctl daemon-reload
systemctl start zfs-import-tank

# Revert the system to its prior state for cleanliness
umount /etc/systemd/system/zfs-import-tank.service
systemctl daemon-reload

# Finish booting
systemctl default