---
title: "NixOS: Stuck Boot (bad systemd unit)"
date: 2024-01-07
layout: Post
---

One of my systems at home runs NixOS and receives some (encrypted) backups via
`zfs send`/`zfs recv` shenanigans.  I don't want to actually decrypt these at
boot, but I forgot to set `boot.zfs.requestEncryptionCredentials` appropriate,
so I got dropped into a systemd recovery prompt.

---

To fix this enough that the system would boot, I manually made sure that I had
imported and loaded the necessary keys for the ZFS pool in question (named
"tank") manually:

```
zpool import tank
zfs load-key tank/enc
```

Because zfs-import-tank is configured as `Type=oneshot` and
`RemainAfterExit=true`, the unit only needs to be marked as successful once,
then we can reload back to the "broken" config, but the fact that the unit ran
will be remembered.

As such, to make the system finish booting enough that I could reliably switch
to a new system with a fixed config, I copied the systemd unit to /tmp, and
sneakily edited it to replace the `ExecStart` with
`/nix/var/nix/profiles/system/sw/bin/true`, then bind-mounted the unit over the
one in /etc:

```
cat /etc/systemd/system/zfs-import-tank.service > /tmp/zfs-import-tank.service
vim /tmp/zfs-import-tank.service
mount --bind /tmp/zfs-import-tank.service /etc/systemd/system/zfs-import-tank.service

# Start our hacked-up zfs-import-tank service
systemctl daemon-reload
systemctl start zfs-import-tank

# Revert the system to its prior state for cleanliness
umount /etc/systemd/system/zfs-import-tank.service
systemctl daemon-reload

# Finish booting
systemctl default
```