depot/web/lukegbcom/posts/2023-08-19-nixos-on-xen-debootstrap-style.md

---
title: "NixOS on Xen PV... debootstrap style"
date: 2023-08-19
layout: Post
hero: https://images.unsplash.com/photo-1481729379561-01e43a3e1ed4
hero credit: https://unsplash.com/photos/H6HNYGsyeKQ
hero credit text: "Noah Buscher"
classes:
  header: header-black-gradient
---

One of my work colleagues was commenting that they like the Xen PV model -
where you have a fairly lightweight hypervisor that runs cooperating kernels
(or, as Xen calls them, "domains"). They've been meaning to try out NixOS
but couldn't figure out how to build a debootstrap-style root FS.

---

NixOS is interesting in that really the only thing that _needs_ to exist on the
disk is a /nix/store directory containing a built system -- everything else
somewhat springs out of nothing as long as the boot process can figure out what
exactly it was you were intending to boot.

This means that, if you have a system with Nix on (which doesn't have to be
NixOS -- I'm going to use Debian 12 running as a Xen dom0 to demonstrate), then
it's relatively easy to build a Xen PV compatible rootfs.

## Table of contents

## Getting your host system set up

### Installing Nix itself

You'll need Nix installed. I prefer to use the [Determinate Nix
Installer](https://github.com/DeterminateSystems/nix-installer) in most cases
rather than the official one - it does a better job, and provides an uninstall
tool that works, so if you decide you're not interested in Nix anymore then
it's easy to get rid of again.

So...

```text
$ curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install
info: downloading installer https://install.determinate.systems/nix/tag/v0.11.0/nix-installer-x86_64-linux
`nix-installer` needs to run as `root`, attempting to escalate now via `sudo`...
Nix install plan (v0.11.0)
Planner: linux (with default settings)

Planned actions:
* Create directory `/nix`
* Fetch `https://releases.nixos.org/nix/nix-2.17.0/nix-2.17.0-x86_64-linux.tar.xz` to `/nix/temp-install-dir`
* Create a directory tree in `/nix`
* Move the downloaded Nix into `/nix`
* Create build group (GID 30000)
* Setup the default Nix profile
* Place the Nix configuration in `/etc/nix/nix.conf`
* Configure the shell profiles
* Create directory `/etc/tmpfiles.d`
* Configure Nix daemon related settings with systemd
* Remove directory `/nix/temp-install-dir`


Proceed? ([Y]es/[n]o/[e]xplain): y
 INFO Step: Create directory `/nix`
 INFO Step: Provision Nix
 INFO Step: Create build group (GID 30000)
 INFO Step: Configure Nix
 INFO Step: Create directory `/etc/tmpfiles.d`
 INFO Step: Configure Nix daemon related settings with systemd
 INFO Step: Remove directory `/nix/temp-install-dir`
Nix was installed successfully!
To get started using Nix, open a new shell or run `. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh`
```

### Getting a copy of nixpkgs

The Determinate Nix installer doesn't pull a nixpkgs channel; it's expecting you
to use flakes. For the purposes of this demo though, I'm not really interested
in doing that right now, so I'm going to clone nixpkgs myself.

To speed things up, I'm just cloning the latest nixpkgs NixOS release branch (23.05
"Stoat" at the time of writing).

```text
$ git clone --depth 1 --branch nixos-23.05 https://github.com/NixOS/nixpkgs.git
Cloning into 'nixpkgs'...
remote: Enumerating objects: 58617, done.
remote: Counting objects: 100% (58617/58617), done.
remote: Compressing objects: 100% (37070/37070), done.
remote: Total 58617 (delta 2534), reused 55277 (delta 2393), pack-reused 0
Receiving objects: 100% (58617/58617), 43.47 MiB | 9.97 MiB/s, done.
Resolving deltas: 100% (2534/2534), done.
Updating files: 100% (35411/35411), done.
```

## Building a NixOS system

### Writing a system configuration

Next up on our grand tour: a NixOS system configuration. This is the file which describes
NixOS system looks like and should encapsulate most of the non-runtime configuration.

I wrote this file out to `system-configuration.nix`:

```nix
{ config, lib, pkgs, ... }:

{
  # Ensure we have the Xen block device module available during boot.
  boot.initrd.availableKernelModules = [ "xen-blkfront" "xen-kbdfront" ];

  # Mount /dev/xvda1 at /.
  fileSystems."/" = {
    device = "/dev/xvda1";
    fsType = "ext4";
  };

  # Disable GRUB: we're not using it here.
  boot.loader.grub.enable = false;

  networking = {
    # Set our hostname.
    hostName = "nixos-inside-xen";

    # Disable global DHCP but enable it on the NIC we will get.
    useDHCP = false;
    interfaces.enX0.useDHCP = true;

    # Use systemd-networkd, rather than legacy script-based networking.
    useNetworkd = true;
  };

  # Use systemd-as-stage-1, rather than legacy script-based stage 1/stage 2.
  boot.initrd.systemd.enable = true;

  # Set our timezone.
  time.timeZone = "Europe/London";

  # Create a 'user' user, with sudo powers.
  users.users.user = {
    isNormalUser = true;
    extraGroups = [ "wheel" ];
    password = "thisisinsecure";
  };

  # Enable SSH for good measure.
  services.openssh.enable = true;

  # Make sure we have an editor.
  environment.systemPackages = [ pkgs.vim ];
  # This can also be written as:
  # environment.systemPackages = with pkgs; [ vim ];

  system.stateVersion = "23.11";
}
```

### Building the system

Now that we have a NixOS system configuration, we can use the NixOS machinery
to turn this into a system.

```text
# Assuming that 'nixpkgs' and 'system-configuration.nix' are in the current directory:
$ nix-build 'nixpkgs/nixos' -A system --arg configuration ./system-configuration.nix
# The ./ before system-configuration.nix is important to ensure Nix interprets
# it as a path rather than a string.
[... a lot of output later ...]
building '/nix/store/125c1x5n9lgcx4gf7wswdy4m8kawmf4i-etc.drv'...
building '/nix/store/g2mlr67i0z45n695wvc5rsnpiqcmahzk-nixos-system-nixos-inside-xen-23.05pre-git.drv'...
/nix/store/7cxgpci704yqcgqizv5ih5b47n9ckmg9-nixos-system-nixos-inside-xen-23.05pre-git
```

The final line output (which also gets written to the current directory as a
symlink named `result`) is the Nix store path to the system you've just built.

Now we have the system, we can start making the disk image for Xen.

## Making the disk image

### Formatting a disk

For my purposes, I'm going to make a 10GB disk image with a single ext4
partition. If you want to do something different, you'll need to adjust
the NixOS configuration above accordingly, and rebuild it.

```text
# Make a 10G empty file.
$ truncate --size 10G nixos.img

# Create a single partition.
$ fdisk nixos.img

Welcome to fdisk (util-linux 2.38.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS (MBR) disklabel with disk identifier 0x3980dcd5.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-20971519, default 2048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-20971519, default 20971519): 

Created a new partition 1 of type 'Linux' and of size 10 GiB.

Command (m for help): w
The partition table has been altered.
Syncing disks.

# Make a partitioned loop device.
$ sudo losetup --find --partscan --show nixos.img
/dev/loop0

# Format the new partition with ext4.
$ sudo mkfs.ext4 -L NIXOS /dev/loop0p1
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done                            
Creating filesystem with 2621184 4k blocks and 655360 inodes
Filesystem UUID: 77a3e697-e6b0-4645-99f3-47528256d47b
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 
```

### Installing NixOS

We'll need the disk mounted, so I'll do that first:

```text
$ mkdir mnt
$ sudo mount /dev/loop0p1 mnt
```

Now actually installing the system is as simple as using `nix` to copy it:

```text
$ sudo $(which nix) copy --to $(readlink -f mnt) ./result --no-check-sigs
[... progress output ...]

# Now there's a "nix/store" inside the disk.
$ ls mnt/nix
store  var
$ ls mnt/nix/store | wc -l
503
```

We can now set this as the current system inside the image - this mostly just
creates a symlink, but this sets up the generational semantics of NixOS.

```text
$ sudo nix-env -p mnt/nix/var/nix/profiles/system --set ./result
$ ls mnt/nix/var/nix/profiles -l
total 8
drwxr-xr-x 2 root root 4096 Aug 19 14:49 per-user
lrwxrwxrwx 1 root root   13 Aug 19 14:59 system -> system-1-link
lrwxrwxrwx 1 root root   86 Aug 19 14:59 system-1-link -> /nix/store/7cxgpci704yqcgqizv5ih5b47n9ckmg9-nixos-system-nixos-inside-xen-23.05pre-git
```

As you can see, this has created the `system` link, which points at the current
generation (`system-1-link`), which finally points at the actual system.

For our convenience, we'll copy the configuration into the disk image. The
conventional location is `/etc/nixos/configuration.nix`. This isn't strictly
necessary to have a working system, but it makes upkeep much easier.

```text
$ sudo mkdir -p mnt/etc/nixos
$ sudo cp system-configuration.nix /etc/nixos/configuration.nix
```

For better or for worse, we'll also set up the NixOS channel definition.

```text
$ sudo mkdir mnt/root
$ echo "https://nixos.org/channels/nixos-23.05 nixos" | sudo tee mnt/root/.nix-channels
https://nixos.org/channels/nixos-23.05 nixos
```

OK, we're done with the disk image now, so we can unmount it:

```text
$ sudo umount mnt
$ sudo losetup -d /dev/loop0
```

## Booting this in Xen

We now have a disk image with NixOS installed. We don't need to copy the kernel
or ramdisk out of it because we already have it on the host. For longer term use,
though, I suggest using Xen's built in grub emulator or similar to make sure that
things are kept up to date. This will boot the same system configuration every
time.

I wrote this into my `domain.conf` - you'll need to substitute the path to
your own Nix system, and your `nixos.img`.

```ini
kernel = "/nix/store/7cxgpci704yqcgqizv5ih5b47n9ckmg9-nixos-system-nixos-inside-xen-23.05pre-git/kernel"
ramdisk = "/nix/store/7cxgpci704yqcgqizv5ih5b47n9ckmg9-nixos-system-nixos-inside-xen-23.05pre-git/initrd"
memory = 2048
name = "nixos"
vif = [ '' ]
dhcp = "dhcp"
cmdline = "xencons=tty init=/nix/store/7cxgpci704yqcgqizv5ih5b47n9ckmg9-nixos-system-nixos-inside-xen-23.05pre-git/init"
disk = ['/home/lukegb/nix-blogpost/nixos.img,,hda']
```

And now we can boot the system:

```text
$ sudo xl create ./domain.conf -c
[... it boots ...]


<<< Welcome to NixOS 23.05pre-git (x86_64) - hvc0 >>>

Run 'nixos-help' for the NixOS manual.

nixos-inside-xen login: 
```

You can then log in with the `user` / `thisisinsecure` pair. The `user` user
has sudo permission, and when you're done you can shut the VM down. Assuming
that your Xen system is set up similar to mine, with a xenbr0 that has internet
access with a DHCP server (and hopefully even IPv6...!), then you should get
an IP address inside the VM as well.

### Updating the nixpkgs channel

The first thing you'll probably want to do is actually fetch the Nix channel
that we configured while creating the disk image:

```text
[user@nixos-inside-xen:~]$ sudo nix-channel --update
unpacking channels...
```

Now you can freely experiment with Nix.

## Bonus round: Making a trivial configuration change

This is some bonus work, to explore making a NixOS configuration change. You
already have a working system at this point so you can stop reading.

As an example, say you want to have `mtr` available inside your system. It's
not installed by default:

```text
[user@nixos-inside-xen:~]$ mtr --report 8.8.8.8
The program 'mtr' is not in your PATH. It is provided by several packages.
You can make it available in an ephemeral shell by typing one of the following:
  nix-shell -p mtr
  nix-shell -p mtr-gui
```

You can follow the instructions and use `nix-shell` to provide it temporarily:

```text
[user@nixos-inside-xen:~]$ nix-shell -p mtr
[... nix downloads mtr from cache ...]
[nix-shell:~]$ mtr --report 8.8.8.8
Start: 2023-08-19T15:10:53+0100
HOST: nixos-inside-xen            Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- _gateway                   0.0%    10    0.6   0.5   0.5   0.6   0.0
  2.|-- tuvok.gnet-tuvok.mldn-rd. 20.0%    10    2.0   2.2   2.0   2.6   0.2
  3.|-- blade-tuvok.public.as2054  0.0%    10    2.5   2.4   1.9   2.5   0.2
  4.|-- 195.66.224.125             0.0%    10    2.7   3.1   2.5   5.3   0.9
  5.|-- 74.125.242.97              0.0%    10    4.7   4.7   4.2   7.2   0.9
  6.|-- 192.178.46.87              0.0%    10    3.6   3.6   3.2   4.7   0.4
  7.|-- dns.google                 0.0%    10    3.2   3.4   3.0   3.7   0.2
```

except it won't quite work properly, because its `mtr-packet` process isn't
privileged:

```text
[nix-shell:~]$ mtr --udp --report 8.8.8.8
Start: 2023-08-19T15:13:03+0100
HOST: nixos-inside-xen            Loss%   Snt   Last   Avg  Best  Wrst StDev

[nix-shell:~]$ exit
[user@nixos-inside-xen:~]$ 
```

You can solve this by installing it in your NixOS configuration using the
`programs.mtr.enable` configuration option, defined
[in this module](https://github.com/NixOS/nixpkgs/blob/nixos-23.05/nixos/modules/programs/mtr.nix).
This both installs the package (into `environment.systemPackages`), but
also installs a setcap wrapper for `mtr-packet`.

Edit `/etc/nixos/configuration.nix`, and add `programs.mtr.enable = true;`
somewhere, probably just below the `environment.systemPackages` line.

Once you've done that, rebuild and switch to the new system:

```text
[user@nixos-inside-xen:~]$ sudo nixos-rebuild switch
building Nix...
building the system configuration...
these 11 derivations will be built:
  /nix/store/c9nkzphpp4hfgbx4da8pfh4b6xcfwy1s-system-path.drv
  /nix/store/0bm8y3d51c93dr27jkzzaaz0kv43vv62-unit-systemd-fsck-.service.drv
  /nix/store/k984vii2lsh7yjz8acvbfqgph0vyssfj-dbus-1.drv
  /nix/store/j9lmndz2by4ridi3vyzbd32mnay2pn4q-X-Restart-Triggers.drv
  /nix/store/d0j9nmy6qylmgllaw21l6w11b0gkhxkx-unit-dbus.service.drv
  /nix/store/1cjakrjkp764r26fddkpixjp6cff0kq7-user-units.drv
  /nix/store/8l53jy12sbls3ppxc7pqh40ialnw53x4-unit-dbus.service.drv
  /nix/store/vnky6khlsj4zasp4q8g6rwp2b3jhp99l-system-units.drv
  /nix/store/d5zmdd9g1zvv7f5s9w2n3praq8d4k1p6-etc.drv
  /nix/store/qixzqgmf54k8xypajycry4v42llhhwi1-ensure-all-wrappers-paths-exist.drv
  /nix/store/hp0dhaw2rgcpa7y70rfhnmnr2qiw8lm2-nixos-system-nixos-inside-xen-23.05.2891.ae521bd4e460.drv
[...a little bit of output...]
building '/nix/store/hp0dhaw2rgcpa7y70rfhnmnr2qiw8lm2-nixos-system-nixos-inside-xen-23.05.2891.ae521bd4e460.drv'...
Warning: do not know how to make this configuration bootable; please enable a boot loader.
activating the configuration...
setting up /etc...
reloading user units for user...
setting up tmpfiles
reloading the following units: dbus.service
```

...and you now have `mtr` available system-wide, but now it works in UDP mode:

```text
[user@nixos-inside-xen:~]$ mtr --udp --report 8.8.8.8
Start: 2023-08-19T15:19:03+0100
HOST: nixos-inside-xen            Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- _gateway                   0.0%    10    0.5   0.5   0.4   0.6   0.0
  2.|-- tuvok.gnet-tuvok.mldn-rd. 90.0%    10    2.3   2.3   2.3   2.3   0.0
  3.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  4.|-- 195.66.224.125             0.0%    10    2.8   2.8   2.5   3.3   0.3
  5.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0

```