diff --git a/web/lukegbcom/posts/.markdownlint.jsonc b/web/lukegbcom/posts/.markdownlint.jsonc new file mode 100644 index 0000000000..1c7c4642f6 --- /dev/null +++ b/web/lukegbcom/posts/.markdownlint.jsonc @@ -0,0 +1,6 @@ +{ + "MD013": { + "code_blocks": false + }, // enforce line length limits + "MD014": false // allow shell without output +} \ No newline at end of file diff --git a/web/lukegbcom/posts/2023-08-19-nixos-on-xen-debootstrap-style.md b/web/lukegbcom/posts/2023-08-19-nixos-on-xen-debootstrap-style.md new file mode 100644 index 0000000000..28d6b51bcd --- /dev/null +++ b/web/lukegbcom/posts/2023-08-19-nixos-on-xen-debootstrap-style.md @@ -0,0 +1,444 @@ +--- +title: "NixOS on Xen PV... debootstrap style" +date: 2023-08-19 +layout: Post +hero: https://images.unsplash.com/photo-1481729379561-01e43a3e1ed4 +hero credit: https://unsplash.com/photos/H6HNYGsyeKQ +hero credit text: "Noah Buscher" +classes: + header: header-black-gradient +--- + +One of my work colleagues was commenting that they like the Xen PV model - +where you have a fairly lightweight hypervisor that runs cooperating kernels +(or, as Xen calls them, "domains"). They've been meaning to try out NixOS +but couldn't figure out how to build a debootstrap-style root FS. + +--- + +NixOS is interesting in that really the only thing that _needs_ to exist on the +disk is a /nix/store directory containing a built system -- everything else +somewhat springs out of nothing as long as the boot process can figure out what +exactly it was you were intending to boot. + +This means that, if you have a system with Nix on (which doesn't have to be +NixOS -- I'm going to use Debian 12 running as a Xen dom0 to demonstrate), then +it's relatively easy to build a Xen PV compatible rootfs. + +## Table of contents + +## Getting your host system set up + +### Installing Nix itself + +You'll need Nix installed. I prefer to use the [Determinate Nix +Installer](https://github.com/DeterminateSystems/nix-installer) in most cases +rather than the official one - it does a better job, and provides an uninstall +tool that works, so if you decide you're not interested in Nix anymore then +it's easy to get rid of again. + +So... + +```text +$ curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install +info: downloading installer https://install.determinate.systems/nix/tag/v0.11.0/nix-installer-x86_64-linux +`nix-installer` needs to run as `root`, attempting to escalate now via `sudo`... +Nix install plan (v0.11.0) +Planner: linux (with default settings) + +Planned actions: +* Create directory `/nix` +* Fetch `https://releases.nixos.org/nix/nix-2.17.0/nix-2.17.0-x86_64-linux.tar.xz` to `/nix/temp-install-dir` +* Create a directory tree in `/nix` +* Move the downloaded Nix into `/nix` +* Create build group (GID 30000) +* Setup the default Nix profile +* Place the Nix configuration in `/etc/nix/nix.conf` +* Configure the shell profiles +* Create directory `/etc/tmpfiles.d` +* Configure Nix daemon related settings with systemd +* Remove directory `/nix/temp-install-dir` + + +Proceed? ([Y]es/[n]o/[e]xplain): y + INFO Step: Create directory `/nix` + INFO Step: Provision Nix + INFO Step: Create build group (GID 30000) + INFO Step: Configure Nix + INFO Step: Create directory `/etc/tmpfiles.d` + INFO Step: Configure Nix daemon related settings with systemd + INFO Step: Remove directory `/nix/temp-install-dir` +Nix was installed successfully! +To get started using Nix, open a new shell or run `. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh` +``` + +### Getting a copy of nixpkgs + +The Determinate Nix installer doesn't pull a nixpkgs channel; it's expecting you +to use flakes. For the purposes of this demo though, I'm not really interested +in doing that right now, so I'm going to clone nixpkgs myself. + +To speed things up, I'm just cloning the latest nixpkgs NixOS release branch (23.05 +"Stoat" at the time of writing). + +```text +$ git clone --depth 1 --branch nixos-23.05 https://github.com/NixOS/nixpkgs.git +Cloning into 'nixpkgs'... +remote: Enumerating objects: 58617, done. +remote: Counting objects: 100% (58617/58617), done. +remote: Compressing objects: 100% (37070/37070), done. +remote: Total 58617 (delta 2534), reused 55277 (delta 2393), pack-reused 0 +Receiving objects: 100% (58617/58617), 43.47 MiB | 9.97 MiB/s, done. +Resolving deltas: 100% (2534/2534), done. +Updating files: 100% (35411/35411), done. +``` + +## Building a NixOS system + +### Writing a system configuration + +Next up on our grand tour: a NixOS system configuration. This is the file which describes +NixOS system looks like and should encapsulate most of the non-runtime configuration. + +I wrote this file out to `system-configuration.nix`: + +```nix +{ config, lib, pkgs, ... }: + +{ + # Ensure we have the Xen block device module available during boot. + boot.initrd.availableKernelModules = [ "xen-blkfront" "xen-kbdfront" ]; + + # Mount /dev/xvda1 at /. + fileSystems."/" = { + device = "/dev/xvda1"; + fsType = "ext4"; + }; + + # Disable GRUB: we're not using it here. + boot.loader.grub.enable = false; + + networking = { + # Set our hostname. + hostName = "nixos-inside-xen"; + + # Disable global DHCP but enable it on the NIC we will get. + useDHCP = false; + interfaces.enX0.useDHCP = true; + + # Use systemd-networkd, rather than legacy script-based networking. + useNetworkd = true; + }; + + # Use systemd-as-stage-1, rather than legacy script-based stage 1/stage 2. + boot.initrd.systemd.enable = true; + + # Set our timezone. + time.timeZone = "Europe/London"; + + # Create a 'user' user, with sudo powers. + users.users.user = { + isNormalUser = true; + extraGroups = [ "wheel" ]; + password = "thisisinsecure"; + }; + + # Enable SSH for good measure. + services.openssh.enable = true; + + # Make sure we have an editor. + environment.systemPackages = [ pkgs.vim ]; + # This can also be written as: + # environment.systemPackages = with pkgs; [ vim ]; + + system.stateVersion = "23.11"; +} +``` + +### Building the system + +Now that we have a NixOS system configuration, we can use the NixOS machinery +to turn this into a system. + +```text +# Assuming that 'nixpkgs' and 'system-configuration.nix' are in the current directory: +$ nix-build 'nixpkgs/nixos' -A system --arg configuration ./system-configuration.nix +# The ./ before system-configuration.nix is important to ensure Nix interprets +# it as a path rather than a string. +[... a lot of output later ...] +building '/nix/store/125c1x5n9lgcx4gf7wswdy4m8kawmf4i-etc.drv'... +building '/nix/store/g2mlr67i0z45n695wvc5rsnpiqcmahzk-nixos-system-nixos-inside-xen-23.05pre-git.drv'... +/nix/store/7cxgpci704yqcgqizv5ih5b47n9ckmg9-nixos-system-nixos-inside-xen-23.05pre-git +``` + +The final line output (which also gets written to the current directory as a +symlink named `result`) is the Nix store path to the system you've just built. + +Now we have the system, we can start making the disk image for Xen. + +## Making the disk image + +### Formatting a disk + +For my purposes, I'm going to make a 10GB disk image with a single ext4 +partition. If you want to do something different, you'll need to adjust +the NixOS configuration above accordingly, and rebuild it. + +```text +# Make a 10G empty file. +$ truncate --size 10G nixos.img + +# Create a single partition. +$ fdisk nixos.img + +Welcome to fdisk (util-linux 2.38.1). +Changes will remain in memory only, until you decide to write them. +Be careful before using the write command. + +Device does not contain a recognized partition table. +Created a new DOS (MBR) disklabel with disk identifier 0x3980dcd5. + +Command (m for help): n +Partition type + p primary (0 primary, 0 extended, 4 free) + e extended (container for logical partitions) +Select (default p): p +Partition number (1-4, default 1): +First sector (2048-20971519, default 2048): +Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-20971519, default 20971519): + +Created a new partition 1 of type 'Linux' and of size 10 GiB. + +Command (m for help): w +The partition table has been altered. +Syncing disks. + +# Make a partitioned loop device. +$ sudo losetup --find --partscan --show nixos.img +/dev/loop0 + +# Format the new partition with ext4. +$ sudo mkfs.ext4 -L NIXOS /dev/loop0p1 +mke2fs 1.47.0 (5-Feb-2023) +Discarding device blocks: done +Creating filesystem with 2621184 4k blocks and 655360 inodes +Filesystem UUID: 77a3e697-e6b0-4645-99f3-47528256d47b +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 + +Allocating group tables: done +Writing inode tables: done +Creating journal (16384 blocks): done +Writing superblocks and filesystem accounting information: done +``` + +### Installing NixOS + +We'll need the disk mounted, so I'll do that first: + +```text +$ mkdir mnt +$ sudo mount /dev/loop0p1 mnt +``` + +Now actually installing the system is as simple as using `nix` to copy it: + +```text +$ sudo $(which nix) copy --to $(readlink -f mnt) ./result --no-check-sigs +[... progress output ...] + +# Now there's a "nix/store" inside the disk. +$ ls mnt/nix +store var +$ ls mnt/nix/store | wc -l +503 +``` + +We can now set this as the current system inside the image - this mostly just +creates a symlink, but this sets up the generational semantics of NixOS. + +```text +$ sudo nix-env -p mnt/nix/var/nix/profiles/system --set ./result +$ ls mnt/nix/var/nix/profiles -l +total 8 +drwxr-xr-x 2 root root 4096 Aug 19 14:49 per-user +lrwxrwxrwx 1 root root 13 Aug 19 14:59 system -> system-1-link +lrwxrwxrwx 1 root root 86 Aug 19 14:59 system-1-link -> /nix/store/7cxgpci704yqcgqizv5ih5b47n9ckmg9-nixos-system-nixos-inside-xen-23.05pre-git +``` + +As you can see, this has created the `system` link, which points at the current +generation (`system-1-link`), which finally points at the actual system. + +For our convenience, we'll copy the configuration into the disk image. The +conventional location is `/etc/nixos/configuration.nix`. This isn't strictly +necessary to have a working system, but it makes upkeep much easier. + +```text +$ sudo mkdir -p mnt/etc/nixos +$ sudo cp system-configuration.nix /etc/nixos/configuration.nix +``` + +For better or for worse, we'll also set up the NixOS channel definition. + +```text +$ sudo mkdir mnt/root +$ echo "https://nixos.org/channels/nixos-23.05 nixos" | sudo tee mnt/root/.nix-channels +https://nixos.org/channels/nixos-23.05 nixos +``` + +OK, we're done with the disk image now, so we can unmount it: + +```text +$ sudo umount mnt +$ sudo losetup -d /dev/loop0 +``` + +## Booting this in Xen + +We now have a disk image with NixOS installed. We don't need to copy the kernel +or ramdisk out of it because we already have it on the host. For longer term use, +though, I suggest using Xen's built in grub emulator or similar to make sure that +things are kept up to date. This will boot the same system configuration every +time. + +I wrote this into my `domain.conf` - you'll need to substitute the path to +your own Nix system, and your `nixos.img`. + +```ini +kernel = "/nix/store/7cxgpci704yqcgqizv5ih5b47n9ckmg9-nixos-system-nixos-inside-xen-23.05pre-git/kernel" +ramdisk = "/nix/store/7cxgpci704yqcgqizv5ih5b47n9ckmg9-nixos-system-nixos-inside-xen-23.05pre-git/initrd" +memory = 2048 +name = "nixos" +vif = [ '' ] +dhcp = "dhcp" +cmdline = "xencons=tty init=/nix/store/7cxgpci704yqcgqizv5ih5b47n9ckmg9-nixos-system-nixos-inside-xen-23.05pre-git/init" +disk = ['/home/lukegb/nix-blogpost/nixos.img,,hda'] +``` + +And now we can boot the system: + +```text +$ sudo xl create ./domain.conf -c +[... it boots ...] + + +<<< Welcome to NixOS 23.05pre-git (x86_64) - hvc0 >>> + +Run 'nixos-help' for the NixOS manual. + +nixos-inside-xen login: +``` + +You can then log in with the `user` / `thisisinsecure` pair. The `user` user +has sudo permission, and when you're done you can shut the VM down. Assuming +that your Xen system is set up similar to mine, with a xenbr0 that has internet +access with a DHCP server (and hopefully even IPv6...!), then you should get +an IP address inside the VM as well. + +### Updating the nixpkgs channel + +The first thing you'll probably want to do is actually fetch the Nix channel +that we configured while creating the disk image: + +```text +[user@nixos-inside-xen:~]$ sudo nix-channel --update +unpacking channels... +``` + +Now you can freely experiment with Nix. + +## Bonus round: Making a trivial configuration change + +This is some bonus work, to explore making a NixOS configuration change. You +already have a working system at this point so you can stop reading. + +As an example, say you want to have `mtr` available inside your system. It's +not installed by default: + +```text +[user@nixos-inside-xen:~]$ mtr --report 8.8.8.8 +The program 'mtr' is not in your PATH. It is provided by several packages. +You can make it available in an ephemeral shell by typing one of the following: + nix-shell -p mtr + nix-shell -p mtr-gui +``` + +You can follow the instructions and use `nix-shell` to provide it temporarily: + +```text +[user@nixos-inside-xen:~]$ nix-shell -p mtr +[... nix downloads mtr from cache ...] +[nix-shell:~]$ mtr --report 8.8.8.8 +Start: 2023-08-19T15:10:53+0100 +HOST: nixos-inside-xen Loss% Snt Last Avg Best Wrst StDev + 1.|-- _gateway 0.0% 10 0.6 0.5 0.5 0.6 0.0 + 2.|-- tuvok.gnet-tuvok.mldn-rd. 20.0% 10 2.0 2.2 2.0 2.6 0.2 + 3.|-- blade-tuvok.public.as2054 0.0% 10 2.5 2.4 1.9 2.5 0.2 + 4.|-- 195.66.224.125 0.0% 10 2.7 3.1 2.5 5.3 0.9 + 5.|-- 74.125.242.97 0.0% 10 4.7 4.7 4.2 7.2 0.9 + 6.|-- 192.178.46.87 0.0% 10 3.6 3.6 3.2 4.7 0.4 + 7.|-- dns.google 0.0% 10 3.2 3.4 3.0 3.7 0.2 +``` + +except it won't quite work properly, because its `mtr-packet` process isn't +privileged: + +```text +[nix-shell:~]$ mtr --udp --report 8.8.8.8 +Start: 2023-08-19T15:13:03+0100 +HOST: nixos-inside-xen Loss% Snt Last Avg Best Wrst StDev + +[nix-shell:~]$ exit +[user@nixos-inside-xen:~]$ +``` + +You can solve this by installing it in your NixOS configuration using the +`programs.mtr.enable` configuration option, defined +[in this module](https://github.com/NixOS/nixpkgs/blob/nixos-23.05/nixos/modules/programs/mtr.nix). +This both installs the package (into `environment.systemPackages`), but +also installs a setcap wrapper for `mtr-packet`. + +Edit `/etc/nixos/configuration.nix`, and add `programs.mtr.enable = true;` +somewhere, probably just below the `environment.systemPackages` line. + +Once you've done that, rebuild and switch to the new system: + +```text +[user@nixos-inside-xen:~]$ sudo nixos-rebuild switch +building Nix... +building the system configuration... +these 11 derivations will be built: + /nix/store/c9nkzphpp4hfgbx4da8pfh4b6xcfwy1s-system-path.drv + /nix/store/0bm8y3d51c93dr27jkzzaaz0kv43vv62-unit-systemd-fsck-.service.drv + /nix/store/k984vii2lsh7yjz8acvbfqgph0vyssfj-dbus-1.drv + /nix/store/j9lmndz2by4ridi3vyzbd32mnay2pn4q-X-Restart-Triggers.drv + /nix/store/d0j9nmy6qylmgllaw21l6w11b0gkhxkx-unit-dbus.service.drv + /nix/store/1cjakrjkp764r26fddkpixjp6cff0kq7-user-units.drv + /nix/store/8l53jy12sbls3ppxc7pqh40ialnw53x4-unit-dbus.service.drv + /nix/store/vnky6khlsj4zasp4q8g6rwp2b3jhp99l-system-units.drv + /nix/store/d5zmdd9g1zvv7f5s9w2n3praq8d4k1p6-etc.drv + /nix/store/qixzqgmf54k8xypajycry4v42llhhwi1-ensure-all-wrappers-paths-exist.drv + /nix/store/hp0dhaw2rgcpa7y70rfhnmnr2qiw8lm2-nixos-system-nixos-inside-xen-23.05.2891.ae521bd4e460.drv +[...a little bit of output...] +building '/nix/store/hp0dhaw2rgcpa7y70rfhnmnr2qiw8lm2-nixos-system-nixos-inside-xen-23.05.2891.ae521bd4e460.drv'... +Warning: do not know how to make this configuration bootable; please enable a boot loader. +activating the configuration... +setting up /etc... +reloading user units for user... +setting up tmpfiles +reloading the following units: dbus.service +``` + +...and you now have `mtr` available system-wide, but now it works in UDP mode: + +```text +[user@nixos-inside-xen:~]$ mtr --udp --report 8.8.8.8 +Start: 2023-08-19T15:19:03+0100 +HOST: nixos-inside-xen Loss% Snt Last Avg Best Wrst StDev + 1.|-- _gateway 0.0% 10 0.5 0.5 0.4 0.6 0.0 + 2.|-- tuvok.gnet-tuvok.mldn-rd. 90.0% 10 2.3 2.3 2.3 2.3 0.0 + 3.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 + 4.|-- 195.66.224.125 0.0% 10 2.8 2.8 2.5 3.3 0.3 + 5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 + +```