So, the same good friend that put me onto vimwiki and taskwiki
mentioned he'd recently been getting into something called "Nix".
DANGER: If you don't have a lot of spare time, don't even start attempting to "get into something called Nix".
I needed to rebuild my laptop, and also a Pi, so this seemed an ideal opportunity to figure out what all this was about.
This isn't going to be a tutorial or lesson on nix - there are lots of those, that are better than anything I'd write. Here lie a few things I had to figure out along the way in case they help anyone. Especially if you are suddenly taken by the feeling that you've made a mistake and your life is draining away in front of your eyes.
Goal: Imaging a Pi #
Data aside, make my Pi throwaway, in such a way that it's
configuration is in git somewhere, and I can recreate it trivially.
NixOS - i.e. using Nix to define and manage your entire OS seemed ideal for this. Have a look at the example config
- it sorts everything out for you, amazeballs.
Hurdle 0: Getting off on the wrong foot #
The Nix Website sums it up pretty well on the home page:
Nix is a tool that takes a unique approach to package management and system configuration.
Note the unique, I didn't the first few times.
If you're used to package managers like pip, conan, npm, go,
brew, etc. It's not that kind of package manager.
If you're used to other kinds of configuration management tools, it's probably not like those either.
A lot of things you expect may work somewhat differently. You kind of have to spend some time with it to work out where it might work for you. Be prepared to spend a lot of time.
Hurdle 1: Headless #
Pretty much every NixOS install tutorial starts with implicit installation steps. That kind of defeated the point for me, and I don't have a monitor anywhere near my Pi.
Nix also has the ability to build images though. Whoop. We can build an
.img of the configured Pi with nix on my mac, pop the card in and
off we go.
Somehow, I found my way here, under the "Build your own image natively" heading.
Hurdle 2: Flake or not to flake #
Nix is unstable. If you use the current-at-the-time-of-writing release,
there are two different ways to do most things. For example, there is
nix-shell and nix shell.
The short of it is that Nix seems to going through a rewrite and a clean up. Right now, 80% of what you read on the interwebs will be for the "old" way of doing things, another 10% will be people moaning that the new stuff is different. I can understand why, as they've probably invested a significant chunk of their lives learning how the old way works, and change is well, change.
"Flakes" are the new thing. To me, they're a great evolution - as they solve a bunch of fluffier aspects of how nix figures out what versions of things to use.
So, let's go with flakes. This reduces the breadth of helpful interweb based guides, but is a safer footing for the future.
Hurdle 3: Cross compiling #
My mac is aarch64-darwin in nix terms, a Pi 4 is aarch64-linux. The
same arch is helpful, but don't be deceived. The above mentioned doc
suggests this is all we might need:
1 nixpkgs.crossSystem.system = "aarch64-linux";
Ymmv, but for me, this (and qemu) didn't work in all sorts of ways.
Mostly:
error: a 'aarch64-linux' with features {} is required to build
'/nix/store/ds3174h0ycsd54013k86j8xh896gmhi2-healthcheck-commands.txt.drv',
but I am a 'aarch64-darwin' with features {benchmark, big-parallel,
nixos-test}
James does a much better job of describing these woes, and some solutions.
The whole remote builder route seems a lot of complexity though.
Fortunately, if you don't need CPU emulation (ie. its all aarch64 in
this case), there is a much easier way - https://multipass.run.
Spin up an Ubuntu VM, install nix, it just works. Multipass mounts
your home dir under ~/Home by default, so data access is trivial.
Hurdle 4: NixOS image != NixOS install #
Once you have your image flashed, and log into your system, you'll
notice the conspicuous lack of /etc/nixos/configuration.nix.
Nothing you write in your flake that defines the system configuration exists outside of the flake.
That's because (afaict) the images are intended to be the final state of the machine, rather than a dynamic NixOS system. Fair enough. Given nix is a system configuration management system, there must be a way to solve this.
Modules #
Nix lets you split things out into separate modules (files). Maybe we
can use this to include our future configuration.nix into the image
build script? You can!
1{
2 description = "Home Pi Build image";
3 inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-22.11";
4 outputs = { self, nixpkgs }: rec {
5
6 nixosConfigurations.homePi = nixpkgs.lib.nixosSystem {
7
8 system = "aarch64-linux";
9
10 modules = [
11
12 "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
13
14 # Ensure the resulting system has nixpkgs available
15 "${nixpkgs}/nixos/modules/installer/cd-dvd/channel.nix"
16
17 # Define the final config via modules, so they can be included
18 # in confguration.nix in the final image, to allow nixos-rebuild
19 # to be used on the live machine.
20 ./nixos/modules/system.nix
21 ./nixos/modules/services.nix
22 ./nixos/modules/packages.nix
23
24 {
25 # Copy the config over if needed during rebuild.
26 # Nothing defined in this flake is added by default, so the
27 # machine can't be rebuilt live, environment.etc results in
28 # read-only symlinks to the store, so we have to use an
29 # activationScript so they will be normal/editable files.
30 system.activationScripts.installConfiguration =
31 ''
32 if [ ! -f /etc/nixos/configuration.nix ]; then
33 mkdir -p /etc/nixos
34 cp -r ${./nixos}/* /etc/nixos
35 fi
36 '';
37 }
38 ];
39 };
40
41 packages.aarch64-linux.default = images.homePi;
42 images.homePi = nixosConfigurations.homePi.config.system.build.sdImage;
43 };
44}
This took a lot of finessing, these are they key bits that differ from a standard image building flake:
1# Ensure the resulting system has nixpkgs available
2"${nixpkgs}/nixos/modules/installer/cd-dvd/channel.nix"
Without this, the resulting system has no nix channels configured, and
so none of the nix commands will work.
1# Define the final config via modules, so they can be included
2# in confguration.nix in the final image, to allow nixos-rebuild
3# to be used on the live machine.
4./nixos/modules/system.nix
5./nixos/modules/services.nix
6./nixos/modules/packages.nix
This includes the actual configuration (these are just arbitrary splits I
made, you can structure these how ever you like). They're just simple
nix expressions, e.g packages.nix is just:
1{ pkgs, ... }: {
2 environment.systemPackages = with pkgs; [
3 git
4 neovim
5 tmux
6 ];
7}
And now for the last piece of the puzzle:
1{
2 # Copy the config over if needed during rebuild.
3 # Nothing defined in this flake is added by default, so the
4 # machine can't be rebuilt live, environment.etc results in
5 # read-only symlinks to the store, so we have to use an
6 # activationScript so they will be normal/editable files.
7 system.activationScripts.installConfiguration =
8 ''
9 if [ ! -f /etc/nixos/configuration.nix ]; then
10 mkdir -p /etc/nixos
11 cp -r ${./nixos}/* /etc/nixos
12 fi
13 '';
14}
This one took a bit of figuring. We need to copy the config into the
image. This seems to be the only practical way to get it to work.
Thanks to the folks on Matrix for getting me onto the
activationScripts mechanism.
Activation scripts run on the image when it boots. This one simply
checks to see if it is out "first boot" - i.e. no /etc/nixos yet as
the image builder doesn't create it - and copies in the config we used
to build the image.
The real trick is this line:
1 cp -r ${./nixos}/* /etc/nixos
${./nixos} is where the magic happens. This is parsed by the image
build process - because this is a path reference, it is automatically
copied into the store in the resulting image. The store path to the
directory is then put into the actual activation script!
All we have to do now is make our flake local ./nixos directory match
what we want on the final machine. This bit is a little fiddly.
You need two things for a working /etc/nixos, primarily a
configuration.nix, which by default, includes a
hardware-configuration.nix file. How the heck do we know what to put
in that? Fortunately there is nixos-generate-config.
So all we do is build the image once, without the activation script, run
the cmd, and extract the generated config.nix and
hardware-configuration.nix.
When I did this, it was referencing the boot partition via UUID. It's probably worth switching this to a label for portability:
1 fileSystems."/" =
2 { device = "/dev/disk/by-label/NIXOS_SD";
3 fsType = "ext4";
4 };
We also need to include the same config modules we use in the image
builder in configuration.nix:
1 imports = [
2 ./hardware-configuration.nix
3 ./modules/system.nix
4 ./modules/services.nix
5 ./modules/packages.nix
6 ];
Putting all this together, and you should be good!
We have this tree:
1|____flake.nix
2|____nixos
3 |____configuration.nix
4 |____hardware-configuration.nix
5 |____modules
6 |____services.nix
7 |____system.nix
8 |____packages.nix
We can now build and flash our image:
1flake (master) % multipass shell
2Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-60-generic aarch64)
3Last login: Thu Feb 2 00:22:28 2023 from 192.168.64.1
Now we're in a linux VM, our build will be happy compiling
aarch64-linux, then we unpack the result to an actual img:
1ubuntu@primary:~$ cd Home/path/to/flake
2$ nix build
3[1/18/20 built, 58 copied (149.4/149.7 MiB), 25.3 MiB DL] building ext4-fs.img.zst: Creating journal (16384 blocks): done
4$ sudo unzstd result/sd-image/nixos-sd-image-22.11.20230128.ce20e9e-aarch64-linux.img.zst -o pi.img
We can now head back to mac land and flash the card:
Don't forget to update the of= dev for dd to the right one!
1$ exit
2logout
3flake (master) % dd if=./pi.img of=/dev/disk4
Final thoughts #
It took a long time to get here, with a lot of dead ends, but I can see this will be quite useful. Being able to make a stable image, but iterate it in context without having to re-image the machine each time really speeds up setup.
All in all though, I'm parking nix for other use cases. I just don't really get on with how it versions packages, and for managing my daily driver, I find the simplicity of my own simple declarative tool manager far less vexing. More on which another day. Nix is great, and makes the happy path very easy, but I found straying can be incredibly exhausting!