Nix for Advanced Beginners: The Nix Store

Nix sometimes gets a reputation for being confusing. In this article I’m trying to articulate certain ideas that, once I understood them, made Nix much easier to understand.

Nix is a bunch of technologies and projects that build on top of each other

If you google for source code of nix you get the NixOS/nix repository on Github which dubs itself the purely functional package manager. This becomes confusing as you get deeper into Nix.

First, package management in Nix is a relatively high level concept that brings together many of the ideas of Nix. If you start off by copying and pasting commands as though it was any other package manager you’ll get pretty far, but eventually the lower level concepts will creep into the package management abstraction and you won’t understand the error messages.

Second, the code in this repository covers a lot of ground. The nix command has a lot of subcommands to interact with different parts of nix and at different layers of abstraction. Most users won’t need most of the commands.

Third, there’s more commits and effort going into nixpkgs, a different and arguably more important repository.

So what gives?

A word on flakes

There’s two ways to use nix: flakes or no flakes. Flakes is a different way of using the same basic concepts and is widely used despite being marked experimental. Even though flakes and non flakes can work side by side without issue when starting nix you generally want to choose one or the other.

There’s another experimental feature called Nix Command that goes hand-in-hand with flakes, and provides subcommands of nix that do the same thing as various standalone utilities but work differently. I do not know the full background of why this has happened but the subcommands sometimes do the exact same thing as the standalone command and sometimes work completely differently.

For instance nix-build builds a default.nix (which is a non-flakes way of doing things) and nix build builds a flake.nix. You cannot just switch back and forth between these commands. As far as I know you cannot build a default.nix with nix build and you cannot build a flake.nix with nix-build.

On the other hand, nix store gc and nix-store --gc, as far as I can tell, do exactly the same things.

All the concepts I plan to talk about here, those relating to the nix store, live in abstractions below the distinction of flakes and non flakes. In other words, these are concepts shared between both ways of using flakes.

The fundamental building block of Nix: The Nix Store

The basic building block of Nix is the Nix Store which is a database of objects that lives on-disk and where items are stored and looked up by a hash. Unlike git, this database is not a Merkle tree. The hashes objects are stored under can be either the hash of the contents or (more commonly) the hash of instructions to build it.

An object is referenced by a store path. Here is an example of the store path of an object in nix: /nix/store/crrf4bysvarqp0g9pyhcrj47689g4i5l-hello-2.12.1.

Since it’s always the same I’ll elide the store directory, /nix/store/, when discussing objects in the future.¹ The object then has a digest or an irreversible hash and a name, which is required to reference the object and exists to keep humans working with nix sane.

The digest is calculated in multiple steps: first a “fingerprint” is calculated which is the concatenation of relevant information (including hashes) then that’s hashed into a digest. There’s no extractable information in the digest – for instance there’s no way to look at the digest and determine if it’s the hashed content of the object or a hash of the instructions.

Store objects can be files or directories, but nix has a simplified model of files and directories to most filesystems. A file is either a sequence of bytes plus a single bit of metadata for the executable bit or a symlink which is effectively just a string but is almost always a path pointing to an object in the store. A directory maps strings to files and other directories. This means fields like last updated, date created, is readable, owner are not part of the data for that object.

Nix is very intentional about how it sets this unused metadata. Dates are set to one second past the epoch to indicate they shouldn’t be relied on. All files are marked read-only and world-readable and, on Linux, are usually owned by root. This last part has implications for security: nix store objects are designed to be public (at least, to the system) and secrets shouldn’t be stored there.

Stepping back, a very important consequence of this design is that two store paths that exist on two separate valid Nix systems will always be the same. And if you delete and rebuild that store path, or get that store path from a cache it should be the same². Which is how caching works! A nix cache is just a server that tells you where to find an object given the store path.³

Objects in the nix store can depend on other objects in the nix store. Nix checks what dependencies an object has by recursively scanning all files and symlinks to see if there’s a nix store path.

$ nix-store --query --tree /nix/store/crrf4bysvarqp0g9pyhcrj47689g4i5l-hello-2.12.1
/nix/store/crrf4bysvarqp0g9pyhcrj47689g4i5l-hello-2.12.1
├───/nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36
│   ├───/nix/store/2d5spnl8j5r4n1s4bj1zmra7mwx0f1n8-xgcc-13.3.0-libgcc
│   ├───/nix/store/qwjjm4j652ck9izaid7bz63s4hd5bnha-libidn2-2.3.7
│   │   ├───/nix/store/6pqgj71r0850b0cd95yxx0d52zax016i-libunistring-1.2
│   │   │   └───/nix/store/6pqgj71r0850b0cd95yxx0d52zax016i-libunistring-1.2 [...]
│   │   └───/nix/store/qwjjm4j652ck9izaid7bz63s4hd5bnha-libidn2-2.3.7 [...]
│   └───/nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36 [...]
└───/nix/store/crrf4bysvarqp0g9pyhcrj47689g4i5l-hello-2.12.1 [...]

$ rg -F --binary /nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36 /nix/store/crrf4bysvarqp0g9pyhcrj47689g4i5l-hello-2.12.1
/nix/store/crrf4bysvarqp0g9pyhcrj47689g4i5l-hello-2.12.1/bin/hello: binary file matches (found "\0" byte around offset 7)

In this case nix knows glibc-2.40-36 is a dependency of hello-2.12.1 because the bytes /nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36 exist in some file or symlink in hello-2.12.1.

Note that all references must be to the full store path for nix to recognize them.

Adding Objects to the Nix Store

Objects in the nix store that do not come from caches are built from derivations. The derivation files live in the nix store next to the nix store. Here’s /nix/store/vc5qrxldmn41i28la7x94ysqqpq85ijh-hello-2.12.1.drv⁴:

Derive([("out","/nix/store/crrf4bysvarqp0g9pyhcrj47689g4i5l-hello-2.12.1","","")],[("/nix/store/2pllgal9i5wjfqi2h2asrr74y5rbzarn-stdenv-linux.drv",["out"]),("/nix/store/7k0msqyp2dm021sdj0qjgpkzff8xhqzr-bash-5.2p37.drv",["out"]),("/nix/store/a5cyil7j6w4r4v782f4rzyz88z2fm90h-version-check-hook.drv",["out"]),("/nix/store/naxsjszg7hmqy2k64la88d15zj21nf8c-hello-2.12.1.tar.gz.drv",["out"])],["/nix/store/v6x3cs394jgqfbi0a42pam708flxaphh-default-builder.sh"],"x86_64-linux","/nix/store/gwgqdl0242ymlikq9s9s62gkp5cvyal3-bash-5.2p37/bin/bash",["-e","/nix/store/v6x3cs394jgqfbi0a42pam708flxaphh-default-builder.sh"],[("__structuredAttrs",""),("buildInputs",""),("builder","/nix/store/gwgqdl0242ymlikq9s9s62gkp5cvyal3-bash-5.2p37/bin/bash"),("cmakeFlags",""),("configureFlags",""),("depsBuildBuild",""),("depsBuildBuildPropagated",""),("depsBuildTarget",""),("depsBuildTargetPropagated",""),("depsHostHost",""),("depsHostHostPropagated",""),("depsTargetTarget",""),("depsTargetTargetPropagated",""),("doCheck","1"),("doInstallCheck","1"),("mesonFlags",""),("name","hello-2.12.1"),("nativeBuildInputs","/nix/store/bgl23gsw7i5cnrsfd6clm5ygva1k53zw-version-check-hook"),("out","/nix/store/crrf4bysvarqp0g9pyhcrj47689g4i5l-hello-2.12.1"),("outputs","out"),("patches",""),("pname","hello"),("postInstallCheck","stat \"${!outputBin}/bin/hello\"\n"),("propagatedBuildInputs",""),("propagatedNativeBuildInputs",""),("src","/nix/store/pa10z4ngm0g83kx9mssrqzz30s84vq7k-hello-2.12.1.tar.gz"),("stdenv","/nix/store/lzrs17sc8bhi87nb1y1q1bas73j6q10y-stdenv-linux"),("strictDeps",""),("system","x86_64-linux"),("version","2.12.1")])

This file contains the instructions to build hello on system type x86_64-linux (by default nix will fail to build on other system types, since potentially it could lead to different outputs or, more likely, fail). The exact format doesn’t matter, but note it includes build inputs that are the output s not dependent on. For instance it references a src hello-2.12.1.tar.gz, another object in this nix store. This is a very simple derivation file because it basically says, do the default and doesn’t include special instructions. The default instructions are in default-builder.sh in the nix store.

What’s not obvious is these derivation files can have multiple outputs. Usually there’s only one output called out but sometimes a derivation will multiple output objects, for example the main program and library files.

The digest of hello-2.12.1 is of its build instructions, not a hash of the output. Most derivations are like this because it allows you to reference another object by its build instructions without building it. In other words, you know where an object lives before you build it.

The hello-2.12.1.tar.gz object hashed by it’s output. Here’s the derivation,

Derive([("out","/nix/store/pa10z4ngm0g83kx9mssrqzz30s84vq7k-hello-2.12.1.tar.gz","sha256","8d99142afd92576f30b0cd7cb42a8dc6809998bc5d607d88761f512e26c7db20")],[("/nix/store/1q61w81j86s12d3c4z8y6hf2cabkq2gr-curl-8.11.1.drv",["dev"]),("/nix/store/7k0msqyp2dm021sdj0qjgpkzff8xhqzr-bash-5.2p37.drv",["out"]),("/nix/store/vn7dmv1kcajxk88sndf9hl9dz48bxvgz-mirrors-list.drv",["out"]),("/nix/store/ycj0m56p8b0rv9v78mggfa6xhm31rww3-stdenv-linux.drv",["out"])],["/nix/store/g0gn91m56b267ncx05w93kihyqia39cm-builder.sh"],"x86_64-linux","/nix/store/gwgqdl0242ymlikq9s9s62gkp5cvyal3-bash-5.2p37/bin/bash",["-e","/nix/store/g0gn91m56b267ncx05w93kihyqia39cm-builder.sh"],[("SSL_CERT_FILE","/no-cert-file.crt"),("__structuredAttrs",""),("buildInputs",""),("builder","/nix/store/gwgqdl0242ymlikq9s9s62gkp5cvyal3-bash-5.2p37/bin/bash"),("cmakeFlags",""),("configureFlags",""),("curlOpts",""),("curlOptsList",""),("depsBuildBuild",""),("depsBuildBuildPropagated",""),("depsBuildTarget",""),("depsBuildTargetPropagated",""),("depsHostHost",""),("depsHostHostPropagated",""),("depsTargetTarget",""),("depsTargetTargetPropagated",""),("doCheck",""),("doInstallCheck",""),("downloadToTemp",""),("executable",""),("impureEnvVars","http_proxy https_proxy ftp_proxy all_proxy no_proxy HTTP_PROXY HTTPS_PROXY FTP_PROXY ALL_PROXY NO_PROXY NIX_SSL_CERT_FILE NIX_CURL_FLAGS NIX_HASHED_MIRRORS NIX_CONNECT_TIMEOUT NIX_MIRRORS_alsa NIX_MIRRORS_apache NIX_MIRRORS_bioc NIX_MIRRORS_bitlbee NIX_MIRRORS_centos NIX_MIRRORS_cpan NIX_MIRRORS_cran NIX_MIRRORS_debian NIX_MIRRORS_dub NIX_MIRRORS_fedora NIX_MIRRORS_gcc NIX_MIRRORS_gentoo NIX_MIRRORS_gnome NIX_MIRRORS_gnu NIX_MIRRORS_gnupg NIX_MIRRORS_hackage NIX_MIRRORS_hashedMirrors NIX_MIRRORS_ibiblioPubLinux NIX_MIRRORS_imagemagick NIX_MIRRORS_kde NIX_MIRRORS_kernel NIX_MIRRORS_luarocks NIX_MIRRORS_maven NIX_MIRRORS_mozilla NIX_MIRRORS_mysql NIX_MIRRORS_openbsd NIX_MIRRORS_opensuse NIX_MIRRORS_osdn NIX_MIRRORS_postgresql NIX_MIRRORS_pypi NIX_MIRRORS_qt NIX_MIRRORS_sageupstream NIX_MIRRORS_samba NIX_MIRRORS_savannah NIX_MIRRORS_sourceforge NIX_MIRRORS_steamrt NIX_MIRRORS_tcsh NIX_MIRRORS_testpypi NIX_MIRRORS_ubuntu NIX_MIRRORS_xfce NIX_MIRRORS_xorg"),("mesonFlags",""),("mirrorsFile","/nix/store/lfwvcxbhmqnpmqcxbpd5bwx3bhkpjgci-mirrors-list"),("name","hello-2.12.1.tar.gz"),("nativeBuildInputs","/nix/store/s569mjxw77rh0lwll868apq0xw5zlpyy-curl-8.11.1-dev"),("nixpkgsVersion","24.11"),("out","/nix/store/pa10z4ngm0g83kx9mssrqzz30s84vq7k-hello-2.12.1.tar.gz"),("outputHash","sha256-jZkUKv2SV28wsM18tCqNxoCZmLxdYH2Idh9RLibH2yA="),("outputHashMode","flat"),("outputs","out"),("patches",""),("postFetch",""),("preferHashedMirrors","1"),("preferLocalBuild","1"),("propagatedBuildInputs",""),("propagatedNativeBuildInputs",""),("showURLs",""),("stdenv","/nix/store/cf464y41p2x3lh1qvbg6678lc3f8zbd6-stdenv-linux"),("strictDeps",""),("system","x86_64-linux"),("urls","mirror://gnu/hello/hello-2.12.1.tar.gz")])

The instructions in this file says, essentially, use curl to download this file from a mirror. If we didn’t know the hash of this object nix would try to prevent our build scripts from using the internet, but in these types of derivations we have much more flexibility. If the hash doesn’t match at the end the build will fail.

Instantiation versus building

I mentioned earlier it’s helpful to think of nix as layers of abstraction. The most fundamental layer of abstraction is the nix store, which I’m discussing in this article. The next level of abstraction up is the nix language. This is a purpose-built language designed to instantiate derivations. This just means placing the derivation file in the nix store.

Separating these steps helps nix be lazy, since we might specify how to build certain objects, but might not actually need to build them.