Move Nix tool store dir

[TheColorman]

Nix is a package manager and builder, and has static references to the entire build and runtime dependency trees in the Nix store, which is placed at /nix/store by default. Since these paths are static, all instances of Nix must use /nix/store as the store directory to take advantage of caching.

The Nix tool in containerbase changes the nix store to $(get_cache_path)/nix/store, making it practically unusable as the entire dependency tree needs to be built from source (see this explanation in a fork of renovate).

This can be solved in 2 ways:

• Use the canonical /nix/store path for the Nix tool.
• Use a local store, which makes Nix to use chroot to place the target directory at the chroot's /nix/store path. This requires mount and user namespaces.

[viceice]

This nix tool is only created for renovate usage, not for general nix usage.

We probably accept a PR which does not set these env values if they're already set.

Maybe we can also symlink /nix ? Or do i remember correctly and nix will refuse the symlink?

[jamietanna]

When you say:

making it practically unusable as the entire dependency tree needs to be built from source

What are the practical issues here? Is this affecting existing Renovate functionality, when attempting to update Nix packages, or only your nix-update Manager?

(I'm unsure if the plan is to contribute that nix-update Manager to Renovate, or if you're intending to keep it in your fork)

[TheColorman]

This nix tool is only created for renovate usage, not for general nix usage.

The use-case here is updating Nix hashes when updating dependencies.

To ensure reproducibility, Nix requires the hashes of anything downloaded off the internet to be part of the source code, if you want to package and run your program with Nix. In practice, if you're making an NPM-based program and want to package it with Nix, Nix will try to download all NPM dependencies directly, and if the hash of all the downloaded deps doesn't match what's in your Nix source, the build fails.

The solution to this is usually running nix-update - it downloads the deps for you, calculates their hash, and updates the Nix source. This can be done as part of a Renovate postUpgradeTask to ensure it is kept up to date after Renovate bumbs a dependency.

(This is important for me specifically as I have CI that verifies the Nix package is still valid and which must succeed for a PR to be merged. When Renovate updates NPM deps without updating the Nix hash, the Nix package is no longer valid).

(I'm unsure if the plan is to contribute that nix-update Manager to Renovate, or if you're intending to keep it in your fork)

Actually running nix-update is out of scope for this issue, but it can be done by nix run nixpkgs#nix-update. And to clarify, I'm not the author of the fork, but just stumbled across it while investigating this issue.

What are the practical issues here?

To update the dependency hash, Nix essentially needs to attempt a full build of your project. Nix is an isolated ecosystem, and can thus only execute binaries compiled by Nix ecosystem itself. So, with zero caching, if you wanted to update the dependency hash, you first need to compile the entire dependency tree from source (bash, coreutils, glibc, python, sqlite, etc.).

To expand on the reason containerbase can't make use of caching:
Binaries built with Nix will be linked directly to paths in the store dir, usually /nix/store:

❯ ldd /sbin/bash
        linux-vdso.so.1 (0x00007dbcb241e000)
        libreadline.so.8 => /nix/store/4myf3s1i9rahd2my1zs2cqify7y930sk-readline-8.3p3/lib/libreadline.so.8 (0x00007dbcb22a0000)
        libhistory.so.8 => /nix/store/4myf3s1i9rahd2my1zs2cqify7y930sk-readline-8.3p3/lib/libhistory.so.8 (0x00007dbcb2292000)
        libncursesw.so.6 => /nix/store/zlvs6miv8wfki399pmxri7x0sjd3429c-ncurses-6.6/lib/libncursesw.so.6 (0x00007dbcb2216000)
        libdl.so.2 => /nix/store/0d8g8n0a11v6f5m2h416ajyxmnkwc3md-glibc-2.42-67/lib/libdl.so.2 (0x00007dbcb2211000)
        libc.so.6 => /nix/store/0d8g8n0a11v6f5m2h416ajyxmnkwc3md-glibc-2.42-67/lib/libc.so.6 (0x00007dbcb2000000)
        /nix/store/0d8g8n0a11v6f5m2h416ajyxmnkwc3md-glibc-2.42-67/lib/ld-linux-x86-64.so.2 => /nix/store/0d8g8n0a11v6f5m2h416ajyxmnkwc3md-glibc-2.42-67/lib64/ld-linux-x86-64.so.2 (0x00007dbcb2420000)

These paths are directly part of the Nix derivation for bash, which is essentially a recipe with instructions for building bash, along with the Nix store paths required. Nix will then query known binary caches for these paths directly. In practice, this means asking https://cache.nixos.org whether it has a cached build of /nix/store/4myf3s1i9rahd2my1zs2cqify7y930sk-readline-8.3p3/ and promptly downloading and caching it locally.

If the store dir is not /nix/store, https://cache.nixos.org will respond that it does not have the file, and Nix will try to build it locally from source.

---

While I haven't tested it, I'm pretty sure this could be solved by removing the override for the NIX_STORE_DIR variable, and instead passing --store $(get_cache_path)/nix/store to allow Nix to use chroot to place it at /nix/store (Though I'm not familiar enough with containerbase to know how to add that flag to all tool invocations).

Maybe we can also symlink /nix ? Or do i remember correctly and nix will refuse the symlink?

I'm not too familiar with Nix internals, but IIRC symlinks don't work but bind mounts do.