mirror of
https://github.com/simplex-chat/haskell.nix.git
synced 2026-06-03 09:17:32 +00:00
John A. Lotoski
09526c8555
* Spelling and typo fixes in doc and code comments * Trailing whitespace or whitespace only truncation * readTheDocs formatting corrections
275 lines
9.6 KiB
Markdown
275 lines
9.6 KiB
Markdown
# Materialization
|
|
|
|
## What is materialization?
|
|
|
|
Capturing and storing the nix files for a project so that they do
|
|
not need to be built (or checked). This allows us to cache the input
|
|
of an IFD (import from derivation).
|
|
|
|
## Why use materialization?
|
|
|
|
Using functions like `project`, `cabalProject`, `stackProject`
|
|
and `hackage-package` results in a lot of dependencies (all the
|
|
dependencies of nix-tools for instance).
|
|
|
|
* They can be slow to calculate (even if no work needs to be done it
|
|
is not unusual for it to take 5 seconds per project).
|
|
|
|
* They can be slow to build (or download) on machines that do not
|
|
yet have them in the nix store.
|
|
|
|
* Hydra does not show progress because it does not provide feedback
|
|
until it has a list of jobs and the list of jobs cannot depends
|
|
on the nix being present (although this is often blamed on IFD
|
|
it would be the same if it wrote out JSON files and read them in)
|
|
|
|
## When is it ok to materialize?
|
|
|
|
* The nix is unlikely to change frequently (and when it does you
|
|
are happy to manually update it).
|
|
|
|
* You are happy to script something to update the materialized
|
|
nix files automatically.
|
|
* You are certain that the IFD you materialize is not `system`-dependent. If it was you'd
|
|
obtain different nix expressions depending on which `system` the IFD was evaluated.
|
|
|
|
## How can we materialize the nix files?
|
|
|
|
Lets say we want to build `hlint`. We might start with an `hlint`
|
|
file that looks like this:
|
|
|
|
```nix
|
|
let inherit (import ./. {}) sources nixpkgsArgs;
|
|
pkgs = import sources.nixpkgs nixpkgsArgs;
|
|
hlint = pkgs.haskell-nix.hackage-package {
|
|
compiler-nix-name = "ghc884";
|
|
name = "hlint";
|
|
version = "2.2.11";
|
|
};
|
|
in hlint.components.exes.hlint
|
|
```
|
|
|
|
Building this may result in a lot of output, but if you build
|
|
it again it should give just:
|
|
|
|
```
|
|
$ nix-build hlint.nix
|
|
trace: Using latest index state for hlint!
|
|
trace: Using index-state: 2020-04-15T00:00:00Z for hlint
|
|
/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
|
|
```
|
|
|
|
To materialize the nix files we need to take care to pin down the
|
|
inputs. For cabal projects this means we must specify the
|
|
`index-state` of hackage we want to use:
|
|
|
|
```nix
|
|
let inherit (import ./. {}) sources nixpkgsArgs;
|
|
pkgs = import sources.nixpkgs nixpkgsArgs;
|
|
hlint = pkgs.haskell-nix.hackage-package {
|
|
compiler-nix-name = "ghc884";
|
|
name = "hlint";
|
|
version = "2.2.11";
|
|
index-state = "2020-04-15T00:00:00Z";
|
|
};
|
|
in hlint.components.exes.hlint
|
|
```
|
|
|
|
Now if we build again we get a hint telling use how to
|
|
calculate a suitable sha256 hash to turn the derivation
|
|
containing the nix files into a fixed output derivation:
|
|
|
|
```
|
|
$ nix-build hlint.nix
|
|
trace: Using index-state: 2020-04-15T00:00:00Z for hlint
|
|
trace: Get `plan-sha256` with `nix-hash --base32 --type sha256 /nix/store/8z6p4237rin3c6c1lmjwshmj8rdqrhw2-hlint-plan-to-nix-pkgs/`
|
|
/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
|
|
|
|
$ nix-hash --base32 --type sha256 /nix/store/8z6p4237rin3c6c1lmjwshmj8rdqrhw2-hlint-plan-to-nix-pkgs/
|
|
02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4
|
|
```
|
|
|
|
We can add the hash as `plan-sha256` or (`stack-sha256` for
|
|
stack projects)
|
|
|
|
```nix
|
|
let inherit (import ./. {}) sources nixpkgsArgs;
|
|
pkgs = import sources.nixpkgs nixpkgsArgs;
|
|
hlint = pkgs.haskell-nix.hackage-package {
|
|
compiler-nix-name = "ghc884";
|
|
name = "hlint";
|
|
version = "2.2.11";
|
|
index-state = "2020-04-15T00:00:00Z";
|
|
plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
|
|
};
|
|
in hlint.components.exes.hlint
|
|
```
|
|
|
|
Just adding the hash might help reuse of the cached nix, but nix will
|
|
still calculate all the dependencies (which can add seconds to
|
|
`nix-build` and `nix-shell` commands when no other work is needed)
|
|
and users who do not yet have the dependencies in their store will have
|
|
to wait while they are built or downloaded.
|
|
|
|
Running nix build again gives us a hint on what we can do next:
|
|
|
|
```
|
|
$ nix-build hlint.nix
|
|
trace: Using index-state: 2020-04-15T00:00:00Z for hlint
|
|
trace: To materialize, point `materialized` to a copy of /nix/store/kk047cqsjvbj4w8psv4l05abdcnyrqdc-hlint-plan-to-nix-pkgs
|
|
/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
|
|
```
|
|
|
|
To capture the nix we can do something like:
|
|
|
|
```nix
|
|
let inherit (import ./. {}) sources nixpkgsArgs;
|
|
pkgs = import sources.nixpkgs nixpkgsArgs;
|
|
hlint = pkgs.haskell-nix.hackage-package {
|
|
compiler-nix-name = "ghc884";
|
|
name = "hlint";
|
|
version = "2.2.11";
|
|
index-state = "2020-04-15T00:00:00Z";
|
|
plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
|
|
materialized = ./hlint.materialized;
|
|
};
|
|
in hlint.components.exes.hlint
|
|
```
|
|
|
|
Now we can copy the nix files needed and build with:
|
|
|
|
```
|
|
$ cp -r /nix/store/8z6p4237rin3c6c1lmjwshmj8rdqrhw2-hlint-plan-to-nix-pkgs hlint.materialized
|
|
$ nix-build hlint.nix
|
|
/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
|
|
```
|
|
|
|
We may want to run `chmod -R +w hlint.materialized` as the files copied from the
|
|
store will be read only.
|
|
|
|
## How can we check `sha256` and `materialized` are up to date?
|
|
|
|
Let's pretend we had to go back to `hlint` version `2.2.10`.
|
|
We can tell haskell.nix to check the materialization either by:
|
|
|
|
* Removing the materialization files with `rm -rf hlint.materialized`
|
|
* Temporarily adding `checkMaterialization = true;`
|
|
|
|
If we choose to add the `checkMaterialization` flag you would have:
|
|
|
|
```nix
|
|
let inherit (import ./. {}) sources nixpkgsArgs;
|
|
pkgs = import sources.nixpkgs nixpkgsArgs;
|
|
hlint = pkgs.haskell-nix.hackage-package {
|
|
compiler-nix-name = "ghc884";
|
|
name = "hlint";
|
|
version = "2.2.10";
|
|
index-state = "2020-04-15T00:00:00Z";
|
|
plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
|
|
materialized = ./hlint.materialized;
|
|
checkMaterialization = true;
|
|
};
|
|
in hlint.components.exes.hlint
|
|
```
|
|
|
|
This will fail and report the details of what is wrong and how to fix it:
|
|
|
|
```
|
|
$ nix-build hlint.nix
|
|
|
|
...
|
|
|
|
Calculated hash for hlint-plan-to-nix-pkgs was not 02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4. New hash is :
|
|
plan-sha256 = "0zsi3wv92qax33ic4n5dfsqd1r9qam1k75za3c5jqgdxl3hy8vph";
|
|
Materialized nix used for hlint-plan-to-nix-pkgs incorrect. To fix run :
|
|
rm -rf /Users/hamish/iohk/haskell.nix/hlint.materialized
|
|
cp -r /nix/store/ywdhbx9rzzkfc60c5vzk7cins2hnvkgx-hlint-plan-to-nix-pkgs /Users/hamish/iohk/haskell.nix/hlint.materialized
|
|
chmod -R +w /Users/hamish/iohk/haskell.nix/hlint.materialized
|
|
builder for '/nix/store/a5zmgfjfxahapw0q8hd2da5bg7knqvbx-hlint-plan-to-nix-pkgs.drv' failed with exit code 1
|
|
error: build of '/nix/store/a5zmgfjfxahapw0q8hd2da5bg7knqvbx-hlint-plan-to-nix-pkgs.drv' failed
|
|
(use '--show-trace' to show detailed location information)
|
|
```
|
|
|
|
Checking the materialization requires nix to do all the work that materialization
|
|
avoids. So while it might be tempting to leave `checkMaterialization = true` all
|
|
the time, we would be better off just removing `materialized` and `plan-sha256`.
|
|
|
|
## How can we update the nix files with a script?
|
|
|
|
There are versions of the functions (`project'`, `cabalProject'`,
|
|
`stackProject'` and `hackage-project`) that also return the nix as
|
|
`plan-nix` or `stack-nix`. By calling one of these functions without
|
|
the hash and materialized nix we can find out what nix files should be.
|
|
For instance:
|
|
|
|
```nix
|
|
let inherit (import ./. {}) sources nixpkgsArgs;
|
|
pkgs = import sources.nixpkgs nixpkgsArgs;
|
|
hlint = pkgs.haskell-nix.hackage-project {
|
|
compiler-nix-name = "ghc884";
|
|
name = "hlint";
|
|
version = "2.2.10";
|
|
index-state = "2020-04-15T00:00:00Z";
|
|
};
|
|
in hlint
|
|
```
|
|
|
|
```
|
|
$ nix-build hlint.nix -A plan-nix
|
|
trace: Using index-state: 2020-04-15T00:00:00Z for hlint
|
|
trace: Get `plan-sha256` with `nix-hash --base32 --type sha256 /nix/store/ywdhbx9rzzkfc60c5vzk7cins2hnvkgx-hlint-plan-to-nix-pkgs/`
|
|
/nix/store/ywdhbx9rzzkfc60c5vzk7cins2hnvkgx-hlint-plan-to-nix-pkgs
|
|
```
|
|
|
|
We can have the script copy `$(nix-build hlint.nix -A plan-nix --no-out-link)`
|
|
and use `nix-hash` to calculate the new value for `plan-sha256`.
|
|
|
|
## Can we skip making a copy and use `materialized = /nix/store/...`?
|
|
|
|
Yes and it gives us the same speed improvement, however:
|
|
|
|
* It does not help at all in `restricted-eval` mode (Hydra).
|
|
|
|
* Users will still wind up building or downloading the dependencies
|
|
needed to build the nix files (if they do not have them).
|
|
|
|
For those reasons it might be best to make a copy instead
|
|
of using the `/nix/store/...` path directly.
|
|
|
|
If you really want to use the `/nix/store/...` path directly
|
|
you should guard against the path not existing as passing in
|
|
a non-existing path is now an error:
|
|
|
|
```nix
|
|
let inherit (import ./. {}) sources nixpkgsArgs;
|
|
pkgs = import sources.nixpkgs nixpkgsArgs;
|
|
hlintPlan = /nix/store/kk047cqsjvbj4w8psv4l05abdcnyrqdc-hlint-plan-to-nix-pkgs;
|
|
hlint = pkgs.haskell-nix.hackage-package {
|
|
compiler-nix-name = "ghc884";
|
|
name = "hlint";
|
|
version = "2.2.11";
|
|
index-state = "2020-04-15T00:00:00Z";
|
|
plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
|
|
materialized = if __pathExists hlintPlan then hlintPlan else null;
|
|
};
|
|
in hlint.components.exes.hlint
|
|
```
|
|
|
|
Running when no building is needed is still slow in restricted evaluation mode.
|
|
|
|
```
|
|
$ time nix-build --option restrict-eval true -I . --option allowed-uris "https://github.com/NixOS https://github.com/input-output-hk" hlint.nix --show-trace
|
|
trace: Using index-state: 2020-04-15T00:00:00Z for hlint
|
|
/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
|
|
|
|
real 0m4.463s
|
|
user 0m4.440s
|
|
sys 0m0.461s
|
|
$ time nix-build hlint.nix
|
|
/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
|
|
|
|
real 0m2.206s
|
|
user 0m1.665s
|
|
sys 0m0.332s
|
|
```
|