The time taken to run CI is getting very long. Hopefully removing nixpkgs 21.11 and no longer runs full tests for GHC 9.0.2 well make room for GHC 9.4.1.
Uses nixpkgs-unstable for broken out jobs.
This adds a way to specify the `evalSystem` or `evalPackages` explicitly when calling the `project` functions.
Currently if we want to make a `flake` that supports multiple systems we have few options:
* Require builders for all the supported systems (even just for `nix flake show`).
* Pass `--impure` so that haskell.nix can see `builtins.currentSystem` to set up `pkgs.evalPackages` to use that. Unfortunately this prevents nix from caching some of the work it does and often results in it recalculating for each supported system when it would otherwise be cached and take no time at all.
* Add an overlay to replace `evalPackages`. This works, but it is not straight forward.
* Materialize the nix files for the project.
This change allows `evalSystem = "x86_64-linux";` to be passed telling `haskell.nix` to run `cabal` and `nix-tools` on that system. The user will have to have a builder for that system, but does not need to have builders for the others (unless building outputs for them).
Fixes Windows cross compilation for GHC 9.0 and 9.2.
Updates wine to use the version that is included with the chosen Nixpkgs (it used to be pinned to an older version).
Moves the configuration of the GHC source into the GHC derivation. This should make adding Hadrian support easier.
Propagates library dependences (not just pkgconfig ones) on windows so that any DLLs in those libraries can be copied for TH evaluation and to the `/bin` directory of executable components.
Adds gcc and mfcgthreads as library dependencies on Windows so that the DLLs they include will be found.
Use `$pkgsHostTarget` (instead of `ghc-pkg`) to find all the DLLs can copy them to the `/bin` directory of executable components.
Adds support for __int128_t and __uint128_t to language-c to fix aarch64-darwin builds.
Fixed reinstalling packages that come with patched versions in ghcjs.
For ghc 9.0.2 Haskell.nix will use the `stm`and `filepath` source that came with `ghc` when the version desired matches. This is because 9.0.2 shipped with packages that differ from the same version in hackage.
`reinstallableLibGhc` is fixed for GHC 8.10 and above. Rather than including a patch file this works by:
* Setting the `subDir` to the `compiler` directory when building. That way relative paths to other parts of the `ghc` source tree (outside the `compiler` directory) still work.
* Files that are needed, but generated when `ghc` builds are added from the `generated` output of the ghc derivation.
`nonReinstallablePkgs` is now sorted and deduplicated before it is used (this reduces the chance of rebuilds when the list is refactored).
* Include dependencies of haskell.nix that were tracked in `nix/sources.json`
as flake inputs (`flake.lock` replaces `nix/sources.json`).
* Use `flake-compat` to continue to provide a compatible interface for non
flake projects.
Co-authored-by: Alexander Bantyev <balsoft@balsoft.ru>
Co-authored-by: Shea Levy <shea@shealevy.com>
The idea is to keep 8.10.4 in CI while 8.10.5 is still neither in
- a (non-nightly) Stackage LTS resolver
- upstream Nixpkgs-unstable
Hopefully it isn't long before 8.10.5 is more standard and we can remove
8.10.4 from the IOHK build.
This commit also updates documentation (which seems to have fallen out
of date), with some changes and additions to hopefully make things
easier on first-time readers.
* Add nixpkgs-2105
* [WIP] Drop nixpkgs-2003
* Update all the nixpkgs pins
* Default to nixpkgs-2009 for build.nix
* Update individual jobs to nixpkgs 2105
Co-authored-by: Hamish Mackenzie <Hamish.K.Mackenzie@gmail.com>
Anyone using `hpack` and/or `stack` will have a dependency on `internal-nix-tools` (built with a fixed version of ghc). This is fine if your `nixpkgs` is one of the ones we cache, but if not we may have to wait for an extra ghc to build. When this happens it would be nice if it is at least building a more recent version (rather than an older one).
* Delays cabal configure errors until after evaluation when possible.
* Avoids rerunning long `cabal configure` if we already know it will fail.
* Provides a way to get a component from a package or project using a `cabal` like reference.
Code using the `tool` functions will automatically use `getComponent`.
For `(hackage-package {...}).components.library` is also ok.
```
# Consider changing hackage-package use:
(pkgs.haskell-nix.hackage-package {...}).components.exes.something
(pkgs.haskell-nix.hackage-package {...}).getComponent "exe:something"
# For any cabal project:
project.hsPkgs.somepackage.components.exes.something
project.getComponent "somepackage:exe:something"
# or do it in two steps
(project.getPackage "somepackage").getComponent "exe:something"
```
The reason for the new function is that we cannot provide the attribute interface without knowing that packages are in the project first.
Here is how the `cabal configure` error output is handled:
* The `plan-nix` derivation builds even if `cabal configure` fails.
* When it fails, it copies `failed-cabal-configure.nix` to the `$out/default.nix` along with a copy of the `cabal configure` output.
* When `failed-cabal-configure.nix` is imported and used in any way it writes the `cabal configure` output with `__trace` so it will always be visible.
* Instead of a `plan` the imported nix contains a `configurationError` pointing the `cabal configure` output.
* The intermediate functions `configurationError` and bubble it up to where it is needed.
* `getPackage` returns a mostly empty proxy for a real package when there is a `configurationError`
* The `getComponent` function always returns a derivation, but the version in the proxy writes the `cabal configure` output to stdout and calls `exit 1` (so that it will never build).
See https://github.com/input-output-hk/nix-tools/pull/101
We should look up hsPkgs.${pkg-name}.components.exe.${component-name}
instead of including the package as a dependency.
Adding `hsPkgs.${pkg-name}` a tool it is not clear which executable in the package `haskell.nix` should choose.
Haskell.nix did not deal with this well and in fact it could lead to `nix-shell` crashing. For instance using `shellFor` to make a shell for building `haskell-language-server` with `cabal build` crashed as a dependency on `ghcide` the executable (by the `ghcide-bench` executable) caused infinite recursion. Allowing `ghcide-bench` to correctly depend just on `components.exes.ghcide` fixes this.
This PR also includes:
* Updated materialized files built with the new nix-tools and latest index-state
* Small update to GHCJS (pins happy) to fix issue that showed up when the materialization was updated
* A fix for the infinite recursion issue when updating materialized files (by adding ghc-boot-packages-unchecked).
* Performance fix for shellFor (checks just the names of component derivations when filtering).
* Throw better error when conf files are missing
Previously, if some directories were missing .conf files, an error like
this would be thrown:
cp: missing destination file operand after '/nix/store/pnmrz06g0sa4s3yx53hgmr9k8jrh6ww0-ouroboros-network-framework-lib-ouroboros-network-framework-0.1.0.0-haddock-config/lib/ghc-8.6.5/package.conf.d'
While normally this shouldn't occur, I've seen it happen in CI anyways,
with the problem disappearing if the store path that's missing the files
was rebuilt a couple times.
With this commit, it becomes much easier to figure out which store path
is missing the files, which would be very hard to figure out from the
previous message.
This requires explicitly identifying the things tht don't work and
removing them. I used our own `disabled` attribute, because `broken` is
weird and breaks evaluation, which isn't really what we want.
Cuts the number of jobs from over 2000 to 216. This seems to greatly
speeds up the time it takes small changes (where most stuff remains the
same) to be processed by hydra.
Uses https://github.com/NixOS/hydra/issues/715 to reference the
jobs in required by name avoiding OOM issues.
Co-authored-by: Hamish Mackenzie <Hamish.Mackenzie@iohk.io>
* Update default nixpkgs to 20.09
* Move more CI to work on 20.09 by default
* IFD level 1
* Use nixpkgs 20.03 emscripten (needed for ghcjs)
* Add missing materialized files
* Add missing materialized files
* Fix for ghc 8.8 build on darwin
* Fix for ghcjs 8.8 build on darwin
* Fix hls stack test eval for ghc810220201118
* ifdLevel 2
* Oops: accidentally turned on tests for experimental GHC
* Revert "Fix hls stack test eval for ghc810220201118"
This reverts commit b405cd7ee8bd62214bd2d0a61e0317673b5bf1b5.
* ifdLevel 3
* Disable GHC 8.6.5 windows cross on nixpkgs 20.09
* Update supported GHC doc with a table
* Add emscripten comment
Co-authored-by: Hamish Mackenzie <Hamish.Mackenzie@iohk.io>
* Drop 19.09 support
This is a year old. NixOS doesn't even support the last stable release,
let alone the one before that.
Also cuts our CI size by a third, which is always nice.
* Remove 19.09 compatibility pieces
Fixes ghcjs 8.6.5 and 8.8.4.
Uses emscripten for ghcjs 8.8 and adds scripts for building ghcjs 8.8.
Many fixes for ghcjs test failures.
Includes relocatableConfigFiles for making relocatable ghcjs bundles.
This change introduces a `ghc8102-experimental` compiler-nix-name that behaves much like the `ghc8102` version but includes @JoshMeredith's patch `./patches/ghc/ghc-8.10-ubxt.patch` that writes the `cg_binds` field with:
```haskell
liftIO $ registerInterfaceDataWith "ghc/phase/core" hsc_env $ \bh ->
putWithUserData (const $ return ()) bh (map toIfaceBind $ cg_binds cg_guts)
```
Some of the places where we used to depend on the output of `plan-to-nix` containing the correct `nix-name` for the compiler have been update to avoid using incompatible interface files from the `compiler-nix-name = "ghc8102"` version of GHC.
It looks like we can't use 8.8 for booting due to
https://gitlab.haskell.org/ghc/ghc/-/issues/18143
* Adds nixpkgs 20.09 to get mingw-w64 version 6
(also needed for ghc 8.10 windows cross compile)
* Add materialization for ghc 8.10.2 windows cross
* Fix issue with harfbuzz override so it works with nixpkgs 20.09
* Fixes eval time issue for windows index-state test (turns off native-dns cabal flag for windows)
* Include exe extension for windows in coverage test
lib:ghc is still broken for ghc 8.10 (all variants), but it always has been and we should try to fix it properly, but it will be very tricky.
When ghc 8.8.4 and ghc 8.10.2 were added we stopped building the older
versions on ci. This changes adds them back but does not run all
the tests on them.
Changes to the interface of haskell.nix (from the changelog.md file):
* Removed `sources.nixpkgs-default`, use `sources.nixpkgs` instead.
* Removed `./nixpkgs` directory, use `(import ./. {}).sources`
or `./nix/sources.nix` instead.
* Removes V1 interface for details on how to fix old code see:
https://github.com/input-output-hk/haskell.nix/issues/709
* Removed defaultCompilerNixName.
* cabalProject, cabalProject', hackage-project and hackage-package
now require a `compiler-nix-name` argument.
* `haskell-nix.tool` and `.tools` now require a `compiler-nix-name` argument.
New functions `p.tool` and `p.tools` (where p is a project) do not.
Like `shellFor { tools = ... }` they will use the compiler nix name
from the project (including stack projects where it is derived from
the resolver).
* `haskell-nix.alex` and `haskell-nix.happy` have been removed. Use
`p.tool "alex" "3.2.5"` or `shellFor { tools = { alex = "3.2.5"; } }`.
* `haskell-nix.nix-tools` -> `haskell-nix.nix-tools.ghc883` (it includes
the hpack exe now).
* `haskell-nix.cabal-install` ->
`p.tool "cabal" "3.2.0.0"` or `shellFor { tools = { cabal = "3.2.0.0"; } }`
* `haskell-nix.haskellNixRoots` -> `haskell-nix.roots ghc883` or `p.roots`
Other changes:
Adds hpack executable to the nix-tools derivations.
Adds a `cabal-hpack` test to make sure `hpack` works with
`cabalProject`.
Reduces the number of calls to `cabalProject` (particularly when
checking materialization), by giving internal tools a per-compiler
attribute.
Uses happy 1.19.12 when building newer ghc versions.
Updates cabal-install 3.2.0.0 to use the source from github that
is compatible with ghc 8.10.1.
Updates the docs for callCabalProjectToNix.
Adds a license mapping to fix a common warning.
* ghc 8.10
* Adds/Updates/Removes materialization files
* Disable one-shot-kqueue-on-macos patch on ghc-8.10
* Test ghc883 and ghc8101 but only with nix 20.03
* Various smaller fixes.
This should fix the infinite recursion issue with #654
Co-authored-by: Lennart Spitzner <hexagoxel@hexagoxel.de>
Now that the defaultCompilerNixName is used to choose which GHC to use
for tools we need to update haskellNixRoots to make sure the common
tools wind up in the cache.
To do this we add the compilerNixName as a dimension in ci.nix.
This has the added bonus of ensuring the tests a run with the
supported versions of ghc (not just the default one).
haskellNixRoots is also updated to include the tools built with the
defaultCompilerNixName (in particular cabal-install was not pinned
and this was the likely cause of significant builds due to cache
misses).
Although the default ghc used by haskell.nix is ghc 8.6.5 many of
the tools used in haskell.nix are still built with the boot compiler
ghc 8.4.4. These include
* haskell-nix.cabal-install
* haskell-nix.alex
* haskell-nix.happy
This change updates those to ghc 8.6.5 and includes materializations
for the new versions.
When cabal-install is built it is careful to disable materialization
checks on the version of itself used during the build to avoid
infinite recursion.
There was a version of nix-tools built with the boot ghc which was
only used when `checkMaterialization = true`. It was used for
the boot versions of alex, happy and hscolour. These have been update
to use the default (ghc 8.6.5) version of nix-tools and
checkMaterialization is forced off when they are being used to build
ghc. This means the materialization will only be checked for these
when they are built independently (they are included in the test set
via haskellNixRoots).
Three new arguments are added to `default.nix`:
* `defaultCompilerNixName` if not specified "ghc865" is used
* `checkMaterialization` makes it easier to switch on materialization checks
* `system` defaults to `builtins.currentSystem`
This change also moves the work needed for hydra eval to the eval
system using a new `evalPackages` feature. This includes:
* Fetching from git with `fetchgit`
* Building scripts with `runCommand` and `writeTextFile`
* `git ls-files` in `cleanGit`
* running `cabal v2-configure`
* copying materialized files (we are not sure why this is necessary but
if we do not `cp -r` the files nix will not load them on hydra)
Reduce size of `make-config-files.nix` strings by around 80%.
These are unlikely to be the cause of hydra eval time memory
issues in the GB range, but were still quite large (around 10MB for the
`cabal-simple` test case).
There was issue causing excessive builds of the `git` package when
cross compiling. Gory details are a comment in `lib/defaults.nix`
but in short if you use `git` you need an extra `.buildPackages`
one is not enough because it depends on `gdb` and that will
be different in `buildPackages` compared to
`buildPackages.buildPackages`.
Adds missing materialization files for ghc 8.4.4 (only needed
when `checkMaterialization` is on because of other
materialiazations, but good to have).
Changing from absolute links to relative links increases the length of
the parameters passed during linking by half. For things that have many
dependencies, this results in linking failing with an "argument list too
long" error.
Fixes#562
This change also updates the CI configuration files to help make
it easier for CI to cope with this sort of change.
Co-authored-by: Richard Wallace <richard.wallace@simspace.com>
This overlay helps accessing common executable components.
Typically we want to make these available in a nix-shell
created with shellFor. In most cases the package name
will be the same as the executable, but we have a
`toolPackageName` mapping to help when it is not.
```
# To get a single tool:
haskell-nix.tool "cabal" "3.2.0.0"
# This does the same thing as:
(haskell-nix.hackage-package {
name = "cabal-install"
version = "3.2.0.0"
}).components.exes.cabal
# To get an attr set containing multiple tools:
haskell-nix.tools { cabal = "3.2.0.0"; hlint = "2.2.11"; }
# To add tools to a shell:
shellFor { tools = { cabal = "3.2.0.0"; hlint = "2.2.11"; }; }
```
When used in shellFor the tools will be compiled with the same version
of ghc used in the shell (the build ghc in the case of cross
compilation).
Instead of a version string we can use an attr set containing
arguments that will be passed to `cabalProject`.
For instance to specify the ghc used to compile. Use:
```
haskell-nix.tool "cabal" {
version = "3.2.0.0";
ghc = haskell-nix.compiler.ghc883;
}
```
* Update nixpkgs pins to include 20.03, wobble
- Add 20.03
- Remove 19.03 (and old 18.09 file)
- Update 19.09 to latest
- Update github pin to 20.03
- Remove workarounds for 19.03
* Mark openssl as allowedInsecure
* Force python3 for mkdocs
* Include `tests` in path for cross compiled tests
* Add ghc-boot-packages-nix to haskellNixRoots
* Fix naming warnings in tests sources
* Uses a single cleanGit call for all the tests
* Add memoize code for ghc-boot and ghc-extra pkgs
* Memoize ghc-boot and ghc-extra pkgs nix
* Fix release.nix being more strict than ci.nix by updating filterAttrsOnlyRecursive in ci-lib.nix
* Nicer errors when materialized path does not exist
* Updated materialization docs
* Add internalHackageIndexState to set the index-state used within haskell.nix
Hamish Mackenzie
Shea Levy
Hamish Mackenzie
Tom Prince
Michael Peyton Jones
Teo Camarasu
Sukant Hajra
Moisés Ackerman
Silvan Mosberger
John A. Lotoski