mirror of
https://github.com/simplex-chat/haskell.nix.git
synced 2026-06-03 09:17:32 +00:00
Samuel Evans-Powell
48b8674f5f
- Added the ability to generate coverage reports for packages and projects. - Outputs mix and tix information, as well as a HTML report. - Added the "doCoverage" module option that allows users to choose packages to enable coverage for. - Added a "doCoverage" flag to the component builder that outputs HPC information when coverage is enabled. - Added the "overrideModules" library function to make it more ergonomic fo users to enable coverage on existing projects. - Modified the "check" builder to also output ".tix" files (if they exist). This information is required to generate the coverage report. - Added a test for coverage.
181 lines
7.8 KiB
Markdown
181 lines
7.8 KiB
Markdown
# Developer Coverage Overview
|
||
|
||
## Building
|
||
|
||
The implementation of coverage starts with the "doCoverage" flag on
|
||
the builder in `comp-builder.nix`. The doCoverage flag enables and
|
||
disables the Cabal coverage flag and copies any generated coverage
|
||
data to "$out/share/hpc".
|
||
|
||
### Mix and tix files
|
||
|
||
The coverage information for any derivation consists of "mix" and
|
||
"tix" files.
|
||
|
||
Mix files record static information about a source file and are
|
||
generated at build time. They primarily contain a path to the source
|
||
file and information about expressions and regions of the source file,
|
||
which are later referenced by tix files.
|
||
|
||
Tix files contain dynamic information about a test run, recording when
|
||
a portion of a source file is touched by a test. These are generated
|
||
when the test is run.
|
||
|
||
### Multiple local packages
|
||
|
||
In the context of multiple local packages, there are a few types of
|
||
coverage we might be interested in:
|
||
- How well does the tests for this package cover the package library?
|
||
- How well does the tests for this package cover the libraries of
|
||
other packages in this project?
|
||
- Both of the above.
|
||
|
||
To facilitate expressing any of these classifications of coverage, the
|
||
`lib/cover.nix` function provides the `mixLibraries` argument. If
|
||
you're just interested in how the tests cover the package library, you
|
||
provide that library as an argument to `mixLibraries`. If you're
|
||
interested in how the tests also cover other local packages in the
|
||
project, you can also provide those libraries as arguments to
|
||
mixLibraries.
|
||
|
||
The `projectCoverageReport` and `coverageReport` attributes that are
|
||
provided by default on projects and packages respectively provide
|
||
coverage information for *all* local packages in the project. This is
|
||
to mimic the behaviour of Stack, which seems to be the expectation of
|
||
most people. Of course, you can use the `projectCoverageReport` and
|
||
`coverageReport` functions to construct your own custom coverage
|
||
reports (as detailed in the [coverage tutorial](../tutorials/coverage.md#custom)).
|
||
|
||
## Coverage reports
|
||
|
||
### Package reports
|
||
|
||
The coverage information generated will look something like this:
|
||
|
||
```bash
|
||
/nix/store/...-my-project-0.1.0.0-coverage-report/
|
||
└── share
|
||
└── hpc
|
||
└── vanilla
|
||
├── html
|
||
│ └── my-library-0.1.0.0
|
||
│ ├── my-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
|
||
│ │ ├── My.Lib.Config.hs.html
|
||
│ │ ├── My.Lib.Types.hs.html
|
||
│ │ └── My.Lib.Util.hs.html
|
||
│ ├── hpc_index_alt.html
|
||
│ ├── hpc_index_exp.html
|
||
│ ├── hpc_index_fun.html
|
||
│ └── hpc_index.html
|
||
├── mix
|
||
│ └── my-library-0.1.0.0
|
||
│ └── my-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
|
||
│ ├── My.Lib.Config.mix
|
||
│ ├── My.Lib.Types.mix
|
||
│ └── My.Lib.Util.mix
|
||
└── tix
|
||
└── my-library-0.1.0.0
|
||
├── my-library-0.1.0.0.tix
|
||
├── my-test-1
|
||
│ └── my-test-1.tix
|
||
└── unit-test
|
||
└── unit-test.tix
|
||
```
|
||
|
||
- The mix files are copied verbatim from the library built with
|
||
coverage.
|
||
- The tix files for each test are copied from the check run verbatim
|
||
and are output to ".../tix/<libraryname>/<testname>/<testname>.tix".
|
||
- The tix files for each library are generated by summing the tix
|
||
files for each test, but excluding any test modules. This tix file
|
||
is output to ".../tix/<libraryname>/<libraryname>.tix".
|
||
- Test modules are determined by inspecting the plan for the project
|
||
(i.e. for the project "my-project" and test-suite "my-test-1", the
|
||
test modules are read from:
|
||
`my-project.checks.my-test-1.config.modules`)
|
||
- The hpc HTML reports for each library are generated from their
|
||
respective tix files (i.e. the
|
||
`share/hpc/vanilla/html/my-library-0.1.0.0` report is generated from
|
||
the
|
||
`share/hpc/vanilla/tix/my-library-0.1.0.0/my-library-0.1.0.0.tix`
|
||
file)
|
||
|
||
### Project-wide reports
|
||
|
||
The coverage information for an entire project will look something
|
||
like this:
|
||
|
||
```bash
|
||
/nix/store/...-coverage-report
|
||
└── share
|
||
└── hpc
|
||
└── vanilla
|
||
├── html
|
||
│ ├── index.html
|
||
│ ├── all
|
||
│ │ ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
|
||
│ │ │ ├── My.Lib.Config.hs.html
|
||
│ │ │ ├── My.Lib.Types.hs.html
|
||
│ │ │ └── My.Lib.Util.hs.html
|
||
│ │ ├── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
|
||
│ │ │ ├── Other.Lib.A.hs.html
|
||
│ │ │ └── Other.Lib.B.hs.html
|
||
│ │ ├── hpc_index_alt.html
|
||
│ │ ├── hpc_index_exp.html
|
||
│ │ ├── hpc_index_fun.html
|
||
│ │ └── hpc_index.html
|
||
│ ├── my-library-0.1.0.0
|
||
│ │ ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
|
||
│ │ │ ├── My.Lib.Config.hs.html
|
||
│ │ │ ├── My.Lib.Types.hs.html
|
||
│ │ │ └── My.Lib.Util.hs.html
|
||
│ │ ├── hpc_index_alt.html
|
||
│ │ ├── hpc_index_exp.html
|
||
│ │ ├── hpc_index_fun.html
|
||
│ │ └── hpc_index.html
|
||
│ └── other-libray-0.1.0.0
|
||
│ ├── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
|
||
│ │ ├── Other.Lib.A.hs.html
|
||
│ │ └── Other.Lib.B.hs.html
|
||
│ ├── hpc_index_alt.html
|
||
│ ├── hpc_index_exp.html
|
||
│ ├── hpc_index_fun.html
|
||
│ └── hpc_index.html
|
||
├── mix
|
||
│ ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
|
||
│ │ ├── My.Lib.Config.mix
|
||
│ │ ├── My.Lib.Types.mix
|
||
│ │ └── My.Lib.Util.mix
|
||
│ └── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
|
||
│ ├── Other.Lib.A.mix
|
||
│ └── Other.Lib.B.mix
|
||
└── tix
|
||
├── all
|
||
│ └── all.tix
|
||
├── my-library-0.1.0.0
|
||
│ ├── my-library-0.1.0.0.tix
|
||
│ ├── my-test-1
|
||
│ │ └── my-test-1.tix
|
||
│ └── unit-test
|
||
│ └── unit-test.tix
|
||
└── another-library-0.1.0.0
|
||
├── another-library-0.1.0.0.tix
|
||
├── my-test-2
|
||
│ └── my-test-2.tix
|
||
└── unit-test
|
||
└── unit-test.tix
|
||
```
|
||
|
||
All of the coverage information is copied verbatim from the coverage
|
||
reports for each of the constituent packages. A few additions are
|
||
made:
|
||
- `tix/all/all.tix` is generated from the union of all the library
|
||
tix files.
|
||
- We use this file when generating coverage reports for
|
||
"coveralls.io".
|
||
- An index page (`html/index.html`) is generated which links to the
|
||
HTML coverage reports of the constituent packages.
|
||
- A synthetic HTML report is generated from the `tix/all/all.tix`
|
||
file. This shows the union of all the coverage information
|
||
generated by each constituent coverage report.
|