haskell.nix/docs/dev/coverage.md

# Developer Coverage Overview

## Building

The implementation of coverage starts with the "doCoverage" flag on
the builder in `comp-builder.nix`. The doCoverage flag enables and
disables the Cabal coverage flag and copies any generated coverage
data to "$out/share/hpc".

### Mix and tix files

The coverage information for any derivation consists of "mix" and
"tix" files.

Mix files record static information about a source file and are
generated at build time. They primarily contain a path to the source
file and information about expressions and regions of the source file,
which are later referenced by tix files.

Tix files contain dynamic information about a test run, recording when
a portion of a source file is touched by a test. These are generated
when the test is run.

### Multiple local packages

In the context of multiple local packages, there are a few types of
coverage we might be interested in:
  - How well does the tests for this package cover the package library?
  - How well does the tests for this package cover the libraries of
    other packages in this project?
  - Both of the above.

To facilitate expressing any of these classifications of coverage, the
`lib/cover.nix` function provides the `mixLibraries` argument. If
you're just interested in how the tests cover the package library, you
provide that library as an argument to `mixLibraries`. If you're
interested in how the tests also cover other local packages in the
project, you can also provide those libraries as arguments to
mixLibraries.

The `projectCoverageReport` and `coverageReport` attributes that are
provided by default on projects and packages respectively provide
coverage information for *all* local packages in the project. This is
to mimic the behaviour of Stack, which seems to be the expectation of
most people. Of course, you can use the `projectCoverageReport` and
`coverageReport` functions to construct your own custom coverage
reports (as detailed in the [coverage tutorial](../tutorials/coverage.md#custom)).

## Coverage reports

### Package reports

The coverage information generated will look something like this:

```shell
/nix/store/...-my-project-0.1.0.0-coverage-report/
└── share
    └── hpc
        └── vanilla
            ├── html
            │   └── my-library-0.1.0.0
            │       ├── my-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
            │       │   ├── My.Lib.Config.hs.html
            │       │   ├── My.Lib.Types.hs.html
            │       │   └── My.Lib.Util.hs.html
            │       ├── hpc_index_alt.html
            │       ├── hpc_index_exp.html
            │       ├── hpc_index_fun.html
            │       └── hpc_index.html
            ├── mix
            │   └── my-library-0.1.0.0
            │       └── my-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
            │           ├── My.Lib.Config.mix
            │           ├── My.Lib.Types.mix
            │           └── My.Lib.Util.mix
            └── tix
                └── my-library-0.1.0.0
                    ├── my-library-0.1.0.0.tix
                    ├── my-test-1
                    │   └── my-test-1.tix
                    └── unit-test
                        └── unit-test.tix
```

- The mix files are copied verbatim from the library built with
  coverage.
- The tix files for each test are copied from the check run verbatim
  and are output to ".../tix/<libraryname>/<testname>/<testname>.tix".
- The tix files for each library are generated by summing the tix
  files for each test, but excluding any test modules. This tix file
  is output to ".../tix/<libraryname>/<libraryname>.tix".
  - Test modules are determined by inspecting the plan for the project
    (i.e. for the project "my-project" and test-suite "my-test-1", the
    test modules are read from:
    `my-project.checks.my-test-1.config.modules`)
- The hpc HTML reports for each library are generated from their
  respective tix files (i.e. the
  `share/hpc/vanilla/html/my-library-0.1.0.0` report is generated from
  the
  `share/hpc/vanilla/tix/my-library-0.1.0.0/my-library-0.1.0.0.tix`
  file)

### Project-wide reports

The coverage information for an entire project will look something
like this:

```shell
/nix/store/...-coverage-report
└── share
    └── hpc
        └── vanilla
            ├── html
            │   ├── index.html
            │   ├── all
            │   │   ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
            │   │   │   ├── My.Lib.Config.hs.html
            │   │   │   ├── My.Lib.Types.hs.html
            │   │   │   └── My.Lib.Util.hs.html
            │   │   ├── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
            │   │   │   ├── Other.Lib.A.hs.html
            │   │   │   └── Other.Lib.B.hs.html
            │   │   ├── hpc_index_alt.html
            │   │   ├── hpc_index_exp.html
            │   │   ├── hpc_index_fun.html
            │   │   └── hpc_index.html
            │   ├── my-library-0.1.0.0
            │   │   ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
            │   │   │   ├── My.Lib.Config.hs.html
            │   │   │   ├── My.Lib.Types.hs.html
            │   │   │   └── My.Lib.Util.hs.html
            │   │   ├── hpc_index_alt.html
            │   │   ├── hpc_index_exp.html
            │   │   ├── hpc_index_fun.html
            │   │   └── hpc_index.html
            │   └── other-libray-0.1.0.0
            │       ├── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
            │       │   ├── Other.Lib.A.hs.html
            │       │   └── Other.Lib.B.hs.html
            │       ├── hpc_index_alt.html
            │       ├── hpc_index_exp.html
            │       ├── hpc_index_fun.html
            │       └── hpc_index.html
            ├── mix
            │   ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
            │   │   ├── My.Lib.Config.mix
            │   │   ├── My.Lib.Types.mix
            │   │   └── My.Lib.Util.mix
            │   └── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
            │       ├── Other.Lib.A.mix
            │       └── Other.Lib.B.mix
            └── tix
                ├── all
                │   └── all.tix
                ├── my-library-0.1.0.0
                │   ├── my-library-0.1.0.0.tix
                │   ├── my-test-1
                │   │   └── my-test-1.tix
                │   └── unit-test
                │       └── unit-test.tix
                └── another-library-0.1.0.0
                    ├── another-library-0.1.0.0.tix
                    ├── my-test-2
                    │   └── my-test-2.tix
                    └── unit-test
                        └── unit-test.tix
```

All of the coverage information is copied verbatim from the coverage
reports for each of the constituent packages. A few additions are
made:
  - `tix/all/all.tix` is generated from the union of all the library
    tix files.
    - We use this file when generating coverage reports for
      "coveralls.io".
  - An index page (`html/index.html`) is generated which links to the
    HTML coverage reports of the constituent packages.
  - A synthetic HTML report is generated from the `tix/all/all.tix`
    file. This shows the union of all the coverage information
    generated by each constituent coverage report.