Chunk.Verify is an odd field - it originally conveys whether a source is going to run with verification, but then, at a certain point in the scanning pipeline, is mutated such that it instead indicates whether the chunk should be scanned with verification - which is not solely dependent on the source's verify flag. This is unnecessarily difficult to understand and maintain. This commit separates those two pieces of information into two flags:
- Chunk.Verify has been renamed to Chunk.SourceVerify
- It is no longer mutated; instead "should this chunk's secrets be verified?" is now captured by a new field on detectableChunk
* feat: allow input source from pipe
While performing a bit of experimentation,
I found it was easier to allow data to flow
from other segments of my machine via piping
it in, rather than flushing it to disk. This
can speed up the process when pre-processing
large amounts of data. This allow allowed me
to keep disk space low and just utilize a large
ram machine.
* (chore) update readme for new source pipe
* (fix) pipe -> stdin per PR
Cody Rose
Tim Strazzere