Chunk.Verify is an odd field - it originally conveys whether a source is going to run with verification, but then, at a certain point in the scanning pipeline, is mutated such that it instead indicates whether the chunk should be scanned with verification - which is not solely dependent on the source's verify flag. This is unnecessarily difficult to understand and maintain. This commit separates those two pieces of information into two flags:
- Chunk.Verify has been renamed to Chunk.SourceVerify
- It is no longer mutated; instead "should this chunk's secrets be verified?" is now captured by a new field on detectableChunk
* Added graphql API for issues and PRs
* enhancements
* more cleanup
* more enhancements
* some final touches
* some more cleanup
* tweaked threads vars
* minor changes
* scan the markdown text not plain text
* ratelimit handling
* added ratelimit handling
* lint error fix
* refactored the thread review comments chunking
* final commit - probably
* refactored the rate limit handling
* adjusted comments
* resolved comments
* remove old camel case func
* renamed featureflag
* resolved martin's comments
* updated test cases
* explicit repositories now bypass wantRepo() filtering entirely.
added ctx to newConnector
* Added test that demonstrates this bypass
* simplified test and focused on enumeration
---------
Co-authored-by: Kashif Khan <70996046+kashifkhan0771@users.noreply.github.com>
* Implement SourceUnitEnumChunker for GitHub
This change refactors the internal scan method to introduce a scanRepo
method to perform the actual scan.
* Export unit fields so the values are captured in the report
* Add comment for scanRepo
* Break out ensureRepoInfoCache into a method
* Update comments and check errors
* Ensure that the repoInfoCache contains the repo during ChunkUnit
* Add integration test for ChunkUnit
* Move s.scanOptions initialization to Init()
* Fix GitHub integration test
* Instrument GitHub source with a UnitReporter
The reporter is currently unused, but is the first step to support
scanning while enumerating.
* Update GitHub unit tests
The previous implementation of targeted file scanning pulled patches out of commit data, which didn't work for binary files (because GitHub doesn't return patches for them). This PR changes the system to always just download the requested file and scan it, which means we get binary file support.
The GitHub source currently applies its authentication configuration as the first step of enumeration. This is incompatible with both targeted scans and scan job reports, and also means that authentication logic has to be duplicated into the validation flow. This PR moves it into Init so that it's available to targeted scans and, eventually, unit-specific scans. This also allows us to remove the copy of the old logic that was in Validate.
As part of the work I've also cleaned up the integration test suite. (Several of them were apparently disabled back when they ran on every push, but now that we're not doing that, we can re-enable them.)
* issue comment scanning
* save progress
* test
* test for pr comment and issue comment
* add pagination support
* linter stuff
* make linter happy
* remove debug log
* readd logging
* github issue resolved
* var const block and handle rate limit
* remove magic number
* make gitURLParse a public function to use more generally
* fix test bug
* make comment scanning OPT-IN
* Use a config struct when scanning and engine source.
* fix tests.
* Move test_helpers to the sources pkg.
* Handle ScanGit error in tests.
* adderss comments.
* Use functional options.
* Remove temp var.
* Add better var names for the setup functions for each config.
* Remove unused var.
* fix error logs.
* fix error logs.
* single line.
* remove blank lines.
* Add unit tests and refactor some logic
* Move integration tests to a separate file behind a build flag
* Fix bugs in normalizeRepos
* Address lint errors
* Sort slices before comparing because order doesn't matter
Cody Rose
Kashif Khan
Jordan TunstillandKashif Khan
ahrav
Miccah
Dustin Decker
Cody Rose
Richard Gomez
Zubair Khan