* use struct-based SourceMetadataFunc signature across git sources
* incorporated feedback
- pass SourceMetadataInfo by value
- remove LegacySourceMetadataFunc
Chunk.Verify is an odd field - it originally conveys whether a source is going to run with verification, but then, at a certain point in the scanning pipeline, is mutated such that it instead indicates whether the chunk should be scanned with verification - which is not solely dependent on the source's verify flag. This is unnecessarily difficult to understand and maintain. This commit separates those two pieces of information into two flags:
- Chunk.Verify has been renamed to Chunk.SourceVerify
- It is no longer mutated; instead "should this chunk's secrets be verified?" is now captured by a new field on detectableChunk
* Backoff from Scan2 which is experimental to legacy pagination API call
This commit rewrite simplifiedGitlabEnumeration to use legacy pagination API call with keyset pagination instead of Scan2 which is currently in experimental state. Note
that this doesn't promise to fix this problem it's just a test to check. It also adds a retry logic in case any 500 error occurs. I added some logs as well to keep track
of no of projects being enumerated.
* implemented builtin retry mechanism for gitlab and proper handling of next page
* fixed basic auth
* Some enhancements
Reversed the gitlab cloud logic to add membership flag, so that we use the default false for non gitlab.com instances.
This can help if the issue really was membership flag as mentioned in some gitlab issues.
Also added simple flag in list projects to get only minimal fields in response instead of big json response for each project.
Added test case as well.
* enhance the test case
* add wrapper reporter to append project details to chunk metadata
* use cache to store project details
* revert unnecessary change
* delete from cache when done with scanning in ChunkUnit, implement PR suggestions
* query project details using repo instead of having it in source unit
* revert removal of build tag
* Incorporated PR comments
Previously, the GitLab include and ignore lists were only applied during repository enumeration, which meant that they would be ignored after enumeration completed. For large environments, post-enumeration scanning can take days, and it was awkward that the include/ignore lists could effectively not be modified during that time. This PR changes things such that repositories can be configured to be skipped even post-enumeration.
Importantly, repositories cannot be "un-ignored" post-enumeration. This is unfortunate, but this PR still represents improvement on the status quo.
* Added flag to configure projects per page in gitlab enumeration
* changed comments wording
* Resolved conflicts
* use features instead of flag
* updated comment
* fix legacy json flag for gitlab private repos
* some code
* incorporated code from main
* remove cloned repositories after the scan is complete
* enhanced the code
* comment addressed
* changed the approach for persisting the repositories for legacy json printing
* addressed comments; updated the variable names
* gitlab groups init
* added list group projects api
* list group projects updated
* added duplicate repo scan check
* comments addressed
* added error when repo and group id flags are provided at the same time
* added test case for gitlab group projects
* initial code changes
* move args before clone command
* add comments
* added flag for github source to support backward compatibility
* added flag for gitlab source to support backward compatibility
* inverse the logic for enterprise
* remove print statement
* remove flag defaults
* updated comments and removed CloneRepoUsingTokenInHeader function
* false->true
We've been seeing some suspicious behavior from the GitLab API when we're scanning a very large organization that has lots of group sharing. Since reproducing the behavior in a test environment isn't really tractable, this PR adds an optional configuration flag that can be used to investigate the phenomenon in the actual environment we're seeing it in.
* Implemented gitlab inclusion globbing.
Included test.
* implemented two new flags for gitlab scan, includeRepo and excludeRepo to support globbing.
Apply globbing filter when repos is not provided.
* implemented integration test for inclusion globbing
remove test to check errors if globs are invalid.
* made changes to support glob compile errors.
modified changes to support glob compilation errors.
* removed unused context from few functions.
Some source use client libraries that can emit errors that contain sensitive information - in particular, git-facing libraries that embed tokens into repository URLs. This PR introduces a way of redacting them - starting with GitLab (where we've seen this most recently), but in theory extensible to other sources as needed.
This implementation uses a custom zap core; this might also be possible with a custom zap encoder, but I didn't test it out.
(The deleted core.go file was entirely unused.)
We have some evidence that some GitLab repos are getting incorrectly ignored, but it's not clear why this is happening, so this PR adds some more logging to the relevant code.
This PR modifies the GitLab source:
* emits a new "groups enumerated" metric
* logs more information about group enumeration
* emits the repo enumeration metric inside getAllProjectRepos, which means it will work when units are flipped on
* emits the repo enumeration metric more granularly
* Add Display method to SourceUnit and Kind member to the CommonSourceUnit
* Make SourceUnitID return the ID and a kind
These two values together uniquely represent a unit.
* Implement SourceUnitEnumChunker for GitLab
* Add GitLab engine integration test
* Use a SliceReporter instead of checking for nil reporters
* Use more generic VisitorReporter
* Merge logic from getReposFromGitlab into getAllProjectRepos
* Update integration test to have a lower bound
Unfortunately, the GitLab integration test does not appear to be
deterministic. Sometimes 36390 chunks are found, sometimes 36312, or
even lower.
ChunkReporter is more flexible and will allow code reuse for unit
chunking. ChanReporter was added as a way to maintain the original
channel functionality, so this PR should not alter existing behavior.
The previous implementation used int64 for both, which can be mixed up
easily. Using distinct types adds a layer of type safety checked by the
compiler.
This PR implements validation of Gitlab source configuration.
I was hoping to be able to unify more of the implementation of Validate and Chunks, but there was more divergence than I expected. Specifically, Chunks handles a fair number of Gitlab errors that aren't configuration errors (e.g. "Gitlab returned a repo with an unparseable URL"). Accommodating these in the Validate code path felt wrong, and I wasn't able to create a common code path that could accommodate both Validate and Chunks without looking awful.
* Implement CommonSourceUnitUnmarshaller
* Add SourceUnitUnmarshaller to all sources using
All sources, with the exception of git, will use the CommonSourceUnit as
they only contain a single type of unit to scan.
* Fix method comments to adhere to Go's style guide
Amaan Ullah
Cody Rose
Muneeb Ullah Khan
Kashif Khan
Mustansir
Shahzad Haider
Abdul Basit
Eng Zer Jun
Bob Idleandahrav
Cody Rose
Miccah
trufflesteeeveandMiccah
Dustin Decker
Richard Gomez