* use struct-based SourceMetadataFunc signature across git sources
* incorporated feedback
- pass SourceMetadataInfo by value
- remove LegacySourceMetadataFunc
Chunk.Verify is an odd field - it originally conveys whether a source is going to run with verification, but then, at a certain point in the scanning pipeline, is mutated such that it instead indicates whether the chunk should be scanned with verification - which is not solely dependent on the source's verify flag. This is unnecessarily difficult to understand and maintain. This commit separates those two pieces of information into two flags:
- Chunk.Verify has been renamed to Chunk.SourceVerify
- It is no longer mutated; instead "should this chunk's secrets be verified?" is now captured by a new field on detectableChunk
Fix a bug where repository names ending with a hyphen (e.g., "my-repo-")
would have the trailing hyphen stripped when parsing the URL, causing
404 errors when trying to access the repository via the GitHub API.
The issue was in getRepoURLParts() which reconstructed the URL via
url.URL.String() and then re-parsed it. This process could lose
trailing special characters in some cases.
The fix uses repoURL.Host and repoURL.Path directly instead of
reconstructing via String(), which preserves the original path
including any trailing hyphens.
Fixes#4679
* Added graphql API for issues and PRs
* enhancements
* more cleanup
* more enhancements
* some final touches
* some more cleanup
* tweaked threads vars
* minor changes
* scan the markdown text not plain text
* ratelimit handling
* added ratelimit handling
* lint error fix
* refactored the thread review comments chunking
* final commit - probably
* refactored the rate limit handling
* adjusted comments
* resolved comments
* remove old camel case func
* renamed featureflag
* resolved martin's comments
* updated test cases
* Removed redundant IncludeRepos mentions.
* removed proto for IncludeRepos and remade protos
* removed another instance of includeRepos
* reverted proto removal and
* actually deprecated the field
* ran make protos
* explicit repositories now bypass wantRepo() filtering entirely.
added ctx to newConnector
* Added test that demonstrates this bypass
* simplified test and focused on enumeration
---------
Co-authored-by: Kashif Khan <70996046+kashifkhan0771@users.noreply.github.com>
* Changes to fix Enterprise UI filtering of Github Hosted Scanner Repositories to Include
* trying to pass linting isssues
* trying to solve linting errors
* Added upstream repo filtering, improvements to normalizeRepo, and unit tests for both.
* Fixing tests impacted by PR 4426
* fix legacy json flag for gitlab private repos
* some code
* incorporated code from main
* remove cloned repositories after the scan is complete
* enhanced the code
* comment addressed
* changed the approach for persisting the repositories for legacy json printing
* addressed comments; updated the variable names
* initial code changes
* move args before clone command
* add comments
* added flag for github source to support backward compatibility
* added flag for gitlab source to support backward compatibility
* inverse the logic for enterprise
* remove print statement
* remove flag defaults
* updated comments and removed CloneRepoUsingTokenInHeader function
* false->true
GitHub Realtime needs to be able to authenticate. This PR adds relevant fields to the relevant protobuf message, and exports/tweaks some stuff from the github source package to permit its (eventual) use by the GitHub Realtime implementation:
The Connector interface and its constructors are now exported
Connector.Clone now accepts additional clone arguments
The github-specific connection construction has been moved into github.go
The injected rate limit handler no longer accepts error reporters that it never uses
I also added an API endpoint field to the GitHub Realtime proto message, because we need that too and I was here.
* nit
* update rate limit handler to use reporter
* update process repos to use rate limit handler with unit reporter
* update getReposByOrgOrUser to report err
* update dedupreporter to report err
* add errReporter interface to handle both types of reporters
* convert to error reporter types
* update handleRateLimit signature
* use reporters for all rate limit handlers in github
* get repo url before err check
* use iterator
* remove err log
* nit
* pluralize
* remove err log
* update tests
* make linter happy
* Implement SourceUnitEnumChunker for GitHub
This change refactors the internal scan method to introduce a scanRepo
method to perform the actual scan.
* Export unit fields so the values are captured in the report
* Add comment for scanRepo
* Break out ensureRepoInfoCache into a method
* Update comments and check errors
* Ensure that the repoInfoCache contains the repo during ChunkUnit
* Add integration test for ChunkUnit
* Move s.scanOptions initialization to Init()
* Fix GitHub integration test
* Instrument GitHub source with a UnitReporter
The reporter is currently unused, but is the first step to support
scanning while enumerating.
* Update GitHub unit tests
The previous implementation of targeted file scanning pulled patches out of commit data, which didn't work for binary files (because GitHub doesn't return patches for them). This PR changes the system to always just download the requested file and scan it, which means we get binary file support.
The GitHub source generates chunks for targeted scans differently than it does for "normal" scans. One difference was the presence of leading + and - characters, which can interfere with detection in some cases.
The GitHub source currently applies its authentication configuration as the first step of enumeration. This is incompatible with both targeted scans and scan job reports, and also means that authentication logic has to be duplicated into the validation flow. This PR moves it into Init so that it's available to targeted scans and, eventually, unit-specific scans. This also allows us to remove the copy of the old logic that was in Validate.
As part of the work I've also cleaned up the integration test suite. (Several of them were apparently disabled back when they ran on every push, but now that we're not doing that, we can re-enable them.)
* alpha feature for scanning hidden commits on github
* improvements re: git operations
* lint updates
* updating with exec block due to no gh token
* reworked logic into new source
* fixed collisions threshold flag input
* fixed IOutil issues
* removed additions from GH config
---------
Co-authored-by: Joe Leon <joe.leon@trufflesec.com>
Targeted scans should return their errors so that consumers can process them. By creating a type that combines an error with a targeted secret ID, we can return these errors without having to modify the Source interface.
Amaan Ullah
Cody Rose
Mustansir
Pascal THUET
Kashif Khan
Jordan Tunstill
Shahzad Haider
Richard GomezandKashif Khan
Eng Zer Jun
0x1
ahrav
trufflesteeeve
Bob Idleandahrav
JonZeollaandZachary Rice
Miccah
Nash
Cody Rose
joeleonjrandJoe Leon