Fix a bug where repository names ending with a hyphen (e.g., "my-repo-")
would have the trailing hyphen stripped when parsing the URL, causing
404 errors when trying to access the repository via the GitHub API.
The issue was in getRepoURLParts() which reconstructed the URL via
url.URL.String() and then re-parsed it. This process could lose
trailing special characters in some cases.
The fix uses repoURL.Host and repoURL.Path directly instead of
reconstructing via String(), which preserves the original path
including any trailing hyphens.
Fixes#4679
* Removed redundant IncludeRepos mentions.
* removed proto for IncludeRepos and remade protos
* removed another instance of includeRepos
* reverted proto removal and
* actually deprecated the field
* ran make protos
* explicit repositories now bypass wantRepo() filtering entirely.
added ctx to newConnector
* Added test that demonstrates this bypass
* simplified test and focused on enumeration
---------
Co-authored-by: Kashif Khan <70996046+kashifkhan0771@users.noreply.github.com>
* Changes to fix Enterprise UI filtering of Github Hosted Scanner Repositories to Include
* trying to pass linting isssues
* trying to solve linting errors
* Added upstream repo filtering, improvements to normalizeRepo, and unit tests for both.
* Fixing tests impacted by PR 4426
* nit
* update rate limit handler to use reporter
* update process repos to use rate limit handler with unit reporter
* update getReposByOrgOrUser to report err
* update dedupreporter to report err
* add errReporter interface to handle both types of reporters
* convert to error reporter types
* update handleRateLimit signature
* use reporters for all rate limit handlers in github
* get repo url before err check
* use iterator
* remove err log
* nit
* pluralize
* remove err log
* update tests
* make linter happy
* Implement SourceUnitEnumChunker for GitHub
This change refactors the internal scan method to introduce a scanRepo
method to perform the actual scan.
* Export unit fields so the values are captured in the report
* Add comment for scanRepo
* Break out ensureRepoInfoCache into a method
* Update comments and check errors
* Ensure that the repoInfoCache contains the repo during ChunkUnit
* Add integration test for ChunkUnit
* Move s.scanOptions initialization to Init()
* Fix GitHub integration test
* Instrument GitHub source with a UnitReporter
The reporter is currently unused, but is the first step to support
scanning while enumerating.
* Update GitHub unit tests
The GitHub source currently applies its authentication configuration as the first step of enumeration. This is incompatible with both targeted scans and scan job reports, and also means that authentication logic has to be duplicated into the validation flow. This PR moves it into Init so that it's available to targeted scans and, eventually, unit-specific scans. This also allows us to remove the copy of the old logic that was in Validate.
As part of the work I've also cleaned up the integration test suite. (Several of them were apparently disabled back when they ran on every push, but now that we're not doing that, we can re-enable them.)
Targeted scans should return their errors so that consumers can process them. By creating a type that combines an error with a targeted secret ID, we can return these errors without having to modify the Source interface.
This is a follow-up to #2379.
It fixes the following issues:
GitHub API calls missing rate-limit handling
The fix for Refactor GitHub source #2379 (comment) inadvertently resulting in duplicate API calls
This is a follow-up to #1912, which used the headers from the response to determine rate-limiting information, instead of using the values from RateLimitError.Rate. Although that logic seemed solid, I discovered that it did not work in some circumstances. This lead to the "unexpected" path more often than intended, and periodic instances where requests would be made before the ratelimit was refreshed.
* Allow using a glob for include list.
* Update command flag.
* Make comment more clear.
* update comment.
* Allow scanning repo and org at the same time.
Mustansir
Pascal THUET
Jordan Tunstill
0x1
ahrav
trufflesteeeve
JonZeollaandZachary Rice
Miccah
Nash
Cody Rose
Richard Gomez
Dustin Decker