* added detector for artifactory reference tokens
* add artifactory reference token detector to the no cloud endpoints list
* address mustansir feedback; remove the invalid host deletion
* use detectors.DetectorHttpClientWithNoLocalAddresses instead of common.SaneHttpClient() just like sibling artifactory detector
Chunk.Verify is an odd field - it originally conveys whether a source is going to run with verification, but then, at a certain point in the scanning pipeline, is mutated such that it instead indicates whether the chunk should be scanned with verification - which is not solely dependent on the source's verify flag. This is unnecessarily difficult to understand and maintain. This commit separates those two pieces of information into two flags:
- Chunk.Verify has been renamed to Chunk.SourceVerify
- It is no longer mutated; instead "should this chunk's secrets be verified?" is now captured by a new field on detectableChunk
* feat: iterative decoding pipeline with configurable depth
Decoders (base64, UTF-16, escaped unicode) now chain iteratively:
each decoder's output is fed back through all decoders until no new
transformations occur or --max-decode-depth is reached (default: 5).
This finds secrets hidden inside layered encodings, e.g. a base64
Docker auth blob containing a GCP private key, or a UTF-16 file
with base64-encoded credentials.
At depth=1 behavior is identical to the previous implementation.
Extra depths exit early when no new data is produced, so the cost
is <5% wall time on a large repo scan.
Co-authored-by: Dylan Ayrey <dxa4481@rit.edu>
* docs: iterative decoding performance data
Co-authored-by: Dylan Ayrey <dxa4481@rit.edu>
* comment: explain why PLAIN decoder is skipped at depth > 0
Co-authored-by: Dylan Ayrey <dxa4481@rit.edu>
* refactor: extract iterativeDecode, address review feedback
- Extract decode loop into standalone iterativeDecode() function,
separating decoding from channel dispatch (rosecodym, camgunz).
- Drop decodeInput struct, use []byte directly (camgunz).
- Remove redundant maxDepth clamp from scannerWorker (camgunz).
- Inline decoderType variable (camgunz).
- Replace byteSliceSeen with slices.ContainsFunc (camgunz).
Co-authored-by: Dylan Ayrey <dxa4481@rit.edu>
* fix: remove unused decodeLatency metric (lint)
Co-authored-by: Dylan Ayrey <dxa4481@rit.edu>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* set the full match as primary secret for correct line number reporting
* add test case for multiline, update documentation
* update comment
* add test case to engine
* sort regexes before selecting first for deterministic behaviour
* remove leftover todo comment
---------
Co-authored-by: Kashif Khan <70996046+kashifkhan0771@users.noreply.github.com>
There is a test seam in the engine that counts verification overlaps. This commit converts it from a bespoke mutex-wrapping-an-int type to a simple atomic integer. This is a minor thing but by gum I just love deleting code so much that I can't help myself.
The engine tests include two tests that are almost identical: TestVerificationOverlapChunkFalsePositive and TestRetainFalsePositives. This commit unifies them into a single, table-driven test. It also deletes some fake detectors they used, because we have a different fake detector that we can use instead.
The processResult function took a detectableChunk as a parameter, but that was purely out of convenience, because callers originally happened to have that type available. We later added more callers that didn't have a detectableChunk available, so they had to synthesize one. This was tolerable, but we're exploring more refactors that will introduce processResult callers that cannot synthesize a detectableChunk (because they don't have the right information available). This PR therefore modifies the signature of processResult to require only the necessary information so that we're not sandbagged by this unrelated structure down the road.
I was poking around at a refactor idea (in #4558) and realized that it had no test coverage. This PR adds that test coverage. I think I uncovered a bug in the process - see a comment in one of the tests.
* fixed sentry auth token detector
* not sure why this is failing
* Updated http client logic
Co-authored-by: Richard Gomez <32133502+rgmz@users.noreply.github.com>
* resolved comments
* improved decoding logic
* removed response type
* splitted to two versions
* splitted to two versions
* this is confusing error
---------
Co-authored-by: Richard Gomez <32133502+rgmz@users.noreply.github.com>
* adjust error handling to make more explicit
* Add DataOrErr
* update
* fix apk handler to use DataOrErr
* fix
* fix tests
* remove timeout as it wont ever be used
Since Endpoints() isn't a defined interface, we are testing an
implementation detail of EndpointSetter. If that function changes in
anyway, the test will now fail instead of skipping every detector and
passing.
* [fix] Correctly initialize detectors with cloud endpoint customization
We were only initializing if the detector was configured with a custom
endpoint, but not in the default case.
* Add test
* Fix gitlab.v2 detector
We have identified some cases in which the results "cleaning" logic (the logic that eliminates superfluous results) should not run. In order to allow this, we need to expose the cleaning logic to the engine. This PR does so by doing these things:
- Create a CustomResultsCleaner interface that can be implemented by detectors that want to use custom cleaning logic
- Implement this interface for the aws and awssessionkey detectors (and remove their previous invocation of their custom cleaning logic)
- Modify the engine to invoke this logic (conditionally)
This PR also removes the "custom" cleaning logic for the opsgenie, razorpay, and twilio detectors, because it was added erroneously.
This is an alternative implementation of #3233.
This automated test used to run with the real GitLab detectors because they were versioned. However, the test doesn't need real detectors to actually validate the functionality in question, and relying on real detectors means that we're susceptible to token expiration, which we recently discovered when it happened. The test has been updated to use fake detectors (which means it can run correctly in the community suite as well now.)
* draft reverify chunks
* remove
* remove
* reduce dupe map cap
* do not verify chunk
* cli arg and use val for dupe lut
* remove counter
* skipp empty results]
* working on test and normalizing val for comparison
* forgot to save file
* optimize normalize
* reuse map
* remove print
* use levenshtein distance to check dupes
* forgot to leave in emptying map
* use slice
* small tweak
* comment
* use bytes
* praise
* use ctx logger
* add len check
* add comments
* use 8x concurrency for reverifier workers
* revert worker count
* use more workers
* process result directly for any collisions
* continue after decoder match for reverifying
* use map
* use map
* otimization and fix the bug.
* revert worker count
* better option naming
* handle identical secrets in chunks
* update comment
* update comment
* fix test
* use DetecotrKey
* rm out of scope tests and testdata
* rename all reverification elements
* don't re-write map entry
* use correct key
* rename worker, remove log val
* test likelydupe, add eq detector check in loop
* add test
* add comment
* add test
* Set verification error
* Update tests
---------
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
The Aho-Corasick wrapper we have tracks information about whether verification should be enabled on an individual detector basis, but that functionality isn't related to the matching functionality of Aho-Corasick, and including it complicates the implementation. This PR removes it to simplify some things.
This PR removes some code that supported a potential future implementation of detector-specific verification settings, but that feature has not actually been implemented yet, so there's no loss of functionality. If we want that feature we can add it back on top of this in a more separated way.
* Add functionality to update a source's link in the metadata with the updated line number.
* update comment.
* add logic to engine.
* only update link for non empty links.
* add tests for bb.
Casey Tran
Shahzad Haider
Cody Rose
Dylan AyreyandCursor Agent
MustansirandKashif Khan
Amaan Ullah
SyedAliHamadandAmaan Ullah
Miccah
ahrav
joeleonjrandRichard Gomez
Harmon Herring
Cody Rose
Ryan Jacobchick
Shreyas Sriram