* use struct-based SourceMetadataFunc signature across git sources
* incorporated feedback
- pass SourceMetadataInfo by value
- remove LegacySourceMetadataFunc
* Confine symlink state handling to scanSymlink in Filesystem source
* Fix s.canFollowSymlinks snafu
* Update symlink tests to use starting depth 0
* Missed one
* Remove symlink checking from scanFile; this is now always handled in scanSymlink
* Confine errgroup.Groups to scanDir in the Filesystem source (#4808)
* Move path parameter after rootPath parameter in the Filesystem source
* Move the depth parameter too
* Only create an errgroup.Group inside scanDir (where it's used) in the Filesystem source
#4742 (4563dde124) introduced a change to the filesystem source resumption tracking that caused it to start growing linearly with subdirectory count - which causes the payload to get intractably big on large data sets. This commit is an attempt to resolve the issue.
Note that the resumption code still has a bug that can cause data to get inadvertently skipped due to mishandling of the internal parallelization of the scan. This bug has been present for a long time and is present in other sources, so fixing it is out of scope here.
This commit _also_ introduces a new bug related to the fact that lexicographic sorting is not completely appropriate for the resumption check. This needs to be cleaned up as a fast follow, but it's still less serious than the current bug that prevents all scans of large data sets.
* [secret-storage] Thread original chunk data through engine pipeline
Adds OriginalData/ChunkData fields to preserve pre-decode source data
through the scan pipeline:
1. Chunk.OriginalData: captures chunk.Data before iterativeDecode
2. engine.go: sets chunk.OriginalData = chunk.Data before decode
3. ResultWithMetadata.ChunkData: populated by CopyMetadata from
OriginalData (falls back to Data when nil)
This enables downstream consumers (e.g. the dispatcher in thog) to
access the original source data for secret storage encryption.
* Update TestChunkSize for OriginalData field addition
Chunk struct grew from 80 to 104 bytes with the OriginalData []byte
slice header (24 bytes). Field placement is already optimal (adjacent
to Data []byte).
* fix: preserve OriginalData field in EscapedUnicode decoder
The EscapedUnicode decoder constructed a new sources.Chunk manually
copying fields but omitted OriginalData. This caused CopyMetadata to
fall back to the decoded Data instead of the original pre-decode content,
defeating the purpose of preserving original chunk data for secret
storage encryption.
* Address PR review feedback: use testify/assert, add nil-guard comment, remove stale alignment comment
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Chunk.Verify is an odd field - it originally conveys whether a source is going to run with verification, but then, at a certain point in the scanning pipeline, is mutated such that it instead indicates whether the chunk should be scanned with verification - which is not solely dependent on the source's verify flag. This is unnecessarily difficult to understand and maintain. This commit separates those two pieces of information into two flags:
- Chunk.Verify has been renamed to Chunk.SourceVerify
- It is no longer mutated; instead "should this chunk's secrets be verified?" is now captured by a new field on detectableChunk
* enabled symlinks with maximum depth support
* resolved concurrency bugs and added maxDepthOption to cli
* Removed visited path map and tracked symlink depth by maintaining a counter variable
* separated symlink scanning from scanDir
* resolved bugbot comments
* introduced hash as a separator to avoid collisions
This adds a new input source to TruffleHog, accessible via `trufflehog json-enumerator`.
This input source requires a list of filenames, each of which is an NDJSON-formatted sequence of objects that take one of two forms:
Form 1: `{"data": "utf-8 string", "metadata": <non-null JSON value>}`
Form 2: `{"data_b64": "base64-encoded bytestring", "metadata": <non-null JSON value>}`
The `data` / `data_b64` field specifies the content to be scanned. The `metadata` field is arbitrary, and is simply propagated downstream with scan results from the corresponding content.
Note that although `trufflehog json-enumerator` requires a list of filenames to be given, the NDJSON data that you wish to scan may not need to be first written to disk. On Linux and macOS, at least, you can use shell process substitution to set up a named pipe from a producer process, like `trufflehog json-enumerator <(some-program-that-emits-ndjson)`.
Fix a bug where repository names ending with a hyphen (e.g., "my-repo-")
would have the trailing hyphen stripped when parsing the URL, causing
404 errors when trying to access the repository via the GitHub API.
The issue was in getRepoURLParts() which reconstructed the URL via
url.URL.String() and then re-parsed it. This process could lose
trailing special characters in some cases.
The fix uses repoURL.Host and repoURL.Path directly instead of
reconstructing via String(), which preserves the original path
including any trailing hyphens.
Fixes#4679
* Fix Windows file:// URI normalization and index path handling
* add condition to trim leading slash only incase of windows paths
* access path from URL object and identify platform using goos
* simplified path logic
* added test cases for windows powershell & bash
---------
Co-authored-by: Kashif Khan <70996046+kashifkhan0771@users.noreply.github.com>
Fixes#1876
Problem
Files are processed in ~13KB chunks by default (10KB of actual data plus a 3KB peek into the next chunk). To infer the line the engine would take the line from chunk's metadata SourceMetadata.Line and would consider it absolute, that is it's relative to the entire file. The problem is it was never set anywhere. Since every chunk had line equals to 1 in metadata, all secret scan results appeared at line 1 + FragmentLineOffset() which resulted in line being relative to the chunk not the file itself.
Solution
Track cumulative line numbers in handleNonArchiveContent() by counting newlines as chunks are processed. Each DataOrErr now carries the correct starting line, which flows through to SourceMetadata.Filesystem.Line, giving FragmentFirstLineAndLink() the correct fragStart for the final calculation.
Affected Sources
Directly fixed:
Filesystem - now reports accurate line numbers. I have tested it using the test file provided in the issue wget https://gist.githubusercontent.com/det/1526b4c16d0e07ac023d75c912a68658/raw/c3061c14a811205a65cbdcf0065bd3c11d88bfcb/test.txt
Not affected (no Line field in proto):
S3, GCS, Jenkins, stdin - use handlers but their metadata protos lack a Line field. I think it makes sense for S3 and GCS to report line numbers so it could be a good future change.
Partially affected (own line tracking):
Git, GitHub, GitLab - regular text diffs use git-based scanning with built-in line tracking (unaffected). However, binary archives (tar.gz, zip) go through HandleBinary() → handlers.HandleFile() and benefit from this fix.
others like BitBucket use their own implementations so not affected
* Backoff from Scan2 which is experimental to legacy pagination API call
This commit rewrite simplifiedGitlabEnumeration to use legacy pagination API call with keyset pagination instead of Scan2 which is currently in experimental state. Note
that this doesn't promise to fix this problem it's just a test to check. It also adds a retry logic in case any 500 error occurs. I added some logs as well to keep track
of no of projects being enumerated.
* implemented builtin retry mechanism for gitlab and proper handling of next page
* fixed basic auth
* Some enhancements
Reversed the gitlab cloud logic to add membership flag, so that we use the default false for non gitlab.com instances.
This can help if the issue really was membership flag as mentioned in some gitlab issues.
Also added simple flag in list projects to get only minimal fields in response instead of big json response for each project.
Added test case as well.
* enhance the test case
* add wrapper reporter to append project details to chunk metadata
* use cache to store project details
* revert unnecessary change
* delete from cache when done with scanning in ChunkUnit, implement PR suggestions
* query project details using repo instead of having it in source unit
* revert removal of build tag
* Incorporated PR comments
* Added pagination and retry logic on rate-limit errors in docker registry list images calls
* added rate limit error failure test
* added exponential backoff
* updated defered response body discards, added more comments, removed some unnecessary logic in Quay.ListImages()
* readded quay list images query that the requests private images
* added golang.org/x/time/rate to limit registry API calling rate
* switched to using retryable http client in docker registry calls
* modified rate limiter. created constant for max page size in api request
---------
Co-authored-by: Amaan Ullah <aman.ullah.jalal@trufflesec.com>
Co-authored-by: Kashif Khan <70996046+kashifkhan0771@users.noreply.github.com>
We have identified some cases in which it is preferable to time a clone out instead of waiting forever. These situations are unusual, so the CLI option to enable this (which I added for testing) is hidden so that we minimize the risk of baking this option into the interface.
Previously, the GitLab include and ignore lists were only applied during repository enumeration, which meant that they would be ignored after enumeration completed. For large environments, post-enumeration scanning can take days, and it was awkward that the include/ignore lists could effectively not be modified during that time. This PR changes things such that repositories can be configured to be skipped even post-enumeration.
Importantly, repositories cannot be "un-ignored" post-enumeration. This is unfortunate, but this PR still represents improvement on the status quo.
This commit adds timing information to the logs when clones complete successfully. There are several other more robust but also more involved interventions we could do, and I think we should do them, but this is a strict improvement that's easy to get out as a first step.
I also added logging at the precise beginning of the clone command so we know when we're doing that.
* implemented Source unit for S3.
Implemented. integration test
* use bucket as source unit
* remove code duplication, reuse from Chunks
* remove unnecessary change
* remove unused functions
* revisit tests
* revert unnecessary change
* change SourceUnitKind to s3_bucket
* handle nil objectCount inside scanBucket
* handle nil objectCount outside loop
* add bucket to resume log
* add bucket and role to error log, remove enumerating log
* implement sub unit resumption
* add comment to checkpointer for unit scans
* implement SourceUnitUnmarshaller on source with the new S3SourceUnit, add test to test resumption on multiple buckets with concurrent ChunkUnit processing
* add role to SourceUnitID
* Revert "add role to SourceUnitID"
This reverts commit 549e6bede9.
* add role to source unit ID, keep track of resumption using source unit ID instead of just bucket name
* rename bucket -> unitID in UnmarshalSourceUnit
---------
Co-authored-by: Amaan Ullah <aman.ullah.jalal@trufflesec.com>
* Added instrumented transport to docker source api calls to include API call metric emissions
* removed once used variable assignment and setting with transport directly
* Added graphql API for issues and PRs
* enhancements
* more cleanup
* more enhancements
* some final touches
* some more cleanup
* tweaked threads vars
* minor changes
* scan the markdown text not plain text
* ratelimit handling
* added ratelimit handling
* lint error fix
* refactored the thread review comments chunking
* final commit - probably
* refactored the rate limit handling
* adjusted comments
* resolved comments
* remove old camel case func
* renamed featureflag
* resolved martin's comments
* updated test cases
* Removed redundant IncludeRepos mentions.
* removed proto for IncludeRepos and remade protos
* removed another instance of includeRepos
* reverted proto removal and
* actually deprecated the field
* ran make protos
* explicit repositories now bypass wantRepo() filtering entirely.
added ctx to newConnector
* Added test that demonstrates this bypass
* simplified test and focused on enumeration
---------
Co-authored-by: Kashif Khan <70996046+kashifkhan0771@users.noreply.github.com>
The existing implementation of github-experimental requires a github token for the --object-discovery subcommand (that's the only subcommand atm); however, it didn't properly use that token to clone private repos. I used the existing logic in the github/connector.go file to add enable cloning of private repositories.
This was intended to show path:fileName, but `path` in these scopes is
the repo path, not a file. This section was moved in a refactor where
`path` was the file in the old scope, and since both scopes have `path
string`, it was not flagged and easy to miss.
Amaan Ullah
Charlie Gunyon
Cody Rose
Dustin DeckerandCursor Agent
Muneeb Ullah Khan
Brad Larsen
Mustansir
Pascal THUET
Rusted2361andKashif Khan
Shahzad Haider
Gleb Haranin
Jordan Tunstill
joeleonjr
meredith
trufflesteeeve