* use struct-based SourceMetadataFunc signature across git sources
* incorporated feedback
- pass SourceMetadataInfo by value
- remove LegacySourceMetadataFunc
Chunk.Verify is an odd field - it originally conveys whether a source is going to run with verification, but then, at a certain point in the scanning pipeline, is mutated such that it instead indicates whether the chunk should be scanned with verification - which is not solely dependent on the source's verify flag. This is unnecessarily difficult to understand and maintain. This commit separates those two pieces of information into two flags:
- Chunk.Verify has been renamed to Chunk.SourceVerify
- It is no longer mutated; instead "should this chunk's secrets be verified?" is now captured by a new field on detectableChunk
* Fix Windows file:// URI normalization and index path handling
* add condition to trim leading slash only incase of windows paths
* access path from URL object and identify platform using goos
* simplified path logic
* added test cases for windows powershell & bash
---------
Co-authored-by: Kashif Khan <70996046+kashifkhan0771@users.noreply.github.com>
We have identified some cases in which it is preferable to time a clone out instead of waiting forever. These situations are unusual, so the CLI option to enable this (which I added for testing) is hidden so that we minimize the risk of baking this option into the interface.
This commit adds timing information to the logs when clones complete successfully. There are several other more robust but also more involved interventions we could do, and I think we should do them, but this is a strict improvement that's easy to get out as a first step.
I also added logging at the precise beginning of the clone command so we know when we're doing that.
This was intended to show path:fileName, but `path` in these scopes is
the repo path, not a file. This section was moved in a refactor where
`path` was the file in the old scope, and since both scopes have `path
string`, it was not flagged and easy to miss.
This commit prevents TruffleHog from executing arbitrary commands located in archived malicious git repositories. Thanks to Adam Reiser at Cisco Talos for pointing this out!
This approach uses Git's recommended best practice for sanitizing untrusted git configs: git clone all local file:// git repos prior to scanning. Executing git clone does not execute any of the potentially malicious git configs in the untrusted repo directory, and the output only includes "safe" default configs, similar to what we see when cloning from remote.
We explored a few other approaches (allowlist, denylist, etc), but those carried lots of complications.
A few notes about how this works:
This only applies to local repositories scanned using the git subcommand.
Remote git targets are not impacted.
Local git targets are now cloned to temp by default prior to scanning. Users can specify a --clone-path argument if they don't want to use the default temp dir. Users can specify --trust-local-git-config if they want to trust the repo as is and bypass cloning.
Local --bare repos are handled appropriately.
This approach knocks out (most...all?) of this class of malicious git config vulnerabilities.
Testing:
There's coverage for most of the new code, including: test cases for the specific issue reported, local bare clones, local repos with staged commits, etc. All are passing.
* fix legacy json flag for gitlab private repos
* some code
* incorporated code from main
* remove cloned repositories after the scan is complete
* enhanced the code
* comment addressed
* changed the approach for persisting the repositories for legacy json printing
* addressed comments; updated the variable names
* instead of providing list of refs we want to scan, using --mirror parameter .
removed SkipAdditionalRefs flag, cloning using --mirror will include all the refs.
put the check if userinfo is nil to avoid any crash due to invalid memory access.
* keep both -c and --mirror
* keep useGitMirror on by default for OSS
---------
Co-authored-by: Shahzad Haider <76992801+shahzadhaider1@users.noreply.github.com>
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
Co-authored-by: ahrav <ahravdutta02@gmail.com>
* optimize chunking
* use file size for Docker
* refactor: update chunk size reference in Jenkins test case
- Changed the reference from sources.ChunkSize to sources.DefaultChunkSize for clarity and consistency in the TestJenkinsVariousSizes function.
- Reformatted struct field declarations for improved readability.
---------
Co-authored-by: Kashif Khan <70996046+kashifkhan0771@users.noreply.github.com>
* initial code changes
* move args before clone command
* add comments
* added flag for github source to support backward compatibility
* added flag for gitlab source to support backward compatibility
* inverse the logic for enterprise
* remove print statement
* remove flag defaults
* updated comments and removed CloneRepoUsingTokenInHeader function
* false->true
* Polish logging around archives in git to reduce noise
* Log notable skipping at level 2
* Skipping based on extension happens a lot; bump the log level down to 3
* Bump empty reader log level back down to 5
* Put context check back where it was
* Revert "Put context check back where it was"
This reverts commit da710ccd16.
This PR exports some functionality from the git source so that it's usable in a (new, under development) git-flavored source that does not need to wrap an entire git source in order to operate.
One is getSafeRemoteURL. This is a straightforward change.
The other is handleBinary. Right now, all git binary file handling is invoked from Git.ScanRepo, which is itself invoked by our various git-flavored sources.
Since the actual binary file handling function handleBinary only used two pieces of information from the "git" source, I just removed its receiver. One of the pieces of information was a flag that caused the function to be skipped entirely; I moved this one to the (two) call sites. The other I just forwarded as an argument.
* Fix git binary handling and add a smoke test
* hide stdout
* add failure case to smoke test
* run again with deadlock fix
* Add logic to drain reader in the event of an error
* add tests
* be picky
* set author identity
* suppress linter
---------
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
* ensures that cmd.Wait() is always called, even if there's a panic in the FromReader function or if stdOut.Close() returns an error
* close stdout and ensure wait is called when handling binaries
* process cleanup improvements
* lint
When normalizing the git source config, the base and head refs should be normalized to commit hashes, in case
a branch or tag name was used. The `resolveAndSetCommit` function was returning a boolean value which should
indicate whether the input ref was changed from its original value. While this is in itself not a problem,
the caller (`normalizeConfig`) was using this boolean as an error marker, and returning early in case of `false`.
This meant that if the config was already containing a commit hash for the base or head ref, `resolveAndSetCommit`
would set the flag to `false` and `normalizeConfig` would early return erreneously. This caused the logic to find
the ancestor commit to be skipped which caused the bug in the issue #3220.
Since the `resolveAndSetCommit` function was only used in `normalizeConfig`, the signature has been changed to only
return the commit object and an error. The check for early return in `normalizeConfig` now instead relies on the
commit object being `nil` to indicate a failure to resolve the ref.
Refs: #3220
* Remove specialized handler and archive struct and restructure handlers pkg.
* Refactor RPM archive handlers to use a library instead of shelling out
* make rpm handling context aware
* update test
* Refactor AR/deb archive handler to use an existing library instead of shelling out
* Update tests
* Handle non-archive data within the DefaultHandler
* make structs and methods private
* Remove non-archive data handling within sources
* add max size check
* add filename and size to context kvp
* move skip file check and is binary check before opening file
* fix test
* preserve existing funcitonality of not handling non-archive files in HandleFile
* Handle non-archive data within the DefaultHandler
* rebase
* Remove non-archive data handling within sources
* Adjust check for rpm/deb archive type
* add additional deb mime type
* add gzip
* move diskbuffered rereader setup into handler pkg
* remove DiskBuffereReader creation logic within sources
* update comment
* move rewind closer
* reduce log verbosity
* add metrics for file handling
* add metrics for errors
* make defaultBufferSize a const
* add metrics for file handling
* add metrics for errors
* fix tests
* add metrics for max archive depth and skipped files
* update error
* skip symlinks and dirs
* update err
* Address incompatible reader to openArchive
* remove nil check
* fix err assignment
* Allow git cat-file blob to complete before trying to handle the file
* wrap compReader with DiskbufferReader
* Allow git cat-file blob to complete before trying to handle the file
* updates
* use buffer writer
* update
* refactor
* update context pkg
* revert stuff
* update test
* fix test
* remove
* use correct reader
* add metrics for file handling
* add metrics for errors
* fix tests
* rebase
* add metrics for errors
* add metrics for max archive depth and skipped files
* update error
* skip symlinks and dirs
* update err
* fix err assignment
* rebase
* remove
* Update write method in contentWriter interface
* Add bufferReadSeekCloser
* update name
* update comment
* fix lint
* Remove specialized handler and archive struct and restructure handlers pkg.
* Refactor RPM archive handlers to use a library instead of shelling out
* make rpm handling context aware
* update test
* Refactor AR/deb archive handler to use an existing library instead of shelling out
* Update tests
* add max size check
* add filename and size to context kvp
* move skip file check and is binary check before opening file
* fix test
* preserve existing funcitonality of not handling non-archive files in HandleFile
* Handle non-archive data within the DefaultHandler
* rebase
* Remove non-archive data handling within sources
* Handle non-archive data within the DefaultHandler
* add gzip
* move diskbuffered rereader setup into handler pkg
* remove DiskBuffereReader creation logic within sources
* update comment
* move rewind closer
* reduce log verbosity
* make defaultBufferSize a const
* add metrics for file handling
* add metrics for errors
* fix tests
* add metrics for max archive depth and skipped files
* update error
* skip symlinks and dirs
* update err
* Address incompatible reader to openArchive
* remove nil check
* fix err assignment
* wrap compReader with DiskbufferReader
* Allow git cat-file blob to complete before trying to handle the file
* updates
* use buffer writer
* update
* refactor
* update context pkg
* revert stuff
* update test
* remove
* rebase
* go mod tidy
* lint check
* update metric to ms
* update metric
* update comments
* dont use ptr
* update
* fix
* Remove specialized handler and archive struct and restructure handlers pkg.
* Refactor RPM archive handlers to use a library instead of shelling out
* make rpm handling context aware
* update test
* Refactor AR/deb archive handler to use an existing library instead of shelling out
* Update tests
* add max size check
* add filename and size to context kvp
* move skip file check and is binary check before opening file
* fix test
* preserve existing funcitonality of not handling non-archive files in HandleFile
* Adjust check for rpm/deb archive type
* add additional deb mime type
* update comment
* go mod tidy
* update go mod
* Add a buffered file reader
* update comments
* use Buffered File Readder
* return buffer
* update
* fix
* return
* go mod tidy
* merge
* use a shared pool
* use sync.Once
* reorganzie
* remove unused code
* fix double init
* fix stuff
* nil check
* reduce allocations
* updates
* update metrics
* updates
* reset buffer instead of putting it back
* skip binaries
* skip
* concurrently process diffs
* close chan
* concurrently enumerate orgs
* increase workers
* ignore pbix and vsdx files
* add metrics for gitparse's Diffchan
* fix metric
* update metrics
* update
* fix checks
* fix
* inc
* update
* reduce
* Create workers to handle binary files
* modify workers
* updates
* add check
* delete code
* use custom reader
* rename struct
* add nonarchive handler
* fix break
* add comments
* add tests
* refactor
* remove log
* do not scan rpm links
* simplify
* rename var
* rename
* fix benchmark
* add buffer
* buffer
* buffer
* handle panic
* merge main
* merge main
* add recover
* revert stuff
* revert
* revert to using reader
* fixes
* remove
* update
* fixes
* linter
* fix test
* fix comment
* update field name
* fix
* Remove specialized handler and archive struct and restructure handlers pkg.
* Refactor RPM archive handlers to use a library instead of shelling out
* make rpm handling context aware
* update test
* Refactor AR/deb archive handler to use an existing library instead of shelling out
* Update tests
* add max size check
* add filename and size to context kvp
* move skip file check and is binary check before opening file
* fix test
* preserve existing funcitonality of not handling non-archive files in HandleFile
* Adjust check for rpm/deb archive type
* add additional deb mime type
* update comment
* Remove specialized handler and archive struct and restructure handlers pkg.
* Refactor RPM archive handlers to use a library instead of shelling out
* make rpm handling context aware
* update test
* Refactor AR/deb archive handler to use an existing library instead of shelling out
* Update tests
* add max size check
* add filename and size to context kvp
* move skip file check and is binary check before opening file
* fix test
* preserve existing funcitonality of not handling non-archive files in HandleFile
* Adjust check for rpm/deb archive type
* add additional deb mime type
* update comment
* go mod tidy
* update go mod
* go mod tidy
* add comment
* update max depth check to >
* go mod tidy
* rename
* [refactor] - Refactor Archive Handling Logic - Part 4: Non-Archive Data Handling and Cleanup (#2704)
* Handle non-archive data within the DefaultHandler
* make structs and methods private
* Remove non-archive data handling within sources
* Handle non-archive data within the DefaultHandler
* rebase
* Remove non-archive data handling within sources
* add gzip
* move diskbuffered rereader setup into handler pkg
* remove DiskBuffereReader creation logic within sources
* move rewind closer
* reduce log verbosity
* make defaultBufferSize a const
* use correct reader
* address comments
* update test
* [feat] - Add Prometheus Metrics for File Handlers (#2705)
* add metrics for file handling
* add metrics for errors
* add metrics for file handling
* add metrics for errors
* fix tests
* add metrics for max archive depth and skipped files
* update error
* skip symlinks and dirs
* update err
* fix err assignment
* add metrics for file handling
* add metrics for errors
* fix tests
* rebase
* add metrics for errors
* add metrics for max archive depth and skipped files
* update error
* skip symlinks and dirs
* update err
* fix err assignment
* rebase
* remove
* update metric to ms
* update comments
* address comments
* reduce indentations
* add metrics for archive depth
* [bug] - Enhanced Archive Handling to Address Interface Constraints (#2710)
* add metrics for file handling
* add metrics for errors
* add metrics for file handling
* add metrics for errors
* fix tests
* add metrics for max archive depth and skipped files
* update error
* skip symlinks and dirs
* update err
* Address incompatible reader to openArchive
* remove nil check
* fix err assignment
* wrap compReader with DiskbufferReader
* add metrics for file handling
* add metrics for errors
* fix tests
* rebase
* add metrics for errors
* add metrics for max archive depth and skipped files
* update error
* skip symlinks and dirs
* update err
* fix err assignment
* rebase
* remove
* update metric to ms
* update comments
* address comments
* reduce indentations
* replace diskbuffereader with bufferedfilereader
* updtes
* add metric back
* [bug] - Fix bug and simplify git cat-file command execution and output handling (#2719)
* add metrics for file handling
* add metrics for errors
* add metrics for file handling
* add metrics for errors
* fix tests
* add metrics for max archive depth and skipped files
* update error
* skip symlinks and dirs
* update err
* Address incompatible reader to openArchive
* remove nil check
* fix err assignment
* Allow git cat-file blob to complete before trying to handle the file
* wrap compReader with DiskbufferReader
* Allow git cat-file blob to complete before trying to handle the file
* updates
* revert stuff
* update test
* remove
* add metrics for file handling
* add metrics for errors
* fix tests
* rebase
* add metrics for errors
* add metrics for max archive depth and skipped files
* update error
* skip symlinks and dirs
* update err
* fix err assignment
* rebase
* remove
* update metric to ms
* update comments
* address comments
* reduce indentations
* inline
This is a follow-up to #2713 that fixes the strange test error.
As suspected, the failure was caused by additional diffs not being included in the test's expected data.
This fixes#2683. It scans the commit author, committer (which is typically GitHub <noreply@github.com> for GitHub, but can be different), and message.
It also scans Git notes.
When we fail to clone a git repository we log the command output to help with diagnosis. However, this output can include credentials in certain cases (such as certain errors associated with redirects). We don't want to log credentials when this happens.
* Add Display method to SourceUnit and Kind member to the CommonSourceUnit
* Make SourceUnitID return the ID and a kind
These two values together uniquely represent a unit.
This is a follow-up to #1912, which used the headers from the response to determine rate-limiting information, instead of using the values from RateLimitError.Rate. Although that logic seemed solid, I discovered that it did not work in some circumstances. This lead to the "unexpected" path more often than intended, and periodic instances where requests would be made before the ratelimit was refreshed.
Waiting for the sub-command will block until all of `stdout` has been
read. In some cases, we return early due to failed chunking without
reading all of the data, and thus, get stuck waiting for the command to
finish. Closing the pipe will ensure `Wait` does not block on that I/O.
* correctly use the buffered file writer
* use value from source
* reorder fields
* use only the DetectorKey as a map field
* address comments and use factory function
* fix optional params
* remove commented out code
Amaan Ullah
Cody Rose
Rusted2361andKashif Khan
meredith
joeleonjr
Shahzad HaiderandKashif Khan
Sahil SilareandKashif Khan
Charlie Gunyon
Richard Gomez
Valentin B.
Zachary Rice
Miccah
Cody Rose