Chunk.Verify is an odd field - it originally conveys whether a source is going to run with verification, but then, at a certain point in the scanning pipeline, is mutated such that it instead indicates whether the chunk should be scanned with verification - which is not solely dependent on the source's verify flag. This is unnecessarily difficult to understand and maintain. This commit separates those two pieces of information into two flags:
- Chunk.Verify has been renamed to Chunk.SourceVerify
- It is no longer mutated; instead "should this chunk's secrets be verified?" is now captured by a new field on detectableChunk
* Fix Windows file:// URI normalization and index path handling
* add condition to trim leading slash only incase of windows paths
* access path from URL object and identify platform using goos
* simplified path logic
* added test cases for windows powershell & bash
---------
Co-authored-by: Kashif Khan <70996046+kashifkhan0771@users.noreply.github.com>
We have identified some cases in which it is preferable to time a clone out instead of waiting forever. These situations are unusual, so the CLI option to enable this (which I added for testing) is hidden so that we minimize the risk of baking this option into the interface.
This commit prevents TruffleHog from executing arbitrary commands located in archived malicious git repositories. Thanks to Adam Reiser at Cisco Talos for pointing this out!
This approach uses Git's recommended best practice for sanitizing untrusted git configs: git clone all local file:// git repos prior to scanning. Executing git clone does not execute any of the potentially malicious git configs in the untrusted repo directory, and the output only includes "safe" default configs, similar to what we see when cloning from remote.
We explored a few other approaches (allowlist, denylist, etc), but those carried lots of complications.
A few notes about how this works:
This only applies to local repositories scanned using the git subcommand.
Remote git targets are not impacted.
Local git targets are now cloned to temp by default prior to scanning. Users can specify a --clone-path argument if they don't want to use the default temp dir. Users can specify --trust-local-git-config if they want to trust the repo as is and bypass cloning.
Local --bare repos are handled appropriately.
This approach knocks out (most...all?) of this class of malicious git config vulnerabilities.
Testing:
There's coverage for most of the new code, including: test cases for the specific issue reported, local bare clones, local repos with staged commits, etc. All are passing.
* ensures that cmd.Wait() is always called, even if there's a panic in the FromReader function or if stdOut.Close() returns an error
* close stdout and ensure wait is called when handling binaries
* process cleanup improvements
* lint
This is a follow-up to #2713 that fixes the strange test error.
As suspected, the failure was caused by additional diffs not being included in the test's expected data.
* Add Display method to SourceUnit and Kind member to the CommonSourceUnit
* Make SourceUnitID return the ID and a kind
These two values together uniquely represent a unit.
ChunkReporter is more flexible and will allow code reuse for unit
chunking. ChanReporter was added as a way to maintain the original
channel functionality, so this PR should not alter existing behavior.
* issue comment scanning
* save progress
* test
* test for pr comment and issue comment
* add pagination support
* linter stuff
* make linter happy
* remove debug log
* readd logging
* github issue resolved
* var const block and handle rate limit
* remove magic number
* make gitURLParse a public function to use more generally
* fix test bug
* make comment scanning OPT-IN
* Scan binary files for git sources
* Create data chunks in for loop
* Linter feedback and newline commit result
* Use disk buffered reader and chunker function
Cody Rose
Rusted2361andKashif Khan
joeleonjr
Shahzad Haider
Oleksandr Redko
Dustin Decker
Richard Gomez
Miccah
ahrav
Zubair Khan
Bill Rich
trufflesteeeve
