Commit Graph
49 Commits
Author SHA1 Message Date
ahravandGitHub b430dae83e [refactor] - lazy buffer retrieval (#2745)
* only create the contentWriter once

* update test

* Lazily fetch buffer from the pool

* fix tests

* fix test

* remove ctx
2024-04-25 08:27:15 -07:00
ahravandGitHub 8ceeb5d5a1 [bug] - Refactor newDiff constructor to avoid double initialization of contentWriter (#2742)
* only create the contentWriter once

* update test

* correclty use mock

* remove deprecated pkg
2024-04-25 08:01:38 -07:00
Cody RoseandGitHub 11452e8a57 Revert "feat(git): scan commit metadata (#2713)" (#2747)
This reverts commit 81a9c813a1.
2024-04-25 10:56:48 -04:00
Richard GomezandGitHub 81a9c813a1 feat(git): scan commit metadata (#2713)
This fixes #2683. It scans the commit author, committer (which is typically GitHub <noreply@github.com> for GitHub, but can be different), and message.

It also scans Git notes.
2024-04-25 10:13:09 -04:00
ahravandGitHub 4a5fbf8417 [refactor] - Update Write method signature in contentWriter interface (#2721)
* Update write method in contentWriter interface

* fix lint
2024-04-23 08:47:53 -07:00
Richard GomezandGitHub baf7ea1458 feat(gitparse): avoid uneeded calls to strconv.Unquote (#2605) 2024-03-22 08:35:10 -07:00
Richard GomezandGitHub aa862e46bb fix(git): decode unicode paths (#2585) 2024-03-19 08:50:27 -07:00
ahravandGitHub 40bbab8add [cleanup] - Extract buffer logic (#2409)
* extract the buffer logic into it's own package

* address comments
2024-02-15 11:40:34 -08:00
ahravandGitHub e8006f1bee 2396 since commit stopped working (#2402)
* Ensure we handle commits with no diffs correctly.

* cleanup

* add nil check

* address comments

* move comment

* revert

* add comment
2024-02-13 07:21:22 -08:00
3b40c4fa63 Update GitParse to handle quoted binary filenames (#2391)
* fix(gitparse): quoted binary files

* fix(gitparse): use bytes.Cut instead of regexp

* fix lint warning

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2024-02-08 09:25:04 -06:00
ahravandGitHub 7b492a690a [feat] - use diff chan (#2387)
* use diff chan

* address comments

* add comment

* address comments

* use old ordering

* add correct author line

* Add required *Commit arg to newDiff

* address comments
2024-02-06 10:06:10 -08:00
ahravandGitHub 843334222c [not-fixup] - Reduce memory consumption for Buffered File Writer (#2377)
* correctly use the buffered file writer

* use value from source

* reorder fields

* use only the DetectorKey as a map field

* correctly use the buffered file writer

* use value from source

* reorder fields

* add tests and update

* Fix issue with buffer slices growing

* fix test

* fix

* add singleton

* use shared pool

* optimize

* rename and cleanup

* use correct calculation to grow buffer

* only grow if needed

* address comments

* remove unused

* remove

* rip out Grow

* address coment

* use 2k default buffer

* update comment allow large buffers to be garbage collected
2024-02-06 09:22:25 -08:00
MiccahandGitHub 01c9ac7b59 Fix binary file hanging bug in git sources (#2388)
Waiting for the sub-command will block until all of `stdout` has been
read. In some cases, we return early due to failed chunking without
reading all of the data, and thus, get stuck waiting for the command to
finish. Closing the pipe will ensure `Wait` does not block on that I/O.
2024-02-05 15:28:49 -08:00
ahravandGitHub 135cc3eb69 [fixup] - correctly use the buffered file writer (#2373)
* correctly use the buffered file writer

* use value from source

* reorder fields

* use only the DetectorKey as a map field

* address comments and use factory function

* fix optional params

* remove commented out code
2024-02-05 10:43:55 -08:00
ahravandGitHub 9867ce8eb8 Allow for configuring the buffered file writer (#2319)
* Write large diffs to tmp files

* address comments

* Move bufferedfilewriter to own pkg

* update test

* swallow write err

* use buffer pool

* use size vs len

* use interface

* fix test

* update comments

* fix test

* Allow for configuring the buffered file writer

* remove unused

* add missing method

* remove

* remove unused

* move parser and commit struct closer to where they are used

* linter change

* fix snifftest

* address comments

* add more kvp pairs to error

* fix test

* update

* add back missing metadata fields

* address comments

* remove bufferedfile writer

* fix

* address comments

* use unint8

* update interface

* adjust interface

* fix tests

* make linter happy

* fix finalize

* address comments

* update test

* address comments

* lint

* remove guard

* fix test

* fix

* add TODO

* fix tests
2024-01-30 12:51:58 -08:00
ahravandGitHub 7c59ff95d5 [feat] - tmp file diffs (#2306)
* Write large diffs to tmp files

* address comments

* Move bufferedfilewriter to own pkg

* update test

* swallow write err

* use buffer pool

* use size vs len

* use interface

* fix test

* update comments

* fix test

* remove unused

* remove

* remove unused

* move parser and commit struct closer to where they are used

* linter change

* add more kvp pairs to error

* fix test

* update

* address comments

* remove bufferedfile writer

* address comments

* adjust interface

* fix finalize

* address comments

* lint

* remove guard

* fix

* add TODO
2024-01-30 12:30:51 -08:00
Richard GomezandGitHub 241e153dfb fix(gitparse): handle fromFileLine edge case (#2206) 2024-01-04 14:53:08 -08:00
Richard GomezandGitHub e72fdb62e4 fix(gitparse): don't trim filename (#2201) 2023-12-14 08:29:46 -08:00
Bill RichandGitHub 00a00ef651 Fix binary handling (#1999) 2023-10-26 10:07:02 -07:00
Savely KrasovskyandGitHub d062834997 initial support for bare repositories (#1499)
* feat: initial support for bare repositories

* feat: use concatenation instead of formatting and os.Getenv instead of os.Environ

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: go-git update with pre-receive hooks fix

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: remove info about pre-receive hook from README.md for now

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: don't scan staged while using --bare option, fixes to make it work with the latest master

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: small refactor according to #1518

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

---------

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
2023-08-03 11:23:41 -05:00
MiccahandGitHub b54683acb9 gitparse: Use an object for currentDiff (#1573)
* gitparse: Use an object for currentDiff instead of a pointer

* gitparse: Use an object for currentCommit instead of a pointer

* Revert "gitparse: Use an object for currentCommit instead of a pointer"

This reverts commit c5f0708b4a.
2023-07-31 11:39:14 -05:00
MiccahandGitHub 6bd48583ae Fix gitparse from panicking on a nil-pointer (#1570) 2023-07-28 11:15:02 -05:00
Zachary RiceandGitHub 3897454dbb add merge support (#1561) 2023-07-27 09:24:49 -05:00
Richard GomezandGitHub f48a635c34 feat: update gitparse logic (#1486) 2023-07-25 17:52:34 -05:00
Zachary RiceandGitHub 452734adc8 remove head from git diff command, rename unstaged to staged (#1439) 2023-06-29 15:33:30 -05:00
Zachary RiceandGitHub c28c70b399 fix new git file plus plus plus bug (#1386) 2023-06-08 18:29:11 -05:00
Bill RichandGitHub f2924f3061 Make sure context lines are properly handled (#1331)
* Make sure context lines are properly handled

* Fix git test to account for context change
2023-05-05 12:51:27 -07:00
ahravandGitHub 714c480931 Add log to track git log size (#1325)
* Add log to track git log size.

* Add calc for large commits and last commit.
2023-05-02 16:36:39 -07:00
Zachary RiceandGitHub 458c79165a fix extra log messages (#1253)
* fix extra log messages

* add small test, move flag to isindex
2023-04-13 09:53:21 -05:00
Dustin DeckerandGitHub 8f10938bf7 forager requires direct access to gitparse.FromReader (#1233) 2023-04-02 17:54:43 -07:00
Zachary RiceandGitHub fb9ae75661 Support for exclude globs at the git log level (#1202)
* init

* seems to be working

* better comment

* rm conditional

* Add more context to exclude-globs description
2023-03-28 10:46:03 -05:00
ahravandGitHub aa47e5e248 Only scanned staged git changes. (#1143) 2023-03-01 08:58:36 -08:00
Bill RichandGitHub ae2d510ced Gitparse message fix (#1125)
* Fix messages being reused

* Add comment about change.
2023-02-23 15:20:54 -08:00
Bill RichandGitHub f1582aafa9 Drop tabs for filenames with spaces (#1115) 2023-02-16 17:15:32 -08:00
Bill RichandGitHub 9158dcaa80 Correctly parse most filenames with ' and ' (#1113) 2023-02-16 14:11:35 -08:00
MiccahandGitHub 161e499142 [chore] Remove logrus from trufflehog (#1095)
* [chore] Remove logrus from trufflehog

* Minor fixes

* Fix logFatal call

* Fix logrus call
2023-02-14 17:00:07 -06:00
Bill RichandGitHub b37080e6a5 Add max commit size (#1079)
* Add max commit size

* Use common.IsDone

* Use breaks instead of return
2023-02-07 15:25:00 -08:00
Bill RichandGitHub af6e3f8fdf Pull gitparse config options out of pkg consts (#1072)
* Pull gitparse config options out of pkg consts.

* Adjust naming
2023-02-04 13:19:23 -08:00
Bill RichandGitHub 00ebb2ed64 Full git log when targeting base merge commit (#1044)
* Full git log when targeting merge commits

* Full log is needed whenever base is specified.
2023-01-26 09:17:54 -08:00
Bill RichandGitHub ac1dd23d37 Limit diff size to prevent out of control memory use. (#1035)
* Limit diff size to prevent out of control memory use.

* Group consts
2023-01-23 10:14:10 -08:00
MiccahandGitHub 4aab7b7276 Buffer commit log processing (#845)
Some very large commits take a lot of time to process, which we can make
progress on while we are scanning the contents of other commits.
2022-10-12 14:55:08 -05:00
ahravandGitHub 92f40c2031 [THOG-709] - Recover from detector panics (#810) 2022-09-22 07:01:10 -07:00
Bill RichandGitHub 912d8e461d Add context so to avoid splitting creds. (#791)
* Add context so to avoid splitting creds.

* Add context newlines to expected results
2022-09-09 15:00:33 -07:00
Bill RichandGitHub 3fe916fe1e add tests (#785) 2022-09-08 21:46:12 -07:00
Bill RichandGitHub 41936169c7 Use gitparse for unstaged changes. (#775) 2022-09-03 18:01:36 -07:00
Bill RichandGitHub 65a7855713 Split parsing from reader (#774) 2022-09-02 11:02:38 -07:00
Dustin DeckerandGitHub fa9479100e Add common sentry recover library and add into goroutines (#738)
* Add common sentry recover library and add into goroutines

* fix nits
2022-08-29 11:45:37 -07:00
ahravandGitHub 20cdcbc970 [bug] - Fix the starting index value for plus line check. (#734)
* Fix the starting index value for plus line check.

* Set the correct source type for notifications.

* Reset old value.

* Fix the starting index value for plus line check.

* Fix len check.

* Reset old value.

* Add tests.

* Update tests.

* Update tests.
2022-08-25 10:45:35 -07:00
Bill RichandGitHub a0d44a39f1 Use trufflesec git parser (#729)
* Use trufflesec git parser.

* wip

* Fix line numbers and linter feedback
2022-08-23 13:29:20 -07:00