Commit Graph

37 Commits

Author SHA1 Message Date
Mustansir b3b1d4180a [INS-258] Revert includeRepos removal from GitHub source (#4673)
* Revert "Remove include repos (#4469)"

This reverts commit 24c73b0cb6.

* generate protos

* preserve tests for repositories
2026-01-27 19:01:50 +05:00
Jordan Tunstill 24c73b0cb6 Remove include repos (#4469)
* Removed redundant IncludeRepos mentions.

* removed proto for IncludeRepos and remade protos

* removed another instance of includeRepos

* reverted proto removal and

* actually deprecated the field

* ran make protos
2025-10-21 15:28:12 -07:00
Shahzad Haider 2114e77d56 Fix legacy json flag for Github and Gitlab private repos (#4386)
* fix legacy json flag for gitlab private repos

* some code

* incorporated code from main

* remove cloned repositories after the scan is complete

* enhanced the code

* comment addressed

* changed the approach for persisting the repositories for legacy json printing

* addressed comments; updated the variable names
2025-09-03 11:19:28 +05:00
Kashif Khan 6f36a477c6 Added a dedicated optional flag to ignore gists during scan (#4423)
* Added a dedicated optional flag to ignore gists during scan

* added test case
2025-08-25 16:07:05 +05:00
Kashif Khan 07c16636eb Enable cloning repository to a specified location with retention option (#4408)
* Enabled cloning repositories to a specified path with retention option

* Fixes after testing

* resolved lint issue

* resolved comments

* enabled clone path for github basic auth
2025-08-20 18:10:02 +05:00
Kashif Khan 92e9157267 Github and Gitlab Authentication via http.extraHeader for cloning Repositories (#4139)
* initial code changes

* move args before clone command

* add comments

* added flag for github source to support backward compatibility

* added flag for gitlab source to support backward compatibility

* inverse the logic for enterprise

* remove print statement

* remove flag defaults

* updated comments and removed CloneRepoUsingTokenInHeader function

* false->true
2025-05-27 11:24:43 +05:00
0x1 1276d262f2 [scan-9] Update enumeration logic (#3626)
* renaming to enumeration

* update enumeration

* comments

* remove commented out func

---------

Co-authored-by: Miccah Castorina <m.castorina93@gmail.com>
2024-11-25 14:13:03 -05:00
Miccah 6a77b73f51 [chore] Log non-fatal errors encountered during a scan (#3612)
* [chore] Log non-fatal errors encountered during a scan

* Update tests

* Update tests
2024-11-18 16:52:22 -08:00
JonZeolla 4ea311dea9 feat: add github comments timeframe filtering (fixes #3388) (#3390)
* feat: add github comments timeframe filtering

* fixup and generate protos

* Cleanup

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2024-10-15 15:13:36 -04:00
Richard Gomez 8e90c4e669 Scan GitHub wikis #2233 2024-01-31 10:52:24 -05:00
ahrav 5c6ce693c1 [feat] - Make skipping binaries configurable (#2226)
* Make skipping binaries configurable

* remove ioutil

* fix

* address comments

* address comments

* use multi-reader

* remove print

* use const

* fix test

* fix my stupidness
2023-12-15 11:46:27 -08:00
joeleonjr 699547b7d3 consolidated pr and issue descr/comment flags (#1827) 2023-09-27 15:54:02 -04:00
joeleonjr 1e42dae734 added PR and Issue body scanning (#1816)
* added PR and Issue body scanning; adjusted CLI args to fit

* removed print statement from debugging

* removed exclude-commits; adjusted CLI flags

* minor changes to match main branch

* fixing logic

* updating README for --issues and --prs
2023-09-26 12:25:48 -04:00
Miccah dbcb888063 Update Source interface to use SourceID and JobID types (#1774)
The previous implementation used int64 for both, which can be mixed up
easily. Using distinct types adds a layer of type safety checked by the
compiler.
2023-09-14 11:28:24 -07:00
Miccah 72b6a9ec6b Add a SourceType constant to all source packages (#1768) 2023-09-12 17:23:25 -07:00
Miccah be4d0bcb41 Refactor SourceManager to remove Enrollment (#1740)
* Refactor SourceManager to remove Enrollment

Initializing the Source will be the responsibility of the caller. The
SourceManager exposes a GetIDs method for getting a source and job ID.

* Update tests

* Update engine usage

* Update apiClient interface to have one GetIDs method

* Update SourceManager usage in engine
2023-09-12 16:58:38 -07:00
Zubair Khan db89e345d7 correct logging output for github comments and add oss flags (#1632)
* correct logging output

* add flags

* respect oss cli flags for github comment scanning

* improve copy
2023-08-16 18:23:59 -04:00
Miccah 1cd600f70f Use SourceManager in engine (#1586)
* Add SourceManager to Engine struct

* Update Engine methods to use the SourceManager

* Fix GCS test

The original was testing that `Init()` errors weren't surfaced in
`Finish()`, but the `SourceManager` changed that behavior.

* JobProgress race fixes

* Add contextual values

* Remove unused code

* Add debug logs

* Rename WithConcurrency to WithConcurrentSources

* Always forward chunks to the output chunks channel
2023-08-03 13:36:30 -05:00
Brendan Shaklovitz da5301ea1e Exit with non-zero exit code on chunk source error (#1286)
* Exit with non-zero exit code on chunk source error

* Exit with a non-zero exit code whenever we hit an error getting
  chunks. Previously the error would be logged but trufflehog would exit
  with a 0 (success) status code.

* fix gcs test

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
Co-authored-by: ahrav <ahravdutta02@gmail.com>
2023-06-26 11:39:57 -05:00
SAYGIN Metin f2139a7615 Github filter support for exclude and include (#1087)
* test

* Add missing head and base hash back.

---------

Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com>
2023-02-14 08:40:53 -08:00
ahrav c5c8d10d28 [chore] - Remove monolithic config struct (#1091)
* REmove monolithic config struct.

* fix broken test.
2023-02-10 12:43:00 -08:00
Miccah d317ddb51a [chore] Remove logrus from circleci, filesystem, gitlab, and s3 sources (#1089)
* [chore] Remove logrus from circleci, filesystem, gitlab, and s3 sources

* Address comments
2023-02-10 11:02:55 -06:00
Miccah 0ce72ccda3 [chore] Remove logrus from github source (#1086)
* [chore] Remove logrus from github source

* Fix handleRateLimit test

* Fix tests
2023-02-09 18:02:04 -06:00
Miccah 58e8c1e4ac [chore] Remove logrus from engine package (#1085) 2023-02-09 16:55:19 -06:00
ahrav 936a139596 Allow using a glob for include list. (#977)
* Allow using a glob for include list.

* Update command flag.

* Make comment more clear.

* update comment.

* Allow scanning repo and org at the same time.
2022-12-16 13:28:16 -08:00
ahrav 92f40c2031 [THOG-709] - Recover from detector panics (#810) 2022-09-22 07:01:10 -07:00
Dustin Decker fa9479100e Add common sentry recover library and add into goroutines (#738)
* Add common sentry recover library and add into goroutines

* fix nits
2022-08-29 11:45:37 -07:00
ahrav 73f9d3f0a0 [chore] - Use config struct instead of pointer for engine scans. (#709)
* Use a config struct instead of pointer when scanning engine sources.

* use config.
2022-08-12 09:56:24 -07:00
ahrav dcc102a81c [Thog-371] Utilize config struct for engine scans (#700)
* Use a config struct when scanning and engine source.

* fix tests.

* Move test_helpers to the sources pkg.

* Handle ScanGit error in tests.

* adderss comments.

* Use functional options.

* Remove temp var.

* Add better var names for the setup functions for each config.

* Remove unused var.

* fix error logs.

* fix error logs.

* single line.

* remove blank lines.
2022-08-10 10:11:13 -07:00
Miccah 6fa2171a22 Refactor Engine to wait for workers in a Finish method (#581)
* Refactor Engine to wait for workers in a Finish method

This should allow the engine to run multiple concurrent scans if
desired before shutting down.

Additionally, this commit refactors some of the printing logic to the
output package.

* Fix tests
2022-05-25 11:35:44 -05:00
ahrav d2605354fe [THOG-332 ]Remove TokenSource interface from the init method of Source. (#539)
* Remove TokenSource interface from the init method of Source.

* Remove proto message.

* Remove proto message.

* Fix tests.

* Fix filesystem test.
2022-05-13 14:35:06 -07:00
ahrav b0d79180f6 [THOG-314] Add new parameter to the Init method for the source interface. (#529)
* Add new parameter to the Init method for the source interface.

* Add Oauth Token service.

* remove .test file.

* remove .test file.

* Fix param spelling.

* fix tests with new param in init

* Add missing gock lib.
2022-05-10 11:11:43 -07:00
Miccah 78b344d7b8 Add --include-members flag to scan all members of an organization (#118)
* Add --member-repos flag to scan all members of an organization

* Move flag to the githubScan sub-command

Co-authored-by: Miccah Castorina <miccah.castorina@trufflesec.com>
2022-04-01 18:22:37 -07:00
Dustin Decker c20e9f4732 improvements 2022-03-04 08:39:17 -08:00
Dustin Decker 77418fb3f8 module v3 2022-02-15 18:54:47 -08:00
Dustin Decker 152ef6d4e1 add include forks option (#37) 2022-02-15 18:54:47 -08:00
Dustin Decker 03ead2f7ed Integrate GitHub source 2022-02-15 18:54:47 -08:00