* implemented Source unit for S3.
Implemented. integration test
* use bucket as source unit
* remove code duplication, reuse from Chunks
* remove unnecessary change
* remove unused functions
* revisit tests
* revert unnecessary change
* change SourceUnitKind to s3_bucket
* handle nil objectCount inside scanBucket
* handle nil objectCount outside loop
* add bucket to resume log
* add bucket and role to error log, remove enumerating log
* implement sub unit resumption
* add comment to checkpointer for unit scans
* implement SourceUnitUnmarshaller on source with the new S3SourceUnit, add test to test resumption on multiple buckets with concurrent ChunkUnit processing
* add role to SourceUnitID
* Revert "add role to SourceUnitID"
This reverts commit 549e6bede9.
* add role to source unit ID, keep track of resumption using source unit ID instead of just bucket name
* rename bucket -> unitID in UnmarshalSourceUnit
---------
Co-authored-by: Amaan Ullah <aman.ullah.jalal@trufflesec.com>
Closes#4054.
This PR migrates the S3 source to use AWS SDK v2 by following the migration guide https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/migrate-gosdk.html. This brings compatibility improvements and better long-term support from AWS.
To verify the changes manually, I:
Created a test S3 bucket.
Uploaded sample files containing dummy secrets.
Set up an IAM role for access.
Ran trufflehog s3 and confirmed it behaves as expected post-migration.
additional notes by @rosecodym: The existing S3 automated tests passed, modulo some flakiness that we will address in a follow-up commit.
Mustansir
Muneeb Ullah Khan
Eng Zer Jun
ahrav