mirror of
https://github.com/trufflesecurity/trufflehog.git
synced 2026-06-15 13:24:37 +00:00
952df702b3
* feat: iterative decoding pipeline with configurable depth Decoders (base64, UTF-16, escaped unicode) now chain iteratively: each decoder's output is fed back through all decoders until no new transformations occur or --max-decode-depth is reached (default: 5). This finds secrets hidden inside layered encodings, e.g. a base64 Docker auth blob containing a GCP private key, or a UTF-16 file with base64-encoded credentials. At depth=1 behavior is identical to the previous implementation. Extra depths exit early when no new data is produced, so the cost is <5% wall time on a large repo scan. Co-authored-by: Dylan Ayrey <dxa4481@rit.edu> * docs: iterative decoding performance data Co-authored-by: Dylan Ayrey <dxa4481@rit.edu> * comment: explain why PLAIN decoder is skipped at depth > 0 Co-authored-by: Dylan Ayrey <dxa4481@rit.edu> * refactor: extract iterativeDecode, address review feedback - Extract decode loop into standalone iterativeDecode() function, separating decoding from channel dispatch (rosecodym, camgunz). - Drop decodeInput struct, use []byte directly (camgunz). - Remove redundant maxDepth clamp from scannerWorker (camgunz). - Inline decoderType variable (camgunz). - Replace byteSliceSeen with slices.ContainsFunc (camgunz). Co-authored-by: Dylan Ayrey <dxa4481@rit.edu> * fix: remove unused decodeLatency metric (lint) Co-authored-by: Dylan Ayrey <dxa4481@rit.edu> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
68 lines
2.7 KiB
Markdown
68 lines
2.7 KiB
Markdown
# Iterative Decoding Performance
|
||
|
||
Performance characteristics of the `--max-decode-depth` feature, which enables
|
||
chained decoding (e.g., base64 inside UTF-16, double-encoded base64).
|
||
|
||
## How it works
|
||
|
||
At depth 0, all decoders run on the original chunk (identical to pre-existing
|
||
behavior). When a decoder produces new output, that output is fed back through
|
||
all decoders at the next depth level. The loop exits early when no decoder
|
||
produces new data, so unused depth levels are effectively free.
|
||
|
||
The PLAIN (UTF-8) decoder is skipped at depth > 0 since it's a passthrough
|
||
that never transforms data produced by other decoders (their output is already
|
||
valid UTF-8/ASCII).
|
||
|
||
## Filesystem scan benchmark
|
||
|
||
Scanned the trufflehog repository (~4,500 files) with `--no-verification`
|
||
and `--concurrency=1` for deterministic comparison.
|
||
|
||
| Depth | Wall time | Unique results | Delta vs depth=1 |
|
||
|-------|-----------|----------------|-------------------|
|
||
| 1 | 8.05s | 924 | — |
|
||
| 2 | 8.18s | 927 | +3, +1.6% |
|
||
| 3 | 8.09s | 928 | +4, +0.5% |
|
||
| 5 | 8.19s | 928 | +4, +1.7% |
|
||
| 10 | 8.35s | 932 | +8, +3.7% |
|
||
|
||
Results converge by depth 3. Depths 4–5 produce no additional decoded data in
|
||
this corpus, so they add only a single `len() == 0` check per chunk per extra
|
||
depth level.
|
||
|
||
The small unique-result variance at depth 10 is from pre-existing
|
||
nondeterminism in the concurrent detector workers' dedup ordering, not from the
|
||
decoding itself.
|
||
|
||
## Per-decoder microbenchmarks
|
||
|
||
Individual decoder cost is unchanged by this feature (decoders are not
|
||
modified). For reference, base64 decoder latency on random data:
|
||
|
||
| Input size | Latency/op | Allocs |
|
||
|------------|------------|-----------|
|
||
| 100 B | ~250 ns | 96 B / 2 |
|
||
| 1 KB | ~2.25 µs | 96 B / 2 |
|
||
| 10 KB | ~44 ns | 96 B / 2 |
|
||
|
||
The 10 KB case is fast because random bytes rarely form valid base64 substrings
|
||
(the 20-character minimum threshold is never met), so the decoder exits after a
|
||
single O(n) character scan.
|
||
|
||
## Memory overhead
|
||
|
||
Each depth level that produces new decoded data stores one copy of the output
|
||
(typically smaller than the input, since base64 decoding shrinks by ~25%).
|
||
A `seen` list (slice of byte slices) prevents reprocessing identical data.
|
||
At depth 5 on a typical chunk, this list has 0–3 entries. No hashing or maps
|
||
are used.
|
||
|
||
## Choosing a depth
|
||
|
||
| Depth | Use case |
|
||
|-------|----------|
|
||
| 1 | Legacy behavior, no chaining |
|
||
| 2 | Covers base64-in-base64, base64-in-UTF-16, base64-in-escaped-unicode |
|
||
| 5 | Default. Handles deeply nested configs with no measurable cost over depth 2 |
|