Dylan Ayrey
952df702b3
* feat: iterative decoding pipeline with configurable depth Decoders (base64, UTF-16, escaped unicode) now chain iteratively: each decoder's output is fed back through all decoders until no new transformations occur or --max-decode-depth is reached (default: 5). This finds secrets hidden inside layered encodings, e.g. a base64 Docker auth blob containing a GCP private key, or a UTF-16 file with base64-encoded credentials. At depth=1 behavior is identical to the previous implementation. Extra depths exit early when no new data is produced, so the cost is <5% wall time on a large repo scan. Co-authored-by: Dylan Ayrey <dxa4481@rit.edu> * docs: iterative decoding performance data Co-authored-by: Dylan Ayrey <dxa4481@rit.edu> * comment: explain why PLAIN decoder is skipped at depth > 0 Co-authored-by: Dylan Ayrey <dxa4481@rit.edu> * refactor: extract iterativeDecode, address review feedback - Extract decode loop into standalone iterativeDecode() function, separating decoding from channel dispatch (rosecodym, camgunz). - Drop decodeInput struct, use []byte directly (camgunz). - Remove redundant maxDepth clamp from scannerWorker (camgunz). - Inline decoderType variable (camgunz). - Replace byteSliceSeen with slices.ContainsFunc (camgunz). Co-authored-by: Dylan Ayrey <dxa4481@rit.edu> * fix: remove unused decodeLatency metric (lint) Co-authored-by: Dylan Ayrey <dxa4481@rit.edu> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2.7 KiB
Iterative Decoding Performance
Performance characteristics of the --max-decode-depth feature, which enables
chained decoding (e.g., base64 inside UTF-16, double-encoded base64).
How it works
At depth 0, all decoders run on the original chunk (identical to pre-existing behavior). When a decoder produces new output, that output is fed back through all decoders at the next depth level. The loop exits early when no decoder produces new data, so unused depth levels are effectively free.
The PLAIN (UTF-8) decoder is skipped at depth > 0 since it's a passthrough that never transforms data produced by other decoders (their output is already valid UTF-8/ASCII).
Filesystem scan benchmark
Scanned the trufflehog repository (~4,500 files) with --no-verification
and --concurrency=1 for deterministic comparison.
| Depth | Wall time | Unique results | Delta vs depth=1 |
|---|---|---|---|
| 1 | 8.05s | 924 | — |
| 2 | 8.18s | 927 | +3, +1.6% |
| 3 | 8.09s | 928 | +4, +0.5% |
| 5 | 8.19s | 928 | +4, +1.7% |
| 10 | 8.35s | 932 | +8, +3.7% |
Results converge by depth 3. Depths 4–5 produce no additional decoded data in
this corpus, so they add only a single len() == 0 check per chunk per extra
depth level.
The small unique-result variance at depth 10 is from pre-existing nondeterminism in the concurrent detector workers' dedup ordering, not from the decoding itself.
Per-decoder microbenchmarks
Individual decoder cost is unchanged by this feature (decoders are not modified). For reference, base64 decoder latency on random data:
| Input size | Latency/op | Allocs |
|---|---|---|
| 100 B | ~250 ns | 96 B / 2 |
| 1 KB | ~2.25 µs | 96 B / 2 |
| 10 KB | ~44 ns | 96 B / 2 |
The 10 KB case is fast because random bytes rarely form valid base64 substrings (the 20-character minimum threshold is never met), so the decoder exits after a single O(n) character scan.
Memory overhead
Each depth level that produces new decoded data stores one copy of the output
(typically smaller than the input, since base64 decoding shrinks by ~25%).
A seen list (slice of byte slices) prevents reprocessing identical data.
At depth 5 on a typical chunk, this list has 0–3 entries. No hashing or maps
are used.
Choosing a depth
| Depth | Use case |
|---|---|
| 1 | Legacy behavior, no chaining |
| 2 | Covers base64-in-base64, base64-in-UTF-16, base64-in-escaped-unicode |
| 5 | Default. Handles deeply nested configs with no measurable cost over depth 2 |