Line 28 read 'Temporarily disabled while the team keeps required VMP 3.8.x targets on the safe high-budget path'. That is stale relative to the current code: canGeneralizeStructuredLoopHeader (lifter/core/LifterClass.hpp) gates generalization on path-solve context plus nine operational guards, and the corresponding loop_generalization_* microtests pass on main. Describe the actual gating and point readers at docs/LOOP_HANDLING.md.
Co-authored-by: yusufcanislek <yusuf.canislek@meetdandy.com>
Captures the three-phase architecture (detect/generalize/consume), the path-solve context gating table, the GeneralizedLoopControlFieldState layout, mergeValue's widenFirstBackedge contract, the full set of retrieve_generalized_loop_* helpers, and the hardcoded reference-sample addresses (kThemidaControlCursorSlot, the three resolveTargetedThemidаR9 instruction addresses with fire-counts on the reference binary).
Documents known limitations at the bottom: REP SCAS, VMP 3.6 INT 2 dispatcher, the reference-sample hardcodes, unrolling/LICM, multi-way backedges.
Flags that SCOPE.md's 'loop-header generalization temporarily disabled' entry appears to be stale: the code gates generalization on path-solve context (ConditionalBranch / DirectJump / resolved IndirectJump) rather than disabling it wholesale. Not changed in this PR; maintainer decision.
Co-authored-by: yusufcanislek <yusuf.canislek@meetdandy.com>
* docs: sync rewrite workflow guidance
* docs: drop machine-local pointers and fix stale README branch link
* lifter: allow resolved indirect jumps to participate in structured loop generalization
When a register-indirect jmp has already been resolved to a concrete target via solvePath (ConstantInt or solver), it's no longer speculative. If the target also points backward at a visited block, treat it as a loop back-edge for generalization purposes, the same way a direct or conditional jump would be treated.
Introduces currentPathSolveAllowsStructuredLoopGeneralizationForResolvedTarget() alongside the existing narrow predicate. canGeneralizeStructuredLoopHeader gains an opt-in targetResolvedConcretely parameter that routes through the widened check. getLiftedBackedgeBB uses the widened variant so back-edge reuse fires for resolved indirect jumps. resolveTargetBlock passes targetResolvedConcretely=true (its entry condition requires a concrete destination) and extends stackBypassGeneralizedLoopAddresses to include IndirectJump-context inserts.
Ret-path contexts remain excluded. Tests updated: the old runLoopGeneralizationIndirectJumpBlocked splits into runLoopGeneralizationIndirectJumpBlockedWhenUnresolved (unchanged semantics) and runLoopGeneralizationIndirectJumpAllowedWhenResolved (new). runPendingGeneralizedLoopBlockedByContext becomes runPendingGeneralizedLoopByContext with an expectReuse parameter; Ret still expects no reuse, IndirectJump with a resolved target now expects reuse.
---------
Co-authored-by: yusufcanislek <yusuf.canislek@meetdandy.com>
The windows-latest preinstalled clang-cl (currently 20.1.8 at
`C:\Program Files\LLVM\bin\clang-cl.exe`) produces a lifter binary
that segfaults on calc_fib before emitting any IR, causing the rewrite
gate to fail. Clang 21.1.8 has been verified locally to compile the
lifter into a binary that lifts both calc_fib and calc_sum_array to
their expected constant returns (`ret i64 13` and `ret i64 150`).
Rolling back to clang 18.x is not an option: the runner image's MSVC STL
(14.44+) hard-requires clang 19.0.0 or newer via a static_assert in
yvals_core.h. Clang 21 satisfies that bound and dodges the clang 20.1.8
miscompile.
Upgrading via `choco upgrade llvm --version=21.1.8` keeps the existing
`C:\Program Files\LLVM\bin\clang-cl.exe` path valid, so the rest of
the pipeline (Resolve LLVM_DIR, Resolve clang-cl, Configure, Build) is
unchanged.
## Changes
- `.github/workflows/rewrite-strict-gate.yml`: add an "Upgrade clang-cl
to 21.1.8" step before `Resolve LLVM_DIR` that runs `choco upgrade
llvm` and pins `CMAKE_{C,CXX}_COMPILER` to the upgraded binary.
- `scripts/rewrite/instruction_microtests.json`: drop the `ci_skip`
entries on `calc_fib` and `calc_sum_array`.
- `docs/SCOPE.md`: bump the corpus counts to 33 samples / 177 runtime
semantic cases.
## Follow-up
Investigating the underlying clang 20.1.8 miscompile in the lifter is
still worth doing \u2014 it's almost certainly UB somewhere in the
structured-loop recovery path that clang 21 happens to tolerate. Tracked
separately.
Co-authored-by: NaC-L <nac-l@users.noreply.github.com>
PR #93 un-skipped both samples after a clean local Release build proved
they lift correctly, but the windows-latest CI lane still fails on them
`Lifter failed for calc_fib` (run 24077021868). The HANDOFF note that
windows-latest clang-cl produces a different codegen shape than the
locally pinned clang-cl turned out to be the actual root cause; the
"stale build cache" theory only explained the local symptom.
Restoring the `ci_skip` entries unbreaks the rewrite-strict-gate and
rewrite-quick-gate workflows. Real fix tracked as a follow-up: either
teach the lifter the CI codegen shape, or pin the rewrite CI lane to a
toolchain that matches the local one byte-for-byte.
Also reverts the `docs/SCOPE.md` corpus counts to 31 samples / 175 cases.
Co-authored-by: NaC-L <nac-l@users.noreply.github.com>
Both samples were originally CI-skipped because windows-latest clang-cl
produced loop/array codegen shapes that tripped the lifter on CI even
though local runs passed. Since then the rewrite CI lane has been pinned
to the same LLVM 18.1.8 clang-cl used locally (eb49a35, 949acaa, a28a368)
and several structured loop recovery fixes have landed (2989e5a, 2eaa22e),
so the codegen mismatch that motivated the skips is gone.
Verified locally with a clean Release build (`cmd /c scripts\dev\configure_iced.cmd`
followed by `build_iced.cmd`):
- `calc_fib` lifts to `ret i64 13` and passes its semantic case
- `calc_sum_array` lifts to `ret i64 150` and passes its semantic case
- `python test.py all` is fully green: semantic 33/33 (was 31/31),
baseline, micro --check-flags, full handler suite 115/119, determinism
Drops the two `ci_skip` entries from `instruction_microtests.json` and
updates `docs/SCOPE.md` corpus counts to 33 samples / 177 cases.
Co-authored-by: NaC-L <nac-l@users.noreply.github.com>
- Add lift_punpcklqdq handler in Semantics_Misc.ipp (XMM dest, low-quadword
interleave from dest+src into a 128-bit result; rejects MMX/non-XMM forms
via the standard not_implemented bailout)
- Wire OPCODE(punpcklqdq, PUNPCKLQDQ) in x86_64_opcodes.x and add a missing
trailing newline
- Add manual punpcklqdq case to TestInstructions.cpp (rdrand-style XMM seed)
and matching seeds in build_full_handler_seed.py
- Regenerate oracle_seed_full_handlers{,_enriched}.json, oracle_seed_vectors.json,
and oracle_vectors_full_handlers.json with two punpcklqdq vectors
(basic interleave, low-source-zero edge case)
- Drop ci_skip on calc_cout in instruction_microtests.json now that the STL
PUNPCKLQDQ path lifts cleanly (4/4 semantic cases pass locally)
- Keep calc_fib and calc_sum_array ci_skipped: they still trip a separate
lifter dyn_cast assertion that is not related to PUNPCKLQDQ; tracked as
follow-up
- Update docs/SCOPE.md handler counts (115/119 covered, 4 intentional skips)
and corpus counts (31 active samples / 175 cases)
Co-authored-by: NaC-L <nac-l@users.noreply.github.com>
Two new post-optimization passes that run after the final O2 pipeline:
PrototypeMinimizationPass:
- Removes unused function arguments based on Argument::use_empty()
- Typical reduction: 34 params -> 0-2 (e.g. @main(i64 %RCX) instead of all 16 GPRs + 16 XMMs + 2 ptrs)
- Splices basic blocks into new function, remaps argument uses, erases old function
- Updated check_semantic.py to parse actual IR signatures instead of hardcoded 34-param list
CanonicalNamingPass:
- Strips address-derived suffixes from block/value names for deterministic output
- Blocks: entry, bb1, bb2, ... (sequential)
- Values: semantic prefix preserved, address suffix removed (realadd-5368713230- -> realadd)
- Same input now produces byte-identical IR across rebuilds
Also fixed writeFunctionToFile to use stored module pointer M instead of
fnc->getParent() (dangling after prototype minimization erases the old function).
Review fixes:
- CanonicalNamingPass: use StringMap<unsigned> instead of DenseMap<StringRef> (dangling key)
- PrototypeMinimizationPass: restrict call rewriting to CallInst (not InvokeInst/CallBrInst)
- PrototypeMinimizationPass: guard F->eraseFromParent() with use_empty() check
- check_semantic.py: widen define regex to handle dso_local and other prefixes
All 28 samples pass, 146 semantic cases, 56 golden hashes updated.
* test: add jump table regression suite (5 samples, 39 semantic cases)
Add 5 new jump table test cases covering the major dispatch patterns:
- jumptable_rel32.asm: RIP-relative dword offset table (lea+movsxd+add+jmp)
- jumptable_shifted.asm: base-shifted range check (sub before index)
- jumptable_shared_targets.asm: multiple cases sharing handlers
- jumptable_computation.asm: case bodies with symbolic arithmetic
- calc_jumptable_large.c: 16-case dense C switch compiled at /O2
All 5 pass lifting and semantic validation (39 new cases, 146 total).
Update golden hashes (46 -> 56 files), manifest, and docs.
* fix(ci): exclude C-compiled samples from golden IR hashes
C-compiled samples (calc_*) produce address-dependent IR because the
linker places symbols at different addresses depending on toolchain
version, link order, and build environment. The determinism check
comment (test.py L123-125) already documented this exclusion policy
but the golden hash file included them anyway, causing rewrite-quick-gate
to fail on CI.
Remove all 14 calc_* entries from golden_ir_hashes.json (56 -> 42).
C-compiled sample correctness is still validated by semantic tests.
---------
Co-authored-by: yusufcanislek <yusuf.canislek@meetdandy.com>
naci
yusufcanislek
wcscpy
Chrizz
naci
pseuxide
r3bb1t