In-Place Spectre Attacks vs Out-of-Place Spectre Attacks: What Long‑Running Systems Must Learn

TL;DR: In‑place Spectre attacks that reuse stale branch‑target entries bypass most hardware mitigations, so JIT runtimes must purge or fence stale code caches immediately—just as spacecraft software proves that longevity requires proactive integrity checks.

Introduction: From Martian Sols to Millisecond JIT Cycles

On 30 September 2026 the Curiosity rover celebrated its 5 000‑sol anniversary, marking more than a decade of autonomous operation on a hostile planet. Its navigation cameras have taken thousands of panoramas, its software has been patched remotely, and—crucially—its execution environment has never suffered a speculative‑execution side‑channel breach.

Contrast this with the recent demonstration from Vrije Universiteit Amsterdam and Scuola Superiore Sant’Anna: a practical in‑place Spectre variant—Branch Target Reuse (BTR)—that resurrects stale indirect‑branch predictions to leak data from just‑in‑time (JIT) compiled code. The attack sidesteps the bulk of hardware mitigations that were designed for the classic out‑of‑place Spectre‑v2 scenario.

The juxtaposition is stark: a mission that has survived 5 000 sols without a speculative‑execution compromise versus a class of software that recompiles code every few milliseconds and is now vulnerable to a new class of attacks. The lesson is not that space software is magically immune, but that longevity is achieved through layered, proactive defenses that anticipate the failure modes of the underlying execution model.

This article expands on the original overview, providing concrete implementation details, real‑world examples, trade‑offs, and practical guidance for teams that ship long‑running, JIT‑heavy services (browsers, server‑side runtimes, eBPF filters, WebAssembly hosts, etc.).

1. Spectre 101: A Quick Recap

All variants share a common denominator: they exploit the microarchitectural state that survives beyond the architectural view of the program. The CPU’s branch predictor is a perfect example—its entries are shared across contexts and persist across self‑modifying code boundaries unless explicitly cleared.

2. Out‑of‑Place vs In‑Place Spectre Variants

2.1 Out‑of‑Place (Classic) Spectre‑v2

Out‑of‑place attacks assume the attacker can train the BTB to mis‑speculate to a gadget that lives elsewhere in the victim’s address space. The typical steps are:

1. Training phase – The attacker repeatedly executes an indirect branch that resolves to a chosen address, poisoning the BTB entry for a target index.
2. Trigger phase – The victim executes the same indirect branch with a different operand; the CPU speculatively jumps to the poisoned address.
3. Leak phase – The speculative gadget performs a microarchitectural side‑effect (e.g., cache line load) that encodes secret data.

Because the gadget must already exist in the victim’s binary, attackers often rely on Return‑Oriented Programming (ROP) or JIT‑generated “gadgets” that are inadvertently present. The attack complexity is therefore dominated by gadget discovery and BTB training.

Mitigations that flush the BTB on context switches (e.g., Linux’s flushbt on schedswitch) or restrict BTB visibility across privilege levels (IBRS) are effective because the attacker’s training does not survive the flush.

2.2 In‑Place (Branch Target Reuse – BTR)

In‑place attacks, exemplified by Branch Target Reuse, flip the script:

1. JIT compilation creates a piece of native code at address A and registers it as the target of an indirect branch (e.g., a function pointer).
2. The JIT later frees the memory at A and re‑uses the same slot for attacker‑controlled code (address B).
3. The BTB entry for the indirect‑branch slot still points to A (or a stale entry that resolves to A). Because the CPU does not automatically invalidate BTB entries on self‑modifying code, the branch predictor still believes the target is A.
4. When the victim later executes the indirect branch, speculation follows the stale entry, which now points to B (the attacker’s code). The speculative execution of B leaks data before the architectural pipeline realizes the mis‑prediction.

Key differences:

Because BTR re‑uses the victim’s own branch‑target slot, it bypasses many mitigations that assume the attacker’s code resides outside the victim’s address space. The attack works even when IBRS and STIBP are enabled, as long as the same process re‑uses the indirect‑branch slot.

3. Real‑World Demonstrations of BTR

The original paper (Wiebing et al., 2026) evaluated BTR against three JIT environments:

All three attacks succeeded on Intel Core i7‑12700K (microcode 0xB0) and AMD Zen 4 CPUs with IBRS enabled, demonstrating that BTR is architecture‑agnostic as long as the processor retains BTB entries across self‑modifying code.

4. Why JIT‑Heavy Systems Are Prime Targets

The Curiosity rover runs a radiation‑hardened VxWorks kernel on a 200 MHz RAD750 processor. Its code base is essentially static; the BTB entries that are trained during the mission remain valid for the entire lifetime because the rover never recompiles native code at runtime. The rover’s engineers mitigate the few speculative‑execution risks that exist (e.g., by disabling speculative fetch of privileged code) and rely on periodic checksum validation of firmware images transmitted from Earth.

In contrast, a modern browser may compile 10 000 JavaScript functions per second, each occupying a 4 KB page that is later freed and re‑allocated. The BTB entries for indirect calls (e.g., virtual method dispatch) are trained on the old code, then become stale when the page is reclaimed. If an attacker can cause the JIT to allocate a malicious function at the same address, the stale BTB entry becomes a speculative execution primitive.

The core takeaway: dynamic code generation demands explicit BTB management, just as spacecraft software demands explicit integrity checks.

5. Hardware Mitigations: What They Offer and Where They Fall Short

Key observation: All existing hardware mitigations assume cross‑domain contamination (different processes, privilege levels, or threads). BTR exploits intra‑process reuse, which is invisible to IBRS/STIBP. Until microcode adds explicit self‑modify invalidation, software must fill the gap.

6. Operating‑System Level Defenses

6.1 BTB Flush Syscalls

Linux 6.6 introduced speculationforceclear (syscall 332). A typical usage pattern for a JIT runtime is:

/* After emitting a new code page and before making it executable: */
if (syscall(332) != 0) {
    perror("speculation_force_clear failed");
}

The syscall forces the kernel to issue a VERW‑style micro‑code instruction that clears the BTB for the calling thread. The overhead is on the order of 1–2 µs on modern CPUs, which is negligible compared to the cost of JIT compilation (often > 10 µs).

6.2 Page‑Protection Transitions

The classic W^X (Write‑xor‑Execute) policy already forces a mprotect transition from writable to executable. On many kernels, this transition triggers a TLB shootdown but does not flush the BTB. Adding a mprotect(..., PROTEXEC) followed by a speculationforce_clear ensures both architectural and microarchitectural state are consistent.

6.3 Retpoline as a Fallback

Even though retpoline was designed for out‑of‑place Spectre‑v2, it still removes the indirect‑branch BTB entry from the execution path, replacing it with a return‑stack‑based speculation that is confined to the RSB. Modern runtimes can compile indirect calls through a retpoline thunk when the target is a JIT‑generated function. The trade‑off is a 5–10 % performance hit on indirect call‑heavy workloads.

7. Runtime‑Level Countermeasures

Runtime mitigations are the most flexible and can be rolled out without kernel or microcode updates. Below are concrete techniques that JIT developers can adopt today.

7.1 Serialize After Code Emission

Insert a serializing instruction that forces the CPU to complete all prior instructions before proceeding. On x86‑64, lfence (load fence) is sufficient for most speculative‑execution attacks because it stops speculation past the fence.

; Pseudo‑assembly emitted by the JIT after writing a new function
mov rax, <addr_of_new_code>
jmp rax          ; indirect call to the freshly compiled function
lfence           ; fence before any subsequent indirect call

Pros: Minimal code‑size impact (1 byte for lfence).

Cons: Slight latency increase (≈ 30 ns) on the critical path; must be placed after the code becomes executable, not before.

7.2 Explicit BTB Invalidation Intrinsics

Some architectures expose intrinsics to clear a specific BTB entry. On ARMv8.5‑A there is ic ivau (invalidate instruction cache by address) combined with dsb ish to ensure ordering. While not a direct BTB flush, the instruction‑cache invalidation forces the CPU to re‑fetch the target, which indirectly clears stale predictor state.

On x86, developers can use archprctl(ARCHCLEAR_BTB, addr) on Linux kernels that implement the syscall (still experimental).

void invalidate_btb(void *addr) {
#ifdef SYS_arch_clear_btb
    syscall(SYS_arch_clear_btb, addr);
#else
    syscall(332); /* speculation_force_clear */
#endif
}

Pros: Targeted invalidation reduces the need for a full BTB flush.

Cons: Availability is limited; fallback to full flush may be required.

7.3 Code‑Cache Quarantine

Instead of emitting code directly into the executable heap, allocate a temporary, non‑executable buffer. After the JIT finishes emitting, verify that the buffer does not overlap any existing BTB entries (by checking a per‑process BTB fingerprint, see § 7.5). Only then promote the buffer to executable memory using mprotect.

Workflow:

1. Allocate tmpbuf = mmap(NULL, size, PROTREAD|PROTWRITE, MAPPRIVATE|MAP_ANONYMOUS, -1, 0);
2. Emit machine code into tmp_buf.
3. Call speculationforceclear() to clear any stale BTB entries.
4. mprotect(tmpbuf, size, PROTREAD|PROT_EXEC);
5. Publish the function pointer to the rest of the runtime.

Pros: Guarantees that no stale BTB entry can point into the new code.

Cons: Extra memory pressure (duplicate code pages) and an additional mprotect call; may increase latency for hot JIT loops.

7.4 Thunk‑Based Indirect Calls

Force every indirect call to go through a trusted thunk that performs a return‑stack based speculation (retpoline) and then jumps to the real target. The thunk can be statically linked into the runtime, never regenerated, and therefore never subject to BTB poisoning.

void *indirect_thunk(void *target) {
    __asm__ volatile (
        "call 1f\n"
        "1: pop %%rax\n"
        "jmp *%0\n"
        : : "r"(target) : "rax");
}

Pros: Eliminates BTB usage for indirect calls entirely.

Cons: Adds an extra call/return pair; may degrade performance on tight loops (observed ~8 % slowdown in V8 micro‑benchmarks).

7.5 BTB Fingerprinting & Monitoring

Some research prototypes (e.g., Intel’s BTFM – Branch‑Target Fingerprint Monitor) expose a per‑process hash of the BTB state. A runtime can periodically read this hash and compare it against a baseline taken after a clean compilation. A mismatch triggers a full BTB flush and possibly a runtime restart.

While not yet mainstream, the technique can be simulated by:

1. Training the BTB with a known set of indirect calls (e.g., a “calibration” function).
2. Measuring the latency of a subsequent indirect call; a significant drop indicates that the BTB entry is still present.

Pros: Detects silent BTB poisoning without explicit flushes.

Cons: Heuristic; may generate false positives under heavy load.

8. Trade‑offs: Security vs. Performance

Guideline: For high‑throughput services (e.g., V8 in Chrome), a combination of lfence + periodic full BTB flush offers a good balance: the fence costs virtually nothing, and flushing once per JIT compilation (often every few milliseconds) adds only microseconds of overhead. For security‑critical environments (e.g., kernel eBPF, WebAssembly sandbox in a cloud function), code‑cache quarantine plus retpoline may be justified despite the higher latency.

9. Practical Integration Checklist

Below is a step‑by‑step checklist that JIT runtime teams can adopt today. It is organized by phase of the JIT lifecycle.

• ✔️Allocation Phase
• ✔️Use W^X: allocate pages with PROTREAD|PROTWRITE only.
• ✔️Prefer separate allocation pools for executable code vs. data to reduce accidental reuse.
• ✔️Emission Phase
• ✔️Emit machine code without any indirect branches that target unknown addresses.
• ✔️Insert a serializing fence (lfence on x86, dmb ish on ARM) after the last instruction before the page is made executable.
• ✔️Promotion Phase
• ✔️Call speculationforceclear() immediately before mprotect(..., PROT_EXEC).
• ✔️If the platform provides a targeted BTB clear syscall, invoke it with the address range of the newly promoted page.
• ✔️Publication Phase
• ✔️Publish the function pointer only after the fence and BTB flush have completed.
• ✔️Optionally, route the call through a trusted thunk if the runtime already uses retpoline for other indirect calls.
• ✔️Deallocation Phase
• ✔️Before freeing an executable page, invalidate the BTB entry for any indirect branch that previously pointed to it (if supported).
• ✔️Zero‑out the page (or overwrite with a known pattern) to avoid use‑after‑free speculation.
• ✔️Monitoring & Auditing
• ✔️Enable kernel logs for speculationforceclear failures.
• ✔️Periodically run a self‑test that triggers an indirect call to a known stub and measures latency; spikes may indicate stale BTB entries.
• ✔️Integrate a CI test that runs the BTR PoC against the built runtime on supported hardware.

10. Testing BTR Defenses: A Mini‑Lab

To verify that a mitigation works, developers can reproduce a simplified BTR scenario on a test machine.

10.1 PoC Overview

1. Compile a trivial function (void victim(void) { asm volatile(""); }) via the JIT.
2. Store its address in a global function pointer void (*fp)(void).
3. Free the code page (e.g., munmap).
4. Allocate a new page at the same virtual address (use mmap with MAP_FIXED).
5. Write malicious code that performs a cache‑based side‑channel (e.g., mov rax, [secret]; mov [tmp + rax*64], 0).
6. Invoke fp(); the CPU will speculatively execute the malicious code if the BTB entry persisted.

10.2 Validation Steps

Running this lab on a CI pipeline for each supported CPU architecture provides confidence that the runtime’s mitigation stack is effective.

11. Lessons from Spacecraft Software Engineering

The Curiosity rover demonstrates a set of practices that map cleanly onto JIT security:

1. Immutable Firmware Images – Once uploaded, code is never modified. For JIT, treat compiled code pages as immutable for the duration of their lifetime; never patch in‑place.
2. Checksum‑Based Integrity – Every firmware image is signed and verified on boot. JIT runtimes can compute a cryptographic hash of each compiled page and store it in a metadata table; any mismatch triggers a flush and recompilation.
3. Watchdog Timers – Detect hangs or unexpected behavior. A JIT runtime can embed a speculation watchdog that aborts a compilation if the BTB flush takes longer than a threshold, indicating a possible hardware bug.
4. Redundant Validation – Critical flight software runs dual redundant processors that cross‑check each other. Server‑side runtimes can run dual JIT pipelines (e.g., a fast‑path JIT and a slower, verified interpreter) and compare outputs for divergence.

By mirroring these practices, long‑running services can achieve a similar level of resilience: detect, isolate, and recover from speculative‑execution anomalies before they cause data leakage.

12. Future Directions

Developers should track these roadmaps and plan migration paths. In the meantime, defense‑in‑depth remains the safest posture.

13. Conclusion

The emergence of in‑place Spectre attacks—specifically Branch Target Reuse—forces a re‑evaluation of how we protect dynamic code‑generation pipelines. Unlike classic out‑of‑place Spectre‑v2 attacks, BTR exploits the persistence of BTB entries across self‑modifying code within the same process, rendering many hardware‑only mitigations ineffective.

The Curiosity rover’s 5 000‑sol record shows that static, well‑audited code combined with periodic integrity checks can survive for years without speculative‑execution compromise. JIT‑heavy systems can achieve comparable longevity by explicitly managing BTB state, serializing after code emission, and adopting quarantine‑style code promotion.

The practical steps outlined—syscall‑based BTB flushes, fence insertion, thunk‑based indirect calls, and code‑cache quarantine—are all available today on mainstream platforms. Their performance impact is modest (microseconds per compilation) compared to the risk of silent data exfiltration.

In short, speculative‑execution defenses must become a continuous runtime concern, not a one‑off configuration. By treating JIT‑generated code with the same rigor as spacecraft firmware—regular integrity verification, proactive invalidation, and layered redundancy—developers can build services that remain secure for thousands of operational cycles, just like the rover that roams the Red Planet.

Key Takeaways

• ✔️Flush or fence the BTB after every JIT compilation; a single speculationforceclear() call adds ≤ 2 µs overhead.
• ✔️Serialize with lfence (or equivalent) to stop speculation on newly executable pages.
• ✔️Consider retpoline or thunk‑based indirect calls when performance budgets allow; they eliminate BTB reliance.
• ✔️Adopt a code‑cache quarantine model: emit to a non‑executable buffer, verify, then promote with a BTB flush.
• ✔️Mirror spacecraft practices: immutable code pages, checksum validation, watchdog timers, and redundant verification.
• ✔️Plan for upcoming hardware and kernel features (BTI‑SM, automatic BTB flush on PROT_EXEC) to future‑proof your runtime.

#### Further Reading

• ✔️Spectre Mitigations in Modern CPUs: What Engineers Need to Know – a deep dive into IBRS, STIBP, and microcode updates.
• ✔️Secure JIT Compilation: Best Practices for 2027 – guidelines for building resilient JIT pipelines.
• ✔️From Spacecraft to Serverless: Transferring Reliability Strategies – how aerospace verification methods can improve cloud services.

Stay ahead of speculative‑execution threats: treat every JIT compilation as a potential attack surface, and defend it with the same rigor that keeps a rover alive on Mars.

See more articles on The Looplet

Further reading

Read Next

• ✔️Best Way to Secure Systems Against the New RSA Break
• ✔️How to Fix Recent WordPress Plugin Vulnerabilities Across JetFormBuilder, Better Messages, OpenStation, Customer Reviews, and Bookly
• ✔️How to Fix Legal and Safety Risks in Community Mods

Read next: continue with one of these related guides.