Read more

Read more

And HFT developers? We shrugged.

Not because PREEMPT_RT is bad. It’s excellent for what it does. But if you’re running a trading system, the kernel is not your bottleneck. Your data structures are.

Before I show you the benchmarks that prove this, you need to understand what PREEMPT_RT actually does and why it took two decades to get here.

The Twenty-Year Journey

Back in 2004, real-time Linux was a mess. Several groups were working on incompatible approaches to the same problem: how do you make Linux guarantee response times? The efforts were uncoordinated, the patches incomplete, the mailing list discussions legendary in their toxicity.

Then Ingo Molnar did something unexpected. The RedHat kernel scheduler maintainer took pieces from various patch sets, rewrote most of them from scratch, and combined them into something coherent. Thomas Gleixner, who had been working with Dr. Doug Niehaus from Kansas University on the KURT real-time extension, joined forces with Molnar. Steven Rostedt (of ftrace fame) became another major contributor. Within months, they had a “halfway usable Real-Time solution.”

The goal was never “as fast as possible.” As Gleixner puts it: “Real-Time is not as fast as possible; it is as fast as specified.” The difference matters. Real-time means guaranteed completion within defined timeframes. Miss that deadline in a car’s braking system or a medical device, and people die.

The companies who funded this work read like a who’s-who of embedded systems: ARM, BMW, Intel, National Instruments, Texas Instruments, RedHat. Former supporters included Google, IBM, and NXP. Gleixner has been candid about the funding model resembling Jenga: “companies build key products depending on these technologies...seem to take the availability...for granted.”

Why twenty years to merge? Not because the code didn’t work. Because the changes touch core kernel infrastructure. Spinlocks. Interrupt handling. Scheduler internals. Every patch had to be designed so it “does not get in the way of other interests.” Incremental improvements went in quickly. Foundational changes required years of polishing and community alignment.

Why Normal Kernels Have Unbounded Latency

To understand what PREEMPT_RT fixes, you need to understand why standard Linux has terrible worst-case latency.

The latency formula is deceptively simple:

Each term can explode. Here’s why.

Spinlocks disable preemption. When kernel code acquires a spinlock, it disables interrupts on that CPU. Any pending interrupt, including your trading thread’s timer, waits until the spinlock is released. The problem: critical section duration is unbounded. Some driver holding a spinlock for 500 microseconds delays everything behind it.

Interrupt handlers run with interrupts disabled. Linux splits interrupt handling into two parts. The top-half runs immediately when hardware triggers an interrupt, with that interrupt line masked so nothing else can fire on it. The bottom-half (softirq, tasklet, or workqueue) runs later to do the heavy lifting. The problem is the top-half: if it takes 200 microseconds, nothing else on that interrupt line runs for 200 microseconds. Priority doesn’t matter. Your SCHED_FIFO thread with priority 99 waits for a USB driver’s interrupt handler to finish its top-half.

The kernel is not preemptible by default. When your high-priority thread wakes up, it can’t run until the current kernel code path reaches a voluntary preemption point. System calls run to completion. A memcpy() of 10MB in kernel space? Your thread waits.

Priority inversion destroys guarantees. Thread A (high priority) needs a lock. Thread B (low priority) holds it. Thread C (medium priority) preempts B. Now A is blocked behind C, even though A has higher priority. Without priority inheritance, your carefully tuned priorities are meaningless.

The result: a standard kernel might respond in 5 microseconds under no load, then spike to 500+ microseconds when a USB device enumerates or a filesystem flushes. You cannot bound the worst case.

What PREEMPT_RT Actually Does

PREEMPT_RT attacks each source of unbounded latency.

Spinlocks become sleeping locks. In a PREEMPT_RT kernel, spinlock_t doesn’t spin at all. It’s implemented on top of rt_mutex, which means a task waiting for a “spinlock” goes to sleep instead of burning CPU cycles. When the lock is released, the waiter wakes up. Preemption is not disabled while holding these locks.

Interrupt handlers become kernel threads. Almost all interrupt handlers in PREEMPT_RT run in process context, not hardware interrupt context. They’re scheduled like any other thread, with priorities you control. A USB driver’s interrupt handler can’t preempt your trading thread if your thread has higher priority.

Read more

Read more

The fundamental problem with runtime profilers is that they require runtime. You need load, you need the right access patterns, and you need the problem to actually manifest in your profiling window. A false sharing issue that only appears under specific market conditions — say, when two feeds update simultaneously — might never show up in your Tuesday afternoon perf session. But it’s there, lurking in the struct layout, waiting for the MESI protocol to ruin your day during the opening auction.

perf, VTune, cachegrind — these tools answer the question “what is slow right now?” The question they can’t answer is “what structural patterns in my code will be slow when the conditions are right?”

Article continues on HFT University

I went through the talk claim by claim and pulled the source documents. Some of it survives. A fair bit doesn’t — including a flat-wrong “Nvidia ships its first CPU” line, a slightly massaged 79% CrowdStrike stat, and a “Twitter still works” hand-wave that an HFT audience should never let pass.

The good news for those of us shipping latency-sensitive C++: the macro thesis on power and performance-per-watt is real, and the ACM Queue paper showing 0.3% overhead for libc++ hardening at Google scale is the single most actionable number in the entire keynote.

The full fact-check, what to keep, what to discount, and four things HFT shops should do Monday morning:

Article is free for logged in users.

https://hftuniversity.com/post/let-s-be-real-about-herb-sutter-s-c-26-victory-lap

Read more

Read more

Rather than just critique, I ran the article’s code and benchmarked the claims. The results are illuminating.

Complete article in HFT University

This part is about understanding why. The investigation took me through Clang source code, LLVM optimization passes, and eventually to a series of C++ committee papers that reveal an intentional design decision. Contracts and assumes are deliberately kept separate.

Continue article on HFT University

The executive summary is this: contracts with ignore semantic have near-zero overhead. Sometimes they’re faster than no checks at all. But the real surprise came from comparing contracts against manual validation—that’s where the dramatic numbers live.

Article continues…

Or switch to Rust, where the standard library ships the right defaults and you don't need to know any of this. A junior Rust developer using HashMap, BTreeMap, and VecDeque — the first containers they'll find in any tutorial — gets 5,177 cycles P99 on a workload that punishes bad containers mercilessly. A junior C++ developer using the standard library gets 302,653.

That's not a skill gap. That's a standard library gap. And after 28 years of C++ standardization, the committee still hasn't closed it.

Article is free for logged in users (log in with Google)

For all other articles, consider subscribing: https://hftuniversity.com/pricing

I had to find out for myself. What follows is a journey through building a contracts-enabled compiler, running real benchmarks on HFT-style code, and discovering some uncomfortable truths about what contracts can and cannot do for your hot paths.

The State of Play (January 2026)

Here’s the thing nobody tells you upfront: C++26 contracts are not in mainline GCC or Clang. Not yet. If you want to actually test this stuff, you need to build an experimental compiler fork. The standards committee ratified P2900R14, but compiler vendors are still implementing it.

GCC has a tracking issue at PR119061. Clang has Eric Fiselier’s experimental branch. That’s what we’re working with.

This means everything I’m about to show you comes with the usual “experimental implementation” caveats. But the semantics are locked in, and the performance characteristics we measured should hold when this lands in production compilers.

What You Actually Get

Before we dive into building compilers, let me show you what contracts look like in real code. The syntax is clean:

Three things happen here. The pre clause declares a precondition—the caller’s responsibility. The post clause declares a postcondition where r binds to the return value. Inside the function you can use contract_assert for internal invariants.

The magic is in evaluation semantics. You compile with a flag that controls what happens:

Same source code, different behavior at compile time. In production you get zero overhead. In testing you catch violations. This is the promise.

Building Your Own Contracts Compiler

The canonical way to test native contracts is through Eric Fiselier’s fork—the same one powering Compiler Explorer’s contracts compilers. Here’s the full process, pain points included.

First, the dependencies. On Ubuntu/Debian:

Then clone the experimental branch. Use --depth 1 unless you want to wait an hour for git history:

The CMake configuration is straightforward but the choices matter:

Build time is 30-60 minutes on a modern machine. The LLVM_PARALLEL_LINK_JOBS=2 matters—without it, linking will eat all your RAM and the OOM killer will ruin your day.

The First Surprise: Missing Runtime Handler

After building the compiler, I tried to compile some benchmark code with enforce semantic. Linker error:

Turns out the compiler generates calls to a runtime handler when contracts need checking, but the runtime library doesn’t provide it. You need to write your own:

This is the kind of implementation detail you only discover by doing. The committee specified the behavior but left the ABI to implementations, and Fiselier’s fork expects you to provide the handler.

The HFT Problem Contracts Solve

Why do we even care about this? Let me show you a pattern that exists in every trading system I’ve worked on:

Three branches on the hot path. Each one pollutes the branch predictor. The checks are almost always redundant because callers are well-behaved. But removing them is unsafe—one bad call and you’re looking at a Knight Capital situation.

Contracts let you have it both ways:

With ignore semantic, those preconditions generate no code whatsoever. With enforce semantic during testing, every violation triggers your handler. Same source, different compilation targets.

What I Set Out to Measure

The benchmark suite I built covers 16 HFT-style scenarios. The core question for each one: what’s the overhead of contracts compared to manual if-checks and compared to no checks at all?

The scenarios include:

• Order book bounds checking

• Division safety (divide by zero protection)

• Queue invariant validation (ring buffers)

• Market data integrity checks

• Risk manager pre-trade validation

• Black-Scholes input/output validation

• SIMD alignment verification

• Lock-free algorithm invariants

• Protocol parsing validation

Each scenario has three variants:

1. No checks (baseline—undefined behavior if invariants violated)

2. Manual if-checks (traditional defensive programming)

3. With contracts (C++26 contracts with configurable semantic)

We also tested an additional variant using __builtin_assume() to see whether compiler hints provide optimization benefits—a question that became far more interesting than I initially expected.

Setting Up the Benchmark Infrastructure

The benchmark harness uses high-resolution timing with warmup runs and statistical aggregation. Nothing fancy, but it works:

The do_not_optimize barrier prevents the compiler from optimizing away the computation entirely. Every benchmark lies in some way, but these lies are consistent across all variants, which is what matters for comparison.

The test machine is a dedicated benchmark server (aorus) running the contracts-enabled Clang 22.0 with -O3 -march=native. All benchmarks run with CPU frequency scaling disabled.

Coming Up: The Results That Surprised Me

In Part 2, we’ll look at the actual benchmark numbers. Spoiler: contracts with ignore semantic show essentially zero overhead—sometimes they’re even faster than the no-checks baseline. But the story gets more interesting when we compare enforce semantic and start asking whether contracts provide optimization hints.

The data revealed something the committee papers don’t emphasize: the gap between what contracts could do for optimization and what they actually do in current implementations. That investigation led me deep into the Clang source code, and what I found there explains a lot about the C++ standards committee’s philosophy on contracts.

But that’s for Part 3. First, let’s see the numbers.

Next: Part 2 - The Benchmark Results

Read more

Read more

Why the change

Substack does not handle VAT for international subscribers. As our subscriber base has grown to 115+ countries, this creates a compliance gap we can no longer ignore. Our payment processors on hftuniversity.com (Paddle and LemonSqueezy) handle VAT automatically for every country — receipts, tax IDs, everything.

What changes for you

• Paid Substack subscribers: Your account is synced daily to hftuniversity.com. You already have full access there — log in with the same email. All articles, challenges, and badges are available. Nothing breaks.

• Paddle / LemonSqueezy subscribers: Your articles are on hftuniversity.com only. You actually get a better reading experience — full-width code blocks, syntax highlighting, and no Substack sidebar clutter.

• Free Substack subscribers: You’ll continue to receive free posts here. Premium content will link to hftuniversity.com.

What stays the same

• The newsletter email still comes from Substack (for now)

• Past articles remain available on hftuniversity.com primarily.

• Your subscription price doesn’t change

• C++ challenges, badges, and rankings are unaffected (they were always on hftuniversity.com)

What’s better on hftuniversity.com

• Full-width code blocks with proper syntax highlighting

• C++ challenge integration — your benchmark scores appear alongside articles

• Forum for discussion (instead of Substack’s limited comments)

• Verified badges on your profile

If you have any questions, reach out at support@vitorian.com.

— Henrique

Read more