JIT in 5 microseconds: how pgrust compiles every SQL query in real time

A post published on the malisper.me blog and discussed on Hacker News details how the author built a JIT compiler that generates code in around 5 microseconds, fast enough to compile every SQL query, not just a subset of them. The technique is at the heart of pgrust, a database he has been writing in Rust, and the piece reconstructs the method step by step using a regular expression engine as a teaching example.

Why compiling in microseconds matters

JIT (Just In Time) is the practice of generating machine code at runtime, taking advantage of information that only exists at runtime. According to the author, done correctly, this yields gains of 2x to 5x, sometimes more. The classic use case is language interpreters, which only receive the code to execute at runtime, but the same applies to parsing data whose schema is unknown beforehand, exactly the situation of a database processing arbitrary SQL.

The historical problem is the cost of compiling. The author states that no production-ready database today has its own JIT compiler: all of them use LLVM or generate C/C++ code, and both routes suffer from high compilation times. That's why systems like PostgreSQL only trigger JIT (via LLVM) for expensive queries, when the cost of compiling pays off. If each compilation takes microseconds instead of milliseconds, that math changes: it becomes possible to compile everything.

Copy-and-patch: stencils instead of LLVM

The chosen approach is a variant of copy-and-patch. The idea is to keep assembly templates for each operation you want to compile, called stencils. To compile an operation, you take the corresponding stencil and make small adjustments according to the details, much like filling in a real stencil. By chaining together several filled-in stencils, you assemble at runtime a program with performance close to that of hand-written code.

In the post's example, a minimalist regex engine (only literals and repetition) is represented by a Rust AST with three nodes: Literal, Concatenation, and Repetition. The naive interpreter for this AST is under 20 lines long, but the benchmark shows it is 10x to 20x slower than a version hand-written specifically for the regex b(an). It's this gap that the JIT needs to close.

Code generation happens on ARM64 on macOS. Each stencil is a Rust function that returns an array of 32-bit instructions, with the variable points filled in via bit manipulation. The character comparison stencil, for example, inserts the byte to compare and the fallback jump offset directly into the opcode:

fn stencil_char(byte: u8, stencil_pos: usize, fail_pos: usize) -> [u32; 4] {
    [
        0x39400009,                               // ldrb w9, [x0]
        0x7100013F | ((byte as u32) << 10),       // cmp w9, #byte
        0x54000001 | cond_branch_offset(stencil_pos + 2, fail_pos), // b.ne fail
        0x91000400,                               // add x0, x0, #1
    ]
}

A few design decisions keep the generated code lean: the string ends in a null byte, so character comparisons automatically fail at the end of the input, avoiding length checks; and backtracking uses a stack that stores the resume address and position in the string. Registers are fixed (x0 for position and return, x1 and x2 for stack top and base, x9 as a scratch register).

From buffer to executable function

To turn the generated bytes into something callable, the author uses mmap to allocate memory with read, write, and execute permissions, marked with MAP_JIT. On macOS, with Apple's W^X model, you need to toggle protection with pthread_jit_write_protect_np before and after copying the code, and invalidate the instruction cache with sys_icache_invalidate. Once that's done, the buffer is converted into a function pointer via std::mem::transmute and called like any Rust code.

The numbers

The benchmark compared the interpreter, JIT, and hand-written version across different input sizes:

| Size | Interpreter | JIT | Handwritten | JIT Speedup | | --- | --- | --- | --- | --- | | 9 | 45 ns | 3.8 ns | 3.8 ns | 11.7x | | 33 | 103 ns | 7.9 ns | 10.5 ns | 13.0x | | 129 | 597 ns | 30 ns | 32 ns | 19.7x | | 513 | 1,955 ns | 126 ns | 120 ns | 15.5x | | 2,049 | 8,301 ns | 470 ns | 393 ns | 17.7x |

The core result: the JIT code is on par with the hand-written version, sometimes a bit faster, sometimes a bit slower, and both outperform the interpreter by more than an order of magnitude. Since the cost of compiling is around 5μs, it dilutes quickly across any repeated execution.

The AI thesis

The angle that fueled the Hacker News discussion is the author's claim that AI was decisive. He says he had never written real assembly before, having only gone through the microcorruption CTF, and that he would have had real trouble getting the instructions and bit adjustments right without help. With a coding agent, he says, it was enough to hand over the compiler's general shape for the low-level details to be worked out.

From there he challenges the meme that "AI doesn't help because writing code was never the hard part." For JIT compilers, he argues, writing the code was the hard part, and the rarity of this kind of software in practice suggests that building it was historically too costly to be worth it. The conclusion is the project's own thesis: if databases have always been among the hardest kinds of software to build, LLMs open room for more ambitious projects in this territory.

What remains open

The post's example is a toy: regex with no alternation, no lookbehind, and with the AST already parsed. It demonstrates the mechanics, but not the complexity of compiling real SQL, with joins, aggregations, and varied types. The code is also specific to ARM64 on macOS, leaving out x86-64 and Linux, which matters for anyone running a database on a server. And, being a copy-and-patch approach hand-written with AI assistance, maintaining and correcting the assembly-generated stencils remains the responsibility of whoever reviews them, a point the author himself leaves implicit when describing AI as a tool to "handle the details."

For those building runtimes, data engines, or interpreters in Brazil, the practical takeaway is that the barrier to writing your own JIT, instead of relying on LLVM, has gotten lower, and that compiling in microseconds enables applying real-time optimization broadly. The full code and the project are on pgrust's GitHub.