Research exposes 37-year-old OpenBSD kernel bug that turned into root access

Offensive security firm Calif published, on October 16, 2026, the details of a vulnerability that, according to the report, dates back roughly 37 years: a shared memory attachment counter defined as a signed 16-bit short in the original BSD code. Researcher Andrew Griffiths describes the full chain in the text Big Fish: Exploiting a 37-Year-Old Kernel Bug in OpenBSD.

The flaw was reported to OpenBSD earlier this year, along with suggested patches, and has already been fixed via two of the project's official errata, 026 and 018. The case matters to developers because it shows how a data type decision made decades ago can survive successive security audits without being noticed.

A counter that shouldn't overflow

The problem lives in the SysV-style shared memory subsystem. Every time a process attaches a segment, an attachment counter is incremented. This counter, inherited from the original BSD sources, is a signed 16-bit integer, with a maximum limit of 32,767.

Repeatedly calling fork() on a process with shared memory attachment records can push this counter past its limit, making it wrap (wrap) to a negative value. This integer overflow, by itself, already opens the way for a use-after-free via a race condition. Combined with a second flaw, an out-of-bounds write (out-of-bounds write) in the kernel, the result is local privilege escalation to root.

The exploitation chain, step by step

The most striking part of Calif's work isn't just finding the bug, it's managing to exploit it end-to-end on a system with KARL, SMAP, and SMEP enabled, mitigations built into OpenBSD itself, without relying on any information-leak vulnerability to discover memory addresses. In practice, the described chain follows these steps:

1. Map 127 SysV shared memory segments, unmap them with munmap(), and fork() 259 processes to push the attachment counter to -32,643.
2. Use sysctl(KERN_MALLOCSTATS) (fields such as kb_total and kb_totalfree) to monitor the heap and detect when new pages are allocated, enabling precise "heap grooming".
3. Win the race condition by using IPC_RMID on one process while a helper process attaches a new shared memory segment with a confusing size.
4. Align IP socket allocations and set IP_MULTICAST_TTL to a non-default value (2) to persistently allocate a 16-byte ip_moptions struct, whose last word packs two 16-bit counters.
5. Trigger the conditional (-1) out-of-bounds write to overwrite both the membership count and the maximum allowed.
6. Close the socket (close()) to walk the group's members, triggering atomic decrements that alter the self-referential PTE mapping twice, switching it from user access to supervisor access, thereby achieving kernel-mode code execution by overwriting a predictable kernel pointer.
7. Clean up the shared memory accounting and restore the overwritten kernel pointer to obtain a root shell.

Griffiths summarizes the finding this way:

That's the short of the matter, the longer version is more interesting, including the history of the vulnerability, as a short shm_nattch can be traced back ~37 years ago, in the original BSD sources.

Andrew Griffiths, researcher at Calif

In terms of exploitation primitives, the text classifies the result as a classic "write-what-where": the write came from a TAILQ_REMOVE, the "what" was a user memory mapping promoted to supervisor mode (bypassing SMAP and SMEP), and the "where" was a kernel pointer at a predictable location.

Why the pointer was predictable

OpenBSD's KARL relinks the kernel binary on every boot, randomizing function and data addresses. The problem is that the kernel's memory allocators remain deterministic: both the boot allocator and the best-fit allocator lack base address randomization.

This means allocations made early in the boot process always land in the same place on the same hardware, boot after boot. When these predictable structures hold function pointers, they become ideal targets. The exploit specifically targets the ioctl handlers of the lo0 and enc0 interfaces: one is overwritten to divert the execution flow, the other is used to restore the original pointer afterward.

What OpenBSD (and any long-lived kernel) can do beyond this

Beyond the two errata already published, Calif suggests a set of mitigations that go beyond the one-off patch:

• Safe-unlinking: check doubly linked list invariants before performing the unlink write; if violated, call abort() or panic() instead of corrupting memory. FreeBSD and NetBSD already have this technique implemented in sys/queue.h.
• Randomizing the self-referential PTE mapping: randomly pick a PML4 slot (320 + (random_byte & 127)) at boot, turning PTE_BASE into a runtime-computed expression instead of a fixed address.
• Non-deterministic kernel allocator: randomize the base address of allocations at boot (at virtually no cost) and, additionally, the allocation order at runtime (with some overhead).
• Compiler sanitizers in production: Calif has already suggested FreeBSD enable the array bounds sanitizer (part of UBSan) in release builds, after reporting 21 issues it would have caught, some with local privilege escalation impact.
• Alloc Tokens: a compiler feature still under development to isolate allocations that contain pointers from those that don't, making it harder for an overflow in a pointer-free allocation to turn into flow control.

What this changes for maintainers of long-lived systems

OpenBSD is, historically, one of the most audited codebases in existence, with decades of proactive review, replacement of unsafe APIs, and systematic hardening. Even so, a type decision made nearly 40 years ago survived intact until it was found in 2026.

This is a direct reminder for anyone working with code that spans generations: a signed short counter, seemingly harmless in isolation, becomes a memory corruption primitive when combined with a usage path nobody anticipated when the type was chosen. Manual audits of old code tend to review what changed recently, not old assumptions that were never questioned.

Calif closes the piece by pointing to what comes next: AI-assisted vulnerability discovery is expected to speed up the pace at which bugs like this are found, which makes whole-class mitigations (such as sanitizers in production builds and allocator randomization) more urgent than relying solely on reactive per-CVE patching. For teams maintaining kernels, drivers, or any long-lived C code, the case is a concrete argument for running UBSan and equivalents in production too, not just in development CI.