Open source project gives KVM virtual machines near-native Nvidia GPU access

virtio-nvgpu is an experimental virtio device, published by nestrilabs on GitHub, that gives a KVM virtual machine access to the Nvidia driver at almost the same level the host has. Instead of translating graphics API calls (the path taken by projects like Venus), it forwards the Nvidia driver's ioctls straight to /dev/nvidia*, at the kernel ABI layer. The guest runs Nvidia's own user-mode libraries unmodified: same Vulkan, same NVENC, same card.

The use case the project targets is headless streaming: a Wayland compositor inside the VM renders, composites, and encodes frames on the GPU itself, and only the compressed video comes out. The VM has no monitor, and the physical GPU remains the host's.

Why Not Translate the API

The most common approach today for GPU in a VM without dedicated passthrough is virtio-gpu with Venus, which serializes every Vulkan or OpenGL call in the guest, transports it over virtio, and replays it on the host. The project's README lists three practical problems with this: games issue 1,000 to 5,000 draw calls per frame, each serialized and replayed individually, which eats up 6% to 18% of the 16.6 ms budget of a frame at 60 fps just in serialization overhead; the host CPU spends cycles on serialization and replay that could otherwise go to the application; and since GPU buffers belong to the host, the guest compositor can't even see them, so there's no practical path to encode in NVENC without a CPU read round-trip.

virtio-nvgpu attacks this at the root: since the guest runs Nvidia's real user-mode driver, it assembles GPU commands locally itself. Per frame, the project reports 5 to 20 messages crossing the VM boundary, versus roughly 2,000 in the Venus model. VFIO passthrough solves the same problem with native performance, but dedicates the entire card to a single VM, which doesn't work for multi-tenant environments.

The Measured Numbers

The benchmark, described in detail in the BENCHMARKS.md file of the repository, ran on an RTX 3060 with driver 595.99.02, comparing a guest against the same host running bare metal, with an identical headless Vulkan workload on both sides. For frames the bare-metal host takes 39 ms, 9.9 ms, and 2.0 ms to render, the guest came in between 0.4% faster and 1.7% slower, within measurement noise. It's only below 2 ms per frame, a range lighter than any real game draws, that the gap grows: 7.1% more on 0.5 ms frames and 40.8% on 0.05 ms frames. The project's own explanation is that, at that point, what dominates is no longer forwarding but waiting: the guest sleeps waiting for the GPU, and the cost of waking up (about 0.02 ms) starts to weigh on a frame that lasts less than that.

On the CPU side, a guest running uncapped at ~100 fps for 12 seconds consumed 0.37 s of CPU against 0.40 s for the bare-metal host on the same workload. The reason is structural: since the Nvidia user-mode driver submits commands by writing to already-mapped memory, there's nothing for the backend to forward during the rendering loop. Across 813,691 frames, the backend only had to exchange 13,792 messages with the guest, one crossing every 59 frames, almost all of it device setup, not frame submission.

Four VMs Sharing a Single Card

The test most relevant to anyone thinking about shared infrastructure ran four simultaneous guests on the same RTX 3060, each with the same workload: 25.84, 26.49, 25.57, and 25.79 fps, adding up to 103.7 fps versus 102.9 fps for a single guest alone. Median (p50) frame times came in at 39.165, 39.164, 39.168, and 39.165 ms, practically identical. All four guests rendered correctly at the same time and encoded H.264 simultaneously, each at exactly 60 Hz, without hitting any NVENC session limit. The project is clear that four is what it ran, not a limit it found: eight guests hasn't been tested yet.

What This Doesn't Isolate

The project's README is direct about the question any security-minded person would ask first, and the answer isn't 'nothing': there's no IOMMU boundary between the guest's GPU work and the host. The card belongs to the host's Nvidia driver and lives in the host's IOMMU domain; the guest gets the driver's ioctl interface, not the device itself. What separates guest memory from host memory is the GPU's own MMU, with page tables programmed by the RM (Nvidia's Resource Manager) on the guest's behalf, which puts the host's Nvidia driver inside the TCB (trusted computing base). The guest also assembles its own command streams, and that's exactly why there's no per-submission cost.

What reduces that surface today is ABI filtering: ioctls that the ABI profile doesn't describe are rejected, not forwarded (the --permissive-abi flag turns off that check for diagnostics, with an explicit warning). What doesn't exist yet: RM_ALLOC classes and RM control commands aren't filtered, UVM and modeset ioctls have no equivalent table, and the backend today holds the host's descriptors inside the VMM's own process, because isolate, a sandboxed per-guest process planned to hold those descriptors without privilege, is designed but not built (the isolate/ directory in the repository is a design note, not code). The project itself sums it up: this is attack-surface reduction, not hardware isolation. VFIO with IOMMU remains strictly stronger, and for mutually untrusted tenants, VFIO or vGPU remain the answer.

What Already Works and What's Still Missing

What's been tested: nvidia-smi reports real power and memory usage inside the guest, with the host's deviceUUID; vulkaninfo exits with code 0 and offscreen renders are pixel-perfect; a Wayland client presents through a compositor in the guest; NVENC works via Vulkan Video, encoding on the client's own device; imported buffers are host memory, mapped through a shared window.

The project ships three ABI profiles, each covering a range of driver versions: 535.129.03, 580.178.04, and 595.71.05 onward. Versions older than the first range are rejected, not guessed at, because forwarding an ioctl with a layout it's never seen is a way to get a plausible, wrong answer instead of an error. Real testing ran on two cards: the RTX 3060 with driver 595.99.02, which is where all the numbers in BENCHMARKS.md come from, and an RTX A2000 with driver 615.71.09, which renders and enumerates but hasn't been benchmarked or retested since. CUDA is forwarded but only tested past enumeration; the jailer, driver sharing by version, and the full multi-tenant envelope don't exist yet.

Why This Matters for Local Infrastructure in Brazil

The most obvious use case is CI/CD and GPU development environments: instead of reserving an entire card via VFIO for each runner or dev box, and being limited to the number of physical GPUs, it's possible to share a single card across several isolated VMs, which changes the hardware math for anyone building this kind of infrastructure locally, without depending on a cloud instance with a dedicated GPU. The test case with four guests sharing an RTX 3060 without any loss in total throughput is exactly that scenario: builds or tests that need graphics acceleration or NVENC can run in parallel, isolated from each other at the process/VM level, without multiplying the number of cards.

The security caveat, though, is worth taking seriously before putting this in a pipeline that runs third-party code or external PRs: since the host's Nvidia driver sits inside the TCB and isolate hasn't been built yet, this is today a project better suited to trusted internal environments, like a team's dev boxes or streaming test infrastructure, than to multi-tenant runners exposed to untrusted code. For that latter case, the project itself recommends sticking with VFIO or vGPU. The code sits under three different licenses by layer (GPL-2.0 in the guest kernel driver, Apache-2.0 in the device crate, dual BSD-3-Clause/GPL-2.0 in the shared protocol), which makes it easier to adopt the device layer in another VMM without inheriting GPL, something the README itself calls out as a deliberate decision inspired by the layout of chromeos/virtio-media. The direct architectural reference is gVisor's nvproxy, which already forwards Nvidia ioctls from sandboxed containers to the host driver in production, and it's from there that the ABI tables and translation logic that virtio-nvgpu adapts for the KVM VM context come.