NeoMME: open-source multimodal encoder indexes pages as images and fits in 6 KB

H Company published, on the Hugging Face blog, NeoMME, a family of multimodal and multilingual encoders in two sizes: 260M and 800M parameters. The interesting angle for those building RAG isn't just the compact size, but the architecture decision behind it: instead of bolting a pre-trained vision encoder onto a causal language model, NeoMME uses a single bidirectional Transformer that processes text tokens and image patches in the same computational path. Everything was trained from scratch, with weights released under Apache 2.0 and implementation in transformers available since launch day.

For the Brazilian developer who wants multimodal AI in production without depending on a pay-per-token API, this is relevant for a practical reason: visual document retrieval (the expensive step of indexing PDFs) can now run locally, on a model small enough to fit on a modest GPU.

What changes in the architecture

Most recent visual retrievers (ColPali, ColQwen, ColSmol) originate from adapting generative VLMs. That means loading three things: a pre-trained vision tower (like SigLIP2), a projector that maps visual features into the language space, and a causal decoder. The problem, pointed out by the authors, is that retrieval, classification, and labeling tasks don't generate text autoregressively, so all the parameter and compute overhead of the causal decoder is dead weight.

NeoMME cuts that out. Images become a grid of 32×32 patches projected by a small MLP; text uses factorized embeddings; both enter the same bidirectional encoder. A few technical details worth noting:

• Dynamic resolution: the image keeps its aspect ratio and size, spending more tokens on a dense page and fewer on a simple image.
• 16,384-token context, enough for up to two 4K UHD images, with sliding-window attention in most layers and global attention every sixth layer.
• Modern stack: grouped-query attention, query-key normalization, gated attention, 2D rotary embeddings, and squared-ReLU MLPs.
• A 131k-token BPE tokenizer trained from scratch on multilingual text, code, math, and image transcriptions.

Pretraining is the least conventional part: instead of classic masked language modeling, it uses a masked discrete diffusion objective. For multimodal examples, the text corruption rate ranges between 0.3 and 1, with image patches always visible. The idea is to force the model to describe from the image: with little masking, it can guess "cat" in "The [MASK] sat on the mat" from textual context alone; with heavy masking, there's no way out but to look at the image. Each model processed about 524 billion tokens, a small budget compared to ModernBERT's 2 trillion, which motivated the use of the NorMuon optimizer to gain data efficiency.

NeoMME-Retriever and the two heads

The version useful for everyday work is NeoMME-Retriever, fine-tuned for document retrieval using ColPali's page-as-image approach. Instead of extracting text from a PDF with OCR and splitting it into chunks, it ranks screenshots of the pages. This preserves layout, tables, charts, font type, and size, things even a perfect OCR wouldn't capture.

The backbone gains two heads trained jointly, and a single forward pass returns both representations:

• Dense head: performs mean pooling of the hidden states into a normalized vector. Compact and ideal for approximate search (ANN).
• Late-interaction head: projects each token/patch into a 128-dimensional vector. Preserves local matches between query tokens and image regions.

The team's recommendation is to use late-interaction by default (more powerful, compatible with libraries like NextPlaid). For very large corpora, it's possible to combine both: a single forward pass generates the dense embedding, an ANN index retrieves a few candidates, and late-interaction reranks them. The term "late-interaction," it's worth noting, is what Omar Khattab (author of ColBERT) advocates as more precise than "multi-vector," because it describes the granularity of the scoring function, not just the vector count.

The ViDoRe numbers

Results on the ViDoRe benchmark (nDCG) place both models on the Pareto frontier between quality and size:

| Model | Params | v3 (@10) | v2 (@5) | v1 (@5) | |---|---|---|---|---| | ColModernVBERT | 250M | 0.261 | 0.407 | 0.806 | | ColSmol-256M | 256M | 0.207 | 0.348 | 0.797 | | NeoMME-260M | 260M | 0.523 | 0.522 | 0.860 | | ColSmol-500M | 500M | 0.340 | 0.455 | 0.825 | | Vultron Flash | 850M | 0.565 | 0.604 | 0.882 | | NeoMME-800M | 800M | 0.556 | 0.559 | 0.874 | | ColQwen2.5-v0.2 | 3.75B | 0.524 | 0.601 | 0.895 | | ColPali v1.3 | 2.92B | 0.430 | 0.547 | 0.848 |

The honest highlight: NeoMME-260M reaches 0.523 on ViDoRe v3, landing 0.002 behind ColQwen2.5, which uses about 14x more parameters. The 800M model is 0.009 behind Vultron Flash, of similar size. It's not the absolute top of the benchmark (the 850M Vultron Flash still wins), but the quality-to-size ratio is the central argument.

Where production cost really drops

Two points matter for whoever pays the GPU bill:

Throughput. On the same 2048×2048 input on an NVIDIA L40S, NeoMME-Retriever-260M encodes about 51 pages per second, nearly double ColModernVBERT's 26. Indexing faster means less GPU time spent building and updating the index.

Late-interaction index storage. This is the historic Achilles' heel of the multi-vector approach: storage grows linearly with the number of vectors, and a 2048×2048 page generates ~4,200 vectors, or about 2.1 MB in float32 (averaging ~1.5 MB per document on ViDoRe v3). The team combines two techniques: hierarchical token pooling (groups similar vectors and keeps the average) and asymmetric quantization (documents in int8 or binary; queries at higher precision, since they're generated on the fly). The result:

• Pooling 10 + int8 on both sides: from ~1.5 MB to 39 kB per page (39x smaller), keeping more than 99% of nDCG@10.
• Pooling 8 + int8 on the query + binary on the document: 6 kB per page (255x smaller), keeping more than 95% of the quality.

That's the difference between a visual RAG index that blows out the disk and one that runs on lean infrastructure.

How to use it

The model is already in transformers. A single forward pass returns dense and late-interaction together:

from transformers import NeoMMEForRetrieval, NeoMMEProcessor
from sentence_transformers.util import cos_sim, mean_maxsim
import torch

model_name = "Hcompany/NeoMME-260M-Retriever"
processor = NeoMMEProcessor.from_pretrained(model_name)
model = NeoMMEForRetrieval.from_pretrained(model_name, device_map="auto")

# multilingual queries run on the same model
queries = [
    "Quelle partie de la production petroliere provient de champs en mer ?",
    "Which hour of the day had the highest overall electricity generation in 2019?",
]

For fine-tuning, there are separate dense and late-interaction checkpoints compatible with Sentence Transformers v6, loaded via NeoMMEModel. Since Sentence Transformers supports one head per model, each checkpoint trains one head; to train both together, you need NeoMMEForRetrieval with a custom Trainer.

When it's not worth it

A few cautionary points before deploying to production. NeoMME-Retriever specializes in full-page visual retrieval: if your use case is plain text search over already-extracted chunks, a traditional text encoder is still simpler and cheaper. The 6 kB per page configuration is aggressive and loses ~5% of quality, so recall-sensitive workloads should calibrate the point on the compression frontier. It's also worth remembering that the team itself describes the project as "a side quest between two friends," made with limited time and compute, and that the throughput and storage numbers were measured by them on a specific L40S: reproducing it on your GPU is the test that settles the decision. Still, for the local visual RAG scenario, without per-token API costs and under the Apache 2.0 license, it's one of the most interesting open-source options to appear in the compact encoder space.