NEWS

LinkedIn details how it shrank its ranking model from 7 billion to 400 million parameters

LinkedIn details how it shrank its ranking model from 7 billion to 400 million parameters
Image: Redação iMasters
News
Reading
4 min
Tables
1
In short

In a talk at QCon AI, a LinkedIn engineer explains how the company compresses models and context to run ranking LLMs under P99 latency SLAs without blowing up GPU cost.

Ranking search results with an LLM seems simple until you have to respond in milliseconds for 1.3 billion members. That was the problem Sundara Raman Ramachandran, lead engineer on LinkedIn's LLM Inference team, broke down in a talk at QCon AI, published by InfoQ. Ramachandran is also an open source contributor to SGLang, focused on the scoring API used specifically for ranking with LLMs.

The starting point is recognizing that ranking is not text generation. In a chatbot, the model goes through a prefill phase (understanding the context) and then a decode loop, generating token by token. In ranking, you build one prompt per candidate item, all sharing the same long prefix (query, user profile, system instruction), process everything in a batched prefill, and there is no decode: what matters is only the probability of the model answering "yes" or "no" for that item.

How LinkedIn's search pipeline gets to ranking

LinkedIn's search pipeline has three layers. The first, query understanding, turns ambiguous natural language ("find design jobs near me") into something structured, resolving references like "near me" using profile data. The second, retrieval, runs a vector search (KNN) against precomputed embeddings of millions of jobs to bring back the top-K candidates, optimizing for recall rather than precision. The third is ranking, where the LLM comes in.

The production prompt, according to Ramachandran, is literally straightforward: given the query, the member's profile and a job, the model answers whether there is a match or not. The probability of the "yes" token becomes the score used to rank the results.

Three optimization axes: model, context and inference

With strict P99 SLAs and hundreds of candidates to rank per request, the team attacked three fronts at the same time: shrinking the model, shrinking the context, and optimizing inference itself.

From a 7B to a 400M-parameter model

The process started with a 7-billion-parameter teacher model, good enough to understand product policies and relevance, but too heavy for production. Through multi-teacher distillation, that knowledge was transferred to two smaller models, each optimized for a different metric (relevance and engagement), and then combined into a single 0.6-billion-parameter student model.

Then came pruning: removing attention and MLP layers through trial and error, measuring the acceptable quality loss, followed by a new round of distillation and fine-tuning to recover whatever was needed. The final result was a 400-million-parameter model, ready to serve in production.

From full text to a single embedding

A LinkedIn job description can run to 2,000 tokens. Multiply that by 100 candidates in a single ranking request, and that's 200,000 tokens processed at once, unworkable at the scale of 1.3 billion members. The solution was to compress that representation, always offline (the result is stored in a database and only read on the online path):

Item representationTokens per itemThroughput (items/s per GPU)Quality loss
Full text~2,000290baseline
LLM-generated summary~150~2,200less than 2%
Single embedding1~22,000equal or slightly better

The summary alone already brought a 16x reduction in prompt size per item, with a throughput gain of nearly 8x. Compressing even further, down to a single dense embedding per job, multiplied throughput by another 10x, and ranking quality did not drop: it stayed equal or slightly better than using the summarized text.

The best representation for your workload is not necessarily the longest one. It's the smallest one that can give you the acceptable quality your use case needs.

Sundara Raman Ramachandran, lead engineer on LinkedIn's LLM Inference team

Inference: the terrain that remains open

For the inference layer, the team did not treat engines like SGLang, vLLM and TensorRT-LLM as an off-the-shelf solution: according to the talk's official summary, they hunted down bottlenecks specific to the ranking workload, including custom flash attention masks, workarounds for Python's GIL, use of gc.freeze, and non-blocking CUDA memory copies. The available transcript material cuts off before the full detail on this part, but the list already indicates that a good share of the production performance gain did not come from the model or the prompt, but from fine-tuned runtime adjustments.

What changes for those building search and recommendation here

No Brazilian company operates at the scale of 1.3 billion members, but the reasoning applies to any marketplace, job board or e-commerce platform that wants to use an LLM to rerank results without blowing up GPU cost or page latency. Three points are replicable in any stack:

  • Using the probability of the token ("yes"/"no") as the score, instead of generating text, avoids the decode loop and structurally cuts latency for binary or multi-class ranking cases.
  • Distillation across multiple teachers specialized by metric (relevance, engagement) makes it possible to train a smaller model without throwing away quality signals.
  • Compressing the item representation (summary or single embedding, generated offline) usually delivers more throughput gain than any isolated infrastructure optimization, and should come before any capacity planning.

Those who already use SGLang or vLLM to serve models may want to take a closer look at SGLang's scoring API, mentioned by Ramachandran as a project he contributes to, as a ready-made path for ranking cases that only need the probability of the next token, without generating anything.

Translated from the Brazilian Portuguese original · Read the original