How LinkedIn Trains AI Job Search 8x Faster

LinkedIn published details of the training infrastructure behind its AI-powered job search, and what's interesting for those building ML systems isn't the technique itself, but the systems engineering that makes it fast enough to iterate on. According to the report published by InfoQ, the company uses a multi-teacher distillation pipeline that compresses the knowledge of large models into a compact ranking model with 0.6 billion parameters, now in production serving natural-language search for users in the US.

The problem that drove all of this is familiar to any search and recommendation team: migrating from keyword-based systems to unified rankers supervised by LLMs. Training a small model (SLM) to optimize relevance and engagement (clicks, applications) requires querying one or more large teacher models for every training example. Serving those teachers becomes a bottleneck when the system has to handle hundreds of thousands of queries per second at LinkedIn's scale.

The two modes: online and offline

The core insight is a multi-teacher distillation framework built on top of SGLang, the open source LLM serving engine that LinkedIn had already been investing in for previous ranking workloads, instead of a proprietary stack. This framework loads and serves teachers of different sizes, managing tensor-parallel and data-parallel setups. An asynchronous client queries the teachers during training, processes their outputs, and folds them into the distillation losses. LinkedIn calls this Online Multi-teacher Distillation.

Distributing this across multiple nodes with local replicas of the teachers sped up distillation by 3x while keeping latency low. For when the teachers have already stabilized, there's Offline Multi-teacher Distillation: the system precomputes the teachers' outputs and stores them in HDFS or NFS, using them directly during training instead of querying in real time, eliminating repeated computation.

The practical takeaway for teams building LLM-supervised rankers is this strategy split by phase:

| Project phase | Recommended mode | Why | |---|---|---| | Early on, teacher choices still changing | Online | You swap teachers frequently; caching would be wasted work | | Teachers stable, query volume rising | Offline | Precomputing cuts the overhead of repeated serving |

Where the 8x comes from

The headline talks about ~8x faster, and it's worth understanding that this number is cumulative, not a single trick. The online/offline split coexists with a stack of training-level optimizations:

• LiGer to cut memory usage and allow batches 2x larger;
• multi-node training for up to an additional 3.5x speedup;
• FSDP2 for another 20%;
• multi-node clusters with H200 for up to 30% extra on top.

Stacked together, these optimizations add up to the roughly eightfold gain cited in the post. One important detail for anyone who now treats reduced precision as the default: the team evaluated mixed FP8 precision and found no benefit for models under 8B parameters, because the casting overhead outweighs any compute savings. It's the kind of negative result that saves other people weeks of experimentation.

The quality and throughput numbers

On the modeling side, the 0.6B student model distills knowledge from two teachers: an 8B relevance oracle and a 1.7B engagement teacher. The reported result is a 24.48% improvement in NDCG@10 for job search, jumping from 0.7583 to 0.9432.

On the inference side, the work included structured pruning and context compression, which raised ranking throughput from around 290 to over 2,000 items per second per GPU. This is the point that closes the economic case: the company presents the work as a way to get cross-encoder ranking quality while meeting real-time latency, without the cost of running frontier LLM inference on every request.

What changes for ML builders in Brazil

For the Brazilian dev or ML engineer running search, recommendation, or any ranker in production, the takeaway here is less about "LinkedIn is big" and more about architecture decisions that can be reused even with a smaller GPU budget. Multi-teacher distillation doesn't require a pool of ten specialized teachers the way frontier models do, cited in the 2026 survey on distillation (NVIDIA Nemotron 3 Ultra, MiMo-V2-Flash, DeepSeek-V4). Industrial ranking systems use few, task-specific teachers, which is far more feasible to reproduce.

A few direct takeaways from the article:

• Split teacher serving into online and offline based on the experiment's maturity. You don't need both at the same time, and offline cuts cost as volume grows.
• Don't assume FP8 by default on small models. Below 8B, casting can cost more than it saves.
• Training gains are additive, not isolated. LiGer, multi-node data parallelism, FSDP2, and a newer GPU generation each deliver a moderate boost; the big speedup comes from the combination.
• SGLang is the servable foundation. The framework is open source, so you can inspect and adapt it without being locked into a proprietary serving stack.

LinkedIn itself positions the work as a guide for teams, and it's worth contrasting it with a sibling case: in June 2026, Pinterest published "Achieving Near-Linear Training Scalability for Pinterest's Foundation Models", which describes how multi-node training made it possible to build larger teachers whose knowledge was distilled into more efficient students for the Homefeed and Related Pins rankers, cutting experimentation cycles from weeks down to a fraction of that. The difference in focus is telling: Pinterest scales its training framework and migrates to Distributed Checkpoint, while LinkedIn builds a custom SGLang framework to query teachers live during training. These are two routes to the same goal, compressing large models into small students without losing ranking quality.

What remains open

The post doesn't open-source the multi-teacher distillation framework's code itself, only confirming that it runs on top of open source SGLang. The NDCG and throughput numbers are specific to the job search domain and the H200 hardware, so reproducing the same gain in another domain or with more modest GPUs remains an open exercise. Even so, the map is drawn: the online/offline split, the stack of additive optimizations, and the warning about FP8 are engineering decisions that any ML team can test before spending weeks discovering the same thing on their own.