Larger Open Weights and Inference Compilers Redefine the Cost of Running AI in Production

The November 4, 2026 edition of the AINews newsletter, published by Latent Space, dubbed the week "It's Swapping Season." The text talks about people switching jobs, but the technical content shows something more relevant for anyone putting models into production: the market is swapping out entire pieces of infrastructure. Open weights got bigger and more competitive, compilers emerged that re-optimize inference on their own, and context techniques appeared that cut costs without losing benchmark scores. None of this is an isolated announcement: it's the blueprint for the stack that will run the next generation of agents.

Open Weights Get Bigger (and Weirder to Size)

Moonshot released the weights for Kimi K3.1 Pro this week: a multimodal MoE with 2.8 trillion total parameters and 104 billion activated per token, 93 layers, 896 experts, and a 1 million token context window. Support in vLLM and SGLang shipped on day zero, covering AMD, NVIDIA, Rubin, and TPU, with SGLang also running on Trainium 2 and 3, according to the SemiAnalysis coverage cited in the newsletter. On Agent Arena, the model ranks first among open models and thirteenth overall, with a net gain of 5.1% across more than 17,500 agentic sessions.

In the same batch, Ricursive launched Ripple-65B Preview: a dense 65B model trained from scratch, Apache 2.0 licensed, with 12 trillion training tokens and a stated budget of US$32 million covering all of the research. The company claims it's the best American open weight for agentic coding, ahead of much larger MoEs like Beam-501B (which activates 23B parameters per token). On Terminal-Bench 2.1, the cited score is 80.5% against Beam's 80.1%. Prime Intellect, which ran the pre-training clusters, cites 16 times greater compute efficiency on the scaling curve.

The table below summarizes what changed in this wave of releases, with the numbers the material itself discloses:

The counterpoint the newsletter itself notes: critics point out that Ripple-65B still loses to the 284B V4 Flash, and that the "SOTA among Americans" comparison avoids putting Qwen at the same table. The numbers are worth reading with that caveat: every release picks the competitors that most favor its own result.

The Compiler That Re-Optimizes Inference on Its Own

Together Compiler (TKCC) is described in the material as an agentic compiler that continuously re-optimizes production inference as the model, traffic, and hardware change, already serving trillions of tokens per day. Together claims the compiler's agents run 252 times more optimization experiments per hour than previous systems, reaching 1,111 tokens per second on MiniMax M3.1 on B200 and 1,205 on DeepSeek V4.1 Flash on GB300, with batch size 1 and a 24-hour compilation cycle across two nodes. SemiAnalysis says it independently validated one of these results, which lends some credibility to the number.

The thesis behind this is that agentic compilers reach higher performance ceilings than hand-designed abstractions. The TVM team disagrees and is rebuilding TIRx as a shared foundation for agentic kernel development, arguing that DSLs (domain-specific languages) still matter. For whoever decides on infrastructure, the debate is practical: an agentic compiler promises less manual tuning work, but it's still a black box on top of a black box, replacing fine-grained control with automation you can't audit line by line.

Where Cost Really Drops: Context and Kernel

Two smaller findings in the newsletter have a direct impact on the wallet of anyone running agents in production. The first comes from an NVIDIA summary on context compaction: with one action per context, Opus 4.5 scores 70.2 on SWE-bench Verified against 70.0 with full history, but the median peak context drops from 32K to 8K tokens. Compaction at the end of the conversation reaches 74.8, and the pattern repeats on Terminal-Bench, FRAMES, and BFCLv3.

In practice, this means a good part of the cost of running agents in production doesn't come from the model chosen, but from how history is managed: cutting context without losing (and sometimes gaining) quality is a cost lever that doesn't depend on switching models.

The second finding is llama.cpp v0.9.0, which now supports native FP8 (E4M3), multi-node tensor parallelism via NCCL, and the Mistral Large 4 and Nemotron 3.5 Transcribe models. Native FP8 and multi-node parallelism in an open-source runtime reduce dependence on proprietary stacks to scale inference beyond a single machine, which matters a lot for anyone building their own cluster instead of renting managed GPUs.

What Changes for Builders in Brazil

None of these numbers are ready for local use without context translation. A model with 2.8 trillion total parameters, even with only 104B activated, still requires GPU memory that most Brazilian teams don't have available outside the cloud; hardware like the GB300 and Rubin cited in the newsletter practically doesn't exist in domestic data centers today. In short: for this audience, the real gain from Kimi K3.1 Pro and Ripple-65B isn't "running locally," it's having access via Ollama Cloud, Databricks, or SkyPilot with zero data retention and a permissive license, which makes compliance easier without giving up future self-hosting if the infrastructure allows it.

The cost-routing side is more accessible now. Baseten Grid, cited in the newsletter, promises to unify routing, budgeting, and guardrails across providers, with the company claiming a 72% cost cut on coding tasks compared to Opus 5.5. This matters for anyone paying in dollars per token who feels the exchange rate: choosing the right provider per task, instead of locking into a single model, becomes a FinOps practice as important as choosing the model itself.

The energy bill also comes up in the newsletter and deserves attention from anyone evaluating edge or local data center costs: the Joules-to-GDP study shows that Gemma 4 31B delivers 56% of the value of GLM 5.2 while using 23% of the energy, and that moving a model from 1 to 8 H100s consumes 3.2 times more energy for just a 22% speed gain. In a country where electricity weighs heavily on operating costs, this diminishing-returns curve is a concrete argument against scaling hardware just because you can.

Where It's Not Worth Trusting Blindly

Release numbers are self-reported and optimized to look good in the comparison chosen by the vendor itself. Tencent's Hy4, for example, ranks fourth among open models in an independent Vals evaluation, behind Kimi K3.1 Pro, DeepSeek V4.1 Flash, and GLM-5.3, with 21 tasks stalling on malformed tool calls and 18 timing out. This doesn't invalidate the release, but it shows the gap between the announcement and behavior under third-party evaluation.

The same caution applies to Ripple-65B: the newsletter's own material notes that it loses to the 284B V4 Flash and that the comparison avoids putting strong Chinese models like Qwen on the same scale. Before swapping out a production stack for any of these releases, the practical recommendation is to run your own evaluation, on your own use case, with your own set of tasks, because Terminal-Bench and Agent Arena measure things that may not match the real problem being solved.