Hot Chips 2026: the inference chip race that's shaking up the cost of running AI in production

The 37th edition of Hot Chips was dominated by a dispute that rarely makes the spotlight: who builds the silicon that runs model inference, not the one that trains them. According to the summary from Latent Space, the loudest announcement came from OpenAI, which revealed the first benchmarks for Jalapeño, its own inference chip, less than a year after its partnership with Broadcom. The surprise wasn't that the chip existed, that was expected, but the fact that it isn't a niche ASIC and is instead, in the coverage's own words, a direct competitor to Blackwell architecture.

For those building products on top of LLMs, this isn't hardware gossip. Inference cost is the bill that shows up every month on the API invoice or the rented GPU. When a chip promises to do the same work while spending less energy and delivering faster responses, that bill changes, and with it the viability of running models in production in Brazil, where energy, exchange rate, and transatlantic latency already weigh heavily.

What Jalapeño claims to deliver

The numbers released by OpenAI, reported by Latent Space, are specific enough to warrant a skeptical reading. In real model workloads, compared to NVIDIA GB200/GB300 systems, Jalapeño reportedly delivered:

• 1.5 to 1.9x more work per watt at peak throughput;
• 1.7 to 3.6x lower end-to-end latency;
• 2.1 to 4.1x more performance in highly interactive workloads (the typical chatbot and agent case).

The detail that caught analysts' attention is power consumption: the chip is rated at 700W, but reportedly stayed at 550W or below in testing. And there's a technical point that any inference engineer immediately understands: in some scenarios Jalapeño reportedly outperformed rival systems without resorting to tricks like aggressive prefill/decode disaggregation or speculative decoding, techniques the competition used to inflate its own results. If confirmed once the full presentation is released, this suggests a more balanced architecture, one that reduces the classic trade-off between throughput (serving many people) and latency (responding quickly to each one).

The usual caveat applies: these are the manufacturer's numbers, on workloads chosen by the manufacturer. SemiAnalysis, cited in the source, classified the result as unusually strong for a first-generation ASIC and compared the chip directly with Blackwell- and Rubin-class systems. But a first-generation benchmark calls for third-party replication before it becomes an architecture decision.

The compiler written by the model itself

The second-order story is the most interesting one for anyone following applied AI. OpenAI states that GPT-Astra + Codex helped write and optimize the chip's low-level kernels, bringing three unplanned open-weight models to high performance on Jalapeño in roughly two months. For selected attention and MoE blocks, these implementations reportedly ran 1.5 to 1.8x faster than code written by human experts.

If this holds up, it's a concrete sign that compiler and kernel work is being folded back into the model's own improvement loop, no longer confined to the application layer. In practice, the historical bottleneck of porting a new model to a new accelerator (weeks or months of manual kernel engineering) starts to shrink. This matters because the reason NVIDIA has been so hard to dethrone was never just the hardware, it was CUDA and the software ecosystem around it. If the model writes the chip's software, that moat gets shallower.

The other competitors at the table

Jalapeño wasn't alone. Hot Chips brought together the full lineup of those attacking inference from different angles:

• Cerebras CS-5: the wafer-scale bet, a chip the size of a dinner plate, that eliminates GPU-to-GPU communication by keeping the entire model in on-chip memory. It's the architecture designed for minimal latency in large models.
• Groq 3 LPX: the LPU (Language Processing Unit) line that built its reputation on token generation speed through deterministic execution, without the scheduling variability of GPUs.
• Apple M6: the local inference angle. The source notes that Apple started featuring the exo project on its Mac Studio M5 Ultra and Mac Mini M6/M5 Pro pages, with low-latency RDMA over Thunderbolt 5 to cluster Macs and run models like Kimi K3 and GLM-5.3 at API speed. Four M5 Ultra units would reportedly scale to around 4.8 TB/s of aggregate memory bandwidth.

These are three distinct philosophies: Cerebras and Groq sell cloud inference with a predictable-performance characteristic; OpenAI verticalizes to serve its own models more cheaply; Apple pushes inference into the user's own machine. Each one attacks a different part of the cost equation.

What this changes for those building in Brazil

For the Brazilian startup or developer, the practical effect isn't buying a Jalapeño, that chip goes into OpenAI's internal infrastructure starting at the end of the year, with no standalone sale. The effect is indirect and comes through two paths.

First, price pressure. If OpenAI manages to reduce the cost per token served with its own silicon, that opens room for more aggressive API pricing, or at least for holding prices steady while model quality rises. The same logic applies to Groq and Cerebras, which already sell inference as a service and compete on cheap tokens. More competition at the bottom of the stack tends to trickle down to the invoice of whoever consumes the API.

Second, viable local inference. The Apple/exo angle is the most tangible one for Brazil, where latency to US data centers and dollar-denominated cloud costs hurt. Running a mid-sized model locally, on a Mac cluster or a single machine with high memory bandwidth, stops being a hobby and becomes a defensible architecture for anyone handling sensitive data or unwilling to depend on an international round trip. Perplexity itself launched its Portable Computer the same week, built on NVIDIA DGX Spark, with orchestrator, subagent, and harness running 100% locally, though the community's criticism was fair: calling a US$5,000 piece of hardware "local-first" is different from running on a consumer device.

The bottleneck no chip solves

There's a limit that runs through all these announcements and is worth noting: packaging and foundry capacity. As the source points out, citing LearnOpenCV, TSMC's advanced packaging processes (CoWoS) and fab capacity remain the hard bottleneck. It doesn't matter how efficient the architecture is if there's no production line to make the chips at volume. OpenAI already speaks of Gen 2 in advanced development and Gen 3 underway, but even Gen 1 depends on the same manufacturing queue everyone else is fighting over.

The takeaway from Hot Chips 2026 is that the era in which inference economics were dictated exclusively by NVIDIA is cracking. For those putting models into production, the metric to watch is no longer peak TFLOPs but performance per watt, because that's what determines how much each response your product delivers to the user actually costs.