Microsoft bets on Maia 300 to take inference out of Nvidia's hands

Microsoft announced the Azure Maia 300, the third generation of its AI chip system, on October 13. Production begins in 2027. The stated focus is on inference within data centers.

The motivation is clear. Moreover, the company wants to reduce its dependence on external infrastructure vendors, especially Nvidia.

It's worth remembering what inference means in this context. It is, however, the phase in which an already trained model generates responses, as happens every time a question is asked to Copilot.

The numbers the company promises

The comparison with GPUs deserves attention. According to the company, the chip delivers 40% more tokens per megawatt than leading GPU-based systems.

The gain between generations also showed up. Moreover, in a configuration with 96 accelerators, the token volume per rack reaches seven times that of the Maia 200.

The planned scale is impressive. The company intends to expand the family to multi-gigawatt installations starting in 2027.

Microsoft designed the chip prioritizing inference

The architecture follows an approach called inference first. In other words, the design starts from the workloads that already dominate real-world use.

Three scenarios appear in the justification. These include tasks that analyze large volumes of information and maintain context over long periods.

Then come latency-sensitive conversations. Finally, there are agent tasks that require multiple steps.

The bottleneck being addressed is familiar to anyone who works with large models. Moreover, the system uses software-defined data flow, with optimizations to keep data close to the processors.

As a result, latency drops and hardware utilization improves.

Memory and networking underpin the design

These two points stand out as central to the design. The chip uses eight stacks of HBM4.

Memory bandwidth reaches 18.0 TB/s. Power consumption, in turn, reaches up to 950 watts.

Communication completes the design. It includes high-bandwidth interconnects between accelerators and a custom network transport running over Ethernet infrastructure.

Note the choice of Ethernet. It also brings the design closer to standard data center infrastructure.

What the Maia 200 already does today

The current generation runs in production. It handles workloads such as Microsoft Copilot and supports the MAI team in model development.

The rollout continues. GitHub Copilot and Microsoft Foundry are set to join by the end of the year.

The chip runs models from various sources. The list includes OpenAI models, its own models, and open-source families.

Deployment already covers three regions: the United States, Europe, and Australia.

For features built into Teams and Copilot, the company cites up to 1.7 times more tokens per watt compared to leading GPU-based infrastructures.

Microsoft enters a field that already includes Google and Amazon

The custom-silicon strategy is no longer an exception. Google continues with its TPUs, designed for machine learning.

The competition with Nvidia, therefore, gains another vertical competitor. Each cloud now offers its own path for inference.

The company ties this line to a broader initiative called Frontier Transformation. More details are expected at Microsoft Ignite 2026.

What this changes for developers

First, think about portability. Code locked to a single accelerator stack limits future choice.

Second, prefer high-level frameworks. PyTorch and inference runtimes abstract away much of the differences between vendors.

Third, keep an eye on the price per token in the cloud. Custom silicon tends to push that value down.

Also, measure tokens per watt when energy cost comes into play. This metric, however, has been gaining ground in infrastructure decisions.

Follow our profile on Instagram!