96-chip configuration promises seven times its predecessor's token throughput, with production deployments due in 2027
Microsoft has lifted the lid on the next generation of its Azure Maia custom AI chips, saying it will usher in a new phase of its compute infrastructure where everything is closely integrated to meet the demands of agentic inference.
The Redmond giant says its Azure Maia 200, announced earlier this year, is now serving production workloads and that a successor is already planned for 2027.
Azure Maia 300 is described as "a complete intelligent system" designed for the new types of AI workloads emerging today. Microsoft claims a full 96-accelerator configuration will deliver up to seven times Maia 200's rack-scale token throughput, while offering 40 percent more tokens per megawatt than leading GPU-based systems.
Its next-gen chip is created with an inference-first architecture optimized for long-context, memory-intensive and decode-heavy AI workloads, according to a blog post by Microsoft's EVP for Cloud and AI, Scott Guthrie.
The firm envisions it being deployed in rack-scale systems comprising up to 96 accelerators, while each Maia 300 will feature eight HBM4 memory stacks providing up to 18 TB/s of peak memory bandwidth, the better to support execution of large models and long-context workloads.
This sounds similar to the kind of rack-scale AI equipment that has already been developed by Nvidia for its GPU accelerators, and more recently by AMD with its Helion and Helios infrastructure, all designed to make the most of their own specific silicon and software stacks.
Microsoft says that Maia 300 is "built as a complete system, with software, silicon, physical rack infrastructure, networking, and datacenters co-designed to operate as a single platform to meet the demands of cloud-scale agentic inference."
It insists that this system-level approach is necessary for agentic workloads, which involve multi-step execution, long-context decoding and tool use, and "change the computational profile and arithmetic intensity of inference."
The growing emphasis on in-house integration contrasts with Microsoft's 2024 "AI Access Principles," which stressed partnerships across its technology stack.
The firm even complained to the European Commission's competition authorities that Google had a competitive advantage because of its approach.
"Only one company – Google," said Microsoft at the time, "is vertically integrated in a manner that provides it with strength and independence at every AI layer from chips to a thriving mobile app store. Everybody else must rely on partnerships to innovate and compete."
Microsoft now has its own AI chips, rack-scale designs, and internally developed MAI models, giving it more control over its stack. It plans to scale Azure Maia to "multi-gigawatt deployments" starting in 2027.
We asked Microsoft for comment.
Guthrie said that the firm will share more details on its strategy at next month's Microsoft Ignite 2026 event in San Francisco. ®