For eight months, Jalapeno has been the AI industry's most anticipated black box. OpenAI and Broadcom unveiled the custom inference accelerator last October and again in June, but every public update was heavy on architecture and light on hard numbers. On August 25, OpenAI finally opened the box at the Hot Chips conference and shared the first measured benchmark results. They are striking: across three open-weight models, OpenAI's first chip delivered up to 1.9x more useful AI work per watt than a state-of-the-art commercial system, plus dramatically lower latency.
Richard Ho, OpenAI's head of hardware, put it plainly in a press call: "The bottom line is that the results show a very, very significant performance advance over state of the art. Jalapeno can serve more AI work per unit of power, while also returning responses more quickly." The comparison system driving those numbers is an Nvidia Blackwell deployment, which is what makes the benchmark meaningful today and also the reason to keep some skepticism about tomorrow.
The InferenceX Numbers, Decoded
OpenAI tested Jalapeno on InferenceX, a public benchmark created by SemiAnalysis that measures the entire process of serving an AI request rather than raw chip specification. The methodology matters: OpenAI evaluates performance at a matched user experience, measuring useful work completed per unit of power while still meeting the latency customers and interactive agents expect. It is a scope decision designed for agentic workloads, where many sequential steps compound delays into a painfully slow overall task.
The headline results, normalized per kilowatt using each accelerator's published power rating, look like this for GPT-OSS 120B:
- Peak mixed tokens per second per kilowatt rose about 1.9x, from 44,960 to 85,448.
- End-to-end latency dropped about 1.7x, from 1.80 seconds to 1.03 seconds.
- Minimum token-to-token latency improved about 2.7x, with per-user throughput jumping from 535 to 1,459 tokens per second.
- At the comparison system's original throughput point, Jalapeno delivered roughly 53.7x more mixed throughput per kilowatt.
The gains are not confined to one model. On DeepSeek R1 670B, peak throughput per kilowatt rose about 1.7x and end-to-end latency fell about 3.6x. On Kimi K2.5 1T, the largest public model tested, the chip delivered about 1.5x higher peak performance per watt and 3.4x lower end-to-end latency. Across all three models, total gains ranged from 1.5x to 1.9x more AI work per watt at peak throughput, with 1.7x to 3.6x lower end-to-end latency, and 2.1x to 4.1x higher performance on highly interactive workloads.
Why the Architecture Wins
Language-model inference is not a single bottleneck. Prefill, the phase that processes an incoming prompt, is compute-intensive. Decode, which generates the response token by token, is bound more by memory bandwidth. Communication between cores and chips adds latency, leaving processing units idle while they wait for data. A system that excels at one stage can surrender its advantage while moving model state for another.
OpenAI designed Jalapeno to minimize that churn. Model state, including the KV cache generated during response, can be explicitly placed and kept local while the chip activates the right mix of compute, memory, and networking for each inference phase. The network is integral rather than peripheral, with a large domain that keeps an entire workload inside one connected system. The result is a balanced, fungible accelerator that handles both prefill and decode well and adapts to changing model architectures.
AI shaped the silicon itself. OpenAI's models helped move the design from initial concept to tapeout in nine months, an unusually short cycle for a chip of this scale, by shortening design, measurement, and verification loops. The same feedback produces software: AI-generated implementations of selected GPT-OSS attention and mixture-of-experts blocks ran 1.5x to 1.8x faster than the existing human-expert-written kernels. Using Codex with GPT-Astra, the team brought three open-weight models that were outside the original production plan to high performance within two months.
What It Means for the Market
Jalapeno joins the broader first-party silicon movement from Google, AWS, and Meta, but with a twist: OpenAI argues performance per unit of power, not per chip, is the metric that matters for serving intelligence at scale. On that standard, the chip sits on the Pareto frontier, offering combinations of speed and efficiency no single commercial competitor matched in the tested window.
Practical impact is still ahead. Ho said Jalapeno would begin deploying within OpenAI's compute infrastructure by the end of 2026 "in very small volumes," with more significant deployment in 2027. It is the first of a multigenerational roadmap; Gen 2 is deep in development and Gen 3 is taking shape. OpenAI stresses it will keep deploying Nvidia and other partner accelerators for both training and inference, positioning Jalapeno as a complement for the right workloads rather than a wholesale replacement.
The honest caveat is timing. Those results compare against the Nvidia Blackwell generation shipping today, and Ho explicitly noted the competition may advance before Jalapeno's production ramp. But the first measured proof of an in-house inference path for the world's leading model lab changes the geometry of the hardware market. If even a fraction of these per-watt figures survive production qualification, the economics of serving frontier models look very different.
Comments