One Engine, One Model, Two Chip Families
On October 20, DeepSeek published DeepInfer, the inference framework that actually runs its public API, under the Apache 2.0 license. The README is unusually blunt about the problem it exists to solve: serving a very large user base at a price that does not collapse. One production cluster now pushes 155 trillion tokens a day, and at off-peak pricing that works out to somewhere around $38 of revenue per GPU per hour. Those are the numbers DeepSeek wants you to compare against your own bill.
So the fair question is not "is DeepInfer good." It is how it stacks up against the open serving stacks most teams already run, and where the comparison stops being flattering.
Scorecard: DeepInfer vs. the Incumbents
DeepInfer is not a clean-sheet design. The README names vLLM as its original foundation, and DeepSeek described the project in April 2025 as an early fork of vLLM built around its own cluster management tooling. SGLang and TensorRT-LLM remain the other reference points for anyone serving at scale.
- Model coverage: loser. DeepInfer supports exactly one model, DeepSeek-V4.1-Flash, and by default loads the 1017 build DeepSeek shipped the weekend before launch. vLLM runs hundreds of architectures on day zero.
- Hardware breadth: tie, with an asterisk. SM100 Blackwell and Huawei Ascend 950 only. Hopper and every older Nvidia generation are off the list, which rules out most rented capacity.
- Kernel depth: winner. DeepInfer inherits FlashMLA, DeepGEMM and DeepEP, the operator libraries DeepSeek has been publishing for two years. That is real, tested silicon-level work.
- Operations completeness: loser. Key-value cache management and load balancing were not released. More on that below.
- Reproducibility of the pitch: unresolved. Every published benchmark ran on Nvidia hardware, and every figure is DeepSeek's own.
Two concrete optimizations are worth naming because they explain the cost gap better than any marketing slide. Speculative decoding runs through a system called DSpark: a draft model proposes five tokens at a time and a confidence head guesses which ones the main model will accept, so verification compute only goes to the likeliest candidates. DeepSeek says that trims roughly 30% of the draft tokens checked per step and buys 4% to 7% more throughput at production load.
The second trick is a prefill shortcut. Instead of recomputing an entire prompt, DeepInfer re-runs only the final 128 tokens of each prompt through the model's decoder layers, which the company claims nearly halves total prefill computation. Stack that on a 96% input cache hit rate and the economics start to look less like engineering magic and more like aggressive caching combined with a very specific workload.
The headline serving numbers are 400 milliseconds average time-to-first-token and 230 tokens per second of generation, rising to 300 tokens per second on coding tasks. Fast, but the framing matters: DeepInfer is tuned for offline, throughput-first inference, not for latency-sensitive chat products. Teams shipping real-time features should test before migrating.
Nvidia vs Huawei: A Deliberately Even Scorecard
Supporting Ascend 950 alongside SM100 puts a Chinese accelerator on equal footing inside an official DeepSeek release for the first time. That follows a September 30 drop in which DeepSeek and Huawei brought the TileLang kernel language to Ascend chips, and Bloomberg reported plans for at least 160,000 Ascend 950DT accelerators at a new data center in Inner Mongolia. The vendor comparisons only go so far, though, and the fine print says so.
- Two features that keep outputs identical from run to run work only on Nvidia hardware.
- All published benchmarks were measured on Nvidia chips.
- Huawei support exists in the code, but has no published performance figures behind it.
That is a scorecard where the second column is mostly empty, which is a strange kind of parity. The strategic point still lands: an inference engine that treats Ascend as a first-class target weakens CUDA as the default assumption for anyone building a Chinese deployment.
The withheld layer is the more consequential omission. DeepSeek kept its key-value cache management and load-balancing code private, arguing both are tightly coupled to internal infrastructure. Splitting prefill and decode across separate machines is left to operators, with the README noting most of the engineering is already in place. The smallest deployment DeepSeek calls efficient is 64 GPUs for prefill and another 64 for decode. That is a 128-GPU floor before the interesting parts begin.
DeepSeek also shipped deepseek-recipe, a companion library that converts requests into token sequences using the same logic as the official API. It is the unglamorous piece that prevents self-hosted outputs from drifting away from the hosted service, and it may be the most quietly useful thing in the release.
Context matters here. The framework landed one day after the Department of Homeland Security ordered federal civilian agencies to strip DeepSeek products from their systems, and two days before reports that DeepSeek closed roughly $15 billion in funding at double its original target, with CATL and Tencent among the backers, ahead of a possible Shanghai IPO. Giving away the serving engine costs DeepSeek very little market position and buys an enormous amount of goodwill with developers who cannot buy its API.
Verdict: DeepInfer is not a vLLM replacement and was never going to be one. It is a single-model, single-workload, two-chip serving stack with excellent kernels and a missing control plane. If you run V4.1-Flash on Blackwell at high volume, it is worth serious testing. If you need breadth, Hopper, or a real scheduler, the scorecard is lopsided in the other direction, and DeepSeek knows it, because the withheld code is precisely the part that made the numbers possible.
Comments