🔥 BREAKING: DeepSeek-V4.1-Flash now generates code at 2,060 tokens per second, and all you need is eight B300 GPUs, a CUDA 13 toolchain, a Rust build step, and a Redis instance. Check the shed. No? Then this one is not for you.
Baseten open-sourced the engine this week as deepseek-ultrafast, and the numbers are the kind of thing that makes a homelab forum start pricing out datacenter leases at 2am. The Hacker News thread made the front page with one comment that sums up the entire situation: "I'll just go get my 8 x B300 (TP8) from the shed and start this up real quick."
Which is fair. The tokens-per-second leaderboard has quietly turned into the AI hardware equivalent of a dyno chart, and everybody is posting peak numbers.
What Baseten Actually Shipped
The engine is a purpose-built inference stack for one model family, tuned for Blackwell silicon: sm_100a covers B200, sm_103a covers B300 and GB300. It supports the full 1M context window, and it shares KV caches across servers so a conversation that lands on a different replica does not have to re-read the whole prompt.
Here is the sled the whole thing runs on:
- 4 x GB300 (TP4): 1,740 tokens/s decode, 3.30 ms per step, 1M tokens prefilled in 13.2 seconds (76,000 tokens/s).
- 8 x B300 (TP8): 2,060 tokens/s decode, 2.71 ms per step, 1M tokens prefilled in 11.7 seconds (85,000 tokens/s).
- Bare minimum: 4 B200 or 4 B300-class GPUs, CUDA 13, the Rust toolchain, the DeepSeek-V4.1-Flash weights on disk, and Redis running to track cache ownership.
For the nerds: those decode numbers rest on speculative decoding. Each step drafts a block of tokens and then verifies it, and the average accepted length is 5.6 tokens per step. That is why 2.71 ms per step turns into two thousand tokens a second instead of 370. It is also why the figures wobble by prompt: Baseten reports a 1,700 to 1,770 range on four GB300s and 1,980 to 2,170 on eight B300s across 30 runs.
🎮 Then there is the benchmark prompt. Not MMLU. Not a SWE-bench subset. Literally this: "implement the tetris game. output HTML code directly - no explanation." The demo streams a complete playable Tetris implementation at real-time speed with a speedometer overlay, then plays the finished game next to the generated code. The reproduction script ends on a line like 5,142 tokens at 1,717 tokens/s, first token after 0.04 seconds.
Which is the most honest eval of 2026. Nobody actually wants the model to reason about the trolley problem. They want it to finish the game before they lose interest.
Meanwhile, in a Shed in South Carolina
The comedy writes itself when you put this week's two DeepSeek stories side by side. On one side, Baseten measuring thousands of tokens per second on eight datacenter Blackwell parts. On the other, a trial lawyer named Nick Mermiges running roughly 80% of the same V4.1-Flash model on eight used V100s, in a private server he just upgraded with 1TB of RAM and an RTX 3090.
His decode rate: 150 to 180 tokens a second for code, down from 160 to 215 before the precision upgrade. He hoped to hold 80 tokens per second on legal prose and landed at 78 to 86. The crashes he first blamed on PCIe turned out to be, in his own words, "probably one bad PSU."
So the gap between the two setups is about twelvefold in throughput and roughly one datacenter in capital cost. Both machines run the same weights. Only one of them has a shot at generating Tetris during a coffee sip.
The fine print on the Baseten engine, for anyone currently refreshing a GPU retailer:
- It is one model, not a general server. This is a V4.1-Flash engine, not a drop-in vLLM replacement for your whole fleet.
- Cache sharing wants RDMA. The cross-server transfer moves host memory to host memory over the fabric, with Redis recording which replica owns which conversation.
- The kernels were written by agents. Baseten credits coding agents that could read the FlashInfer and SGLang source while they optimized. The tooling wrote the tooling.
- Your harness may not see the number. Users running the model through other clients report nowhere near the advertised rate, which suggests the engine and the harness are a package deal.
☕ The Tea. The interesting part is not 2,060 tokens per second. Peak throughput has been a marketing contest since the first GPU got benchmarked, and it will stay one. The interesting part is that an open-source engine now exists which treats a million-token context as a prefilling problem measured in seconds, and it shipped with reproducible instructions instead of a vendor blog post and a waitlist.
That is the actual flex. Baseten published the hardware list, the build flag, the Redis line, and the exact prompt it used to pad its chart. You can argue with the benchmark or fail to afford the B300s, but you cannot accuse them of hiding the method. What to actually take away:
- For labs: throughput is a publishable artifact now, and the good ones ship reproduction scripts instead of press releases.
- For homelabs: the model weights are free, the speed is not. Same file, wildly different experience.
- For everyone else: a 5,142-token Tetris game has quietly become the unit of AI performance.
And if your shed only has V100s, the lawyer already proved the model runs. Just budget for the PSU.
Token counting, one Tetris at a time.
Comments