Back to Home

GLM-5.3 Too Big for One GPU: Two DGX Stations, 608 Tok/s

For two years the pitch for local AI has been the same: wait for the next smaller model, then squeeze it onto the GPU you already own. GLM-5.3 just walked in and closed that door. NVIDIA's developer blog published a recipe on November 13 for serving the quantized GLM-5.3 NVFP4 checkpoint across two DGX Stations, and the reason is refreshingly blunt. The model does not fit on one GPU anymore.

Numbers first, because they carry the story. Z.ai's mixture-of-experts model ships with 753 billion parameters and activates 40 billion per token. Even with NVFP4 quantization aimed at the MoE block weights and activations, the checkpoint lands at about 464 GB. A single GB300 GPU has 252 GB of HBM3e. There is no clever flag that saves you here. You are short roughly 212 GB before the KV cache even asks for a room.

Two boxes, one endpoint

NVIDIA's answer is tensor parallelism with TP=2: two DGX Stations joined by a ConnectX-8 RDMA rail, each holding about 218 GiB of weights, each computing half of every request, and a single OpenAI-compatible API sitting on the head node. Station A runs rank 0 and answers the world. Station B runs rank 1 headless and quietly does half the math. Neither GPU holds a complete copy. Neither can answer alone.

The engineering strip is worth reading closely, because it is where the enthusiasm is justified:

  • Multi-token prediction with four draft tokens per step, verified by the full model before any of them are accepted.
  • FP8 KV cache storage, which halves cache memory and buys back room for context.
  • Prefix caching, so a repeated system prompt is not reprocessed from scratch on every call.
  • A 98,304 token context ceiling and 93 percent GPU memory utilization on both nodes.
  • A pinned vLLM 0.30.0 arm64 image plus 50 safetensors shards per disk, with a checkpoint hash comparison across both machines.

What 607.8 tokens per second actually means

NVIDIA measured the finished service with AIPerf 0.12.0 on the SPEED-Bench Coding set: 80 real conversations, 89 turns, default reasoning effort, an 8,192 token output cap. The results draw a very honest curve.

  • Peak overall throughput: 607.8 tokens per second at concurrency 16.
  • Concurrency 32: 385.0 tokens per second, with mean request latency of 453.5 seconds.
  • Per-user throughput: 115.3 tokens per second at concurrency 1, falling to 13.0 at concurrency 32.

That is not a marketing chart. It is a machine telling you exactly where its sweet spot sits. Two GB300 systems saturate somewhere around sixteen simultaneous coding conversations, and past that point you are trading queue depth for nothing. For a team of eight to twelve engineers hammering a private model, that is a perfectly good place to live. For a startup trying to serve thousands of users, it is a reminder that a desk-side cluster is not a fleet.

The local-first argument finally has hardware behind it

What makes this a breakthrough rather than a curiosity is what it unlocks. Sensitive code stays on your own network. Agents get prototyped without a per-token invoice attached to every experiment. Prompt iteration stops costing money per run. Tool calling is enabled in the launcher as well, so agent frameworks that speak the OpenAI tools API can point straight at the same local endpoint and behave like they are talking to a hosted provider.

The cost math is the part everyone will run in their heads this week. Two DGX Stations is not a cheap purchase, and nobody should pretend otherwise. But compare it against renting a frontier model at volume for a year of heavy agentic coding, add in the proprietary data that never leaves the building, and the crossover arrives sooner than the just-use-the-API crowd expects. NVIDIA knows this perfectly well. The post closes by calling a frontier model ready to build on, right from your desk.

Skeptics, and there are plenty, will point at the fine print. The default launch exposes an unauthenticated API on every interface unless you firewall the port, which the post flags in bold. A full benchmark sweep takes hours. Two machines means two of everything: two checkpoints, two container stacks, two sets of drives with 464 GB free per node. RDMA cabling mistakes do not fail politely either. They fail as NCCL timeouts forty minutes into loading weights.

Still, the direction of travel is unmistakable. This time last year, an open-weight model at 753 billion parameters was a hosted product with a login page and a rate limit. Now it is a two-box appliance with a published runbook, a pinned image digest, and a benchmark table you can reproduce on your own bench. The gap between an open model and a model I can actually run closed another notch on November 13, and it closed from the top down.

Local AI used to mean compromising on capability and bragging about the compromise. GLM-5.3 NVFP4 moves the compromise to rack space and a second power bill. That is a much better argument to lose.

Comments

No comments yet. Be the first to share your thoughts!