NVIDIA H-series GPU

The NVIDIA H-series is a family of data center GPUs based on the Hopper architecture, named after pioneering computer scientist Grace Hopper. Launched in 2022 as the successor to the Ampere-based A100, the H-series quickly became the dominant GPU platform for AI training and inference at scale. The family includes the H100 (the original flagship), the H200 (a memory-upgraded refresh), and region-specific variants like the H20 and H800 developed in response to US export controls.

Built on TSMC's custom 4N process and featuring the Transformer Engine with FP8 and FP4 support, the H-series GPUs introduced a new generation of AI-optimized hardware. Their combination of massive HBM memory bandwidth, scalable NVLink interconnects, and dedicated tensor core hardware made them the backbone of nearly every major large language model (LLM) training cluster from 2023 onward.

The Hopper microarchitecture was unveiled by NVIDIA CEO Jensen Huang at GTC 2022 on March 22, 2022. It represents a fundamental redesign of NVIDIA's data center GPU architecture, shifting the focus almost entirely toward AI and HPC workloads.

Key Architectural Innovations

• Transformer Engine: A dedicated hardware block that automatically manages FP8 and FP16 precision per tensor layer during training and inference, delivering up to 9x faster AI training over the previous generation.
• Fourth-Gen Tensor Cores: Support for FP8, FP16, BF16, TF32, INT8, and INT4 precision formats, with new DPX instructions for dynamic programming algorithms.
• Streaming Multiprocessor (SM) Redesign: Each SM contains 128 CUDA cores (up from 64 in Ampere), 256 KB of L1 cache, and four fourth-gen Tensor Cores. The H100 SXM has 132 SMs, while the PCIe version has 114.
• HBM3 Memory: First GPU to adopt HBM3 memory technology, providing up to 3.35 TB/s of memory bandwidth on the SXM variant.
• NVLink 4: Up to 900 GB/s total bidirectional bandwidth per GPU (18 links at 50 GB/s each), enabling seamless scaling across multi-GPU configurations.
• PCIe Gen 5: 128 GB/s bidirectional bandwidth per GPU for host communication, doubling the PCIe Gen 4 throughput of Ampere.
• MIG (Multi-Instance GPU): Up to 7 MIG instances per GPU, each with dedicated memory, cache, and compute resources for secure multi-tenant workloads.
• Confidential Computing: Hardware-based trusted execution environments for protecting AI models and data while in use.

The Hopper architecture is fabricated on TSMC's custom 4N process, a 5 nm-class node optimized for NVIDIA. The full H100 GPU contains 80 billion transistors on a 814 mm² die, making it one of the largest and most complex chips ever produced.

The NVIDIA H100 is the first Hopper-based GPU, launched in September 2022 and shipping in volume by early 2023. It became the most sought-after AI accelerator in history, with demand far outstripping supply throughout 2023 and into 2024. The H100 is available in two form factors: SXM5 (high-power, high-bandwidth module) and PCIe Gen 5 (standard add-in card).

Specification	H100 SXM5	H100 PCIe
GPU Architecture	Hopper GH100	Hopper GH100
Transistors	80 billion	80 billion
CUDA Cores	16,896	14,592
Tensor Cores (4th Gen)	528	456
GPU Memory	80 GB HBM3	80 GB HBM3
Memory Bandwidth	3.35 TB/s	2.0 TB/s
FP32 Performance	60 TFLOPS	51 TFLOPS
FP8 / FP16 Tensor	1,979 TFLOPS	1,512 TFLOPS
Interconnect	NVLink 4 (900 GB/s)	NVLink 4 (600 GB/s)
Form Factor	SXM5 Module	PCIe 5.0 x16
TDP	700 W	350 W
MIG Instances	Up to 7	Up to 7

At its peak, the H100 commanded prices of $25,000 to $40,000 per GPU on the secondary market, or approximately $2.50 to $4.50 per GPU-hour from cloud providers. Cloud availability exploded through 2024, with virtually every major provider (AWS, Azure, GCP, CoreWeave, Lambda Labs, RunPod, JarvisLabs) offering H100 instances.

Announced at SC23 in November 2023 and shipping in Q2 2024, the NVIDIA H200 is a mid-cycle refresh of the H100 that replaces HBM3 memory with 141 GB of HBM3e. The H200 is the first GPU to ever ship with HBM3e, offering the same Hopper architecture and compute core count as the H100 but with a massive memory and bandwidth upgrade.

Specification	H200 SXM5	H200 PCIe	H100 SXM5 (for reference)
Architecture	Hopper (enhanced)	Hopper (enhanced)	Hopper
GPU Memory	141 GB HBM3e	141 GB HBM3e	80 GB HBM3
Memory Bandwidth	4.8 TB/s	4.8 TB/s	3.35 TB/s
CUDA Cores	16,896	14,592	16,896
FP8 / FP16 Tensor	3,958 TFLOPS	3,341 TFLOPS	1,979 TFLOPS
INT8 Tensor	3,958 TFLOPS	3,341 TFLOPS	--
TDP	700 W	350 W	700 W
Interconnect	NVLink 4 (900 GB/s)	NVLink 4 (600 GB/s)	NVLink 4 (900 GB/s)

The H200's 76% increase in VRAM (from 80 GB to 141 GB) and 43% higher memory bandwidth (4.8 TB/s vs 3.35 TB/s) made it particularly valuable for inference on large models that previously required multiple H100s. Models like Llama 3.1 70B and Mixtral 8x22B, which barely fit on a single H100 in FP8, ran comfortably on a single H200. The H200 is fully backward-compatible with H100-based systems, allowing datacenter operators to swap H100 modules for H200 without infrastructure changes.

In response to US export restrictions targeting advanced AI GPUs for China, NVIDIA developed several reduced-spec variants of the H-series specifically for the Chinese market. These variants comply with US Bureau of Industry and Security (BIS) export controls while still offering competitive AI performance.

H800

The H800 was the first China-compliant variant of the H100, introduced in early 2023. It is essentially an H100 with reduced NVLink inter-GPU bandwidth, capping the interconnects to stay below export limit thresholds. The H800 was widely adopted by Chinese cloud providers and AI labs including Alibaba, Tencent, Baidu, and ByteDance. However, in October 2023, updated BIS regulations closed the loophole that allowed the H800, effectively banning its export.

H20

The H20 is the successor to the H800, announced in late 2023 and shipping in early 2024. It is derived from the H200 design (not the H100) but with significantly reduced compute capability to comply with tightened export rules. Key specifications include:

• 96 GB HBM3 memory on a 4 TB/s memory bus (6 stacks of 5.2 GT/s HBM3)
• Approximately 148 TFLOPS FP8 compute (roughly 15% of the H200's capability)
• NVLink 4 support at reduced bandwidth
• PCIe Gen 5 form factor
• Competitive pricing to counter Chinese domestic alternatives like Huawei Ascend 910B

Despite its reduced compute, the H20's 96 GB of HBM3 memory and decent memory bandwidth made it attractive for inference workloads. However, Chinese AI labs have increasingly turned to domestic alternatives from Huawei and others as US export controls continue to tighten.

The Grace Hopper Superchip (GH200) is NVIDIA's integrated solution combining a Hopper-based GPU with a Grace ARM-based CPU on a single module, connected via the high-bandwidth NVLink-C2C interconnect. First announced in March 2023, the GH200 delivers up to 7x the bandwidth of standard PCIe connections between the CPU and GPU, eliminating the traditional data movement bottleneck.

The Grace Hopper module connects 72 ARM Neoverse V2 CPU cores (Grace) with an H100 GPU (later H200) via a 900 GB/s coherent interface. This unified memory architecture allows the CPU and GPU to access each other's memory pools without the overhead of traditional PCIe-based data transfer. The GH200 was particularly popular in HPC and large-scale AI training deployments where data movement between CPU and GPU is a significant bottleneck.

NVIDIA's DGX H100 and DGX H200 are turnkey AI supercomputers that integrate eight H-series GPUs into a single node with high-speed networking and storage. First announced alongside the H100 in March 2022, the DGX H100 became the standard building block for enterprise AI infrastructure.

Component	DGX H100	DGX H200
GPUs	8x H100 SXM5	8x H200 SXM5
Total GPU Memory	640 GB HBM3	1,128 GB HBM3e
GPU Interconnect	NVSwitch + NVLink 4	NVSwitch + NVLink 4
CPU	2x Intel Xeon Platinum	2x Intel Xeon Platinum
System Memory	2 TB DDR5	2 TB DDR5
Networking	4x ConnectX-7 400Gb/s	4x ConnectX-7 400Gb/s
Power	~10.2 kW	~10.2 kW

DGX systems scale to massive supercomputer clusters using NVIDIA's NVLink Switch System, which connects up to 256 GPUs in a single NVLink domain. The largest known H100 cluster is Meta's deployment with over 350,000 H100 GPUs across multiple datacenters, used to train Llama 3 and Llama 4.

The NVIDIA H-series faces competition from both NVIDIA's own newer architectures and from AMD and other accelerator vendors.

Specification	H100 SXM5	H200 SXM5	B200 (Blackwell)	AMD MI300X
Architecture	Hopper	Hopper Enhanced	Blackwell	CDNA 3
GPU Memory	80 GB HBM3	141 GB HBM3e	192 GB HBM3e	192 GB HBM3
Memory Bandwidth	3.35 TB/s	4.8 TB/s	8 TB/s	5.2 TB/s
FP8 Tensor TFLOPS	1,979	3,958	9,000	~2,600
Interconnect	NVLink 4 (900 GB/s)	NVLink 4 (900 GB/s)	NVLink 5 (1.8 TB/s)	Infinity Fabric (896 GB/s)
TDP	700 W	700 W	1,000 W	750 W
Transistors	80B	80B	208B	153B

H100 vs H200: The H200 is the same Hopper architecture with upgraded HBM3e memory. For memory-bandwidth-bound workloads like LLM inference and large batch-size training, the H200 delivers roughly 40-50% higher throughput. For compute-bound workloads where the model fits in 80 GB, the H100 performs similarly.

H200 vs B200 (Blackwell): The B200 is NVIDIA's next-generation architecture with 2.3x the FP8 performance, 2x the memory bandwidth, and 192 GB of HBM3e. While more expensive, the B200 offers significantly better performance per watt for training. However, the H200 remains competitive for inference where its ample memory and mature software stack make it a cost-effective choice.

H100/H200 vs AMD MI300X: The MI300X offers 192 GB of HBM3 memory (more than the H200) and competitive compute performance at a lower price point. However, AMD's ROCm software stack has historically lagged behind NVIDIA's CUDA ecosystem in maturity and broad framework support. For organizations already invested in CUDA, the H-series remains the safer choice, while AMD has made significant gains with PyTorch-native support in 2024.

H-series vs Consumer GPUs (RTX 4090/5090): For individual developers and small-scale AI work, consumer GPUs offer dramatically better price-to-performance. An RTX 5090 at $1,999 delivers about 15-20% of an H100's AI performance for less than 10% of the price. However, the H-series GPUs dominate in multi-GPU scaling, memory bandwidth, datacenter reliability features (ECC, MIG, virtualization), and support for the largest model sizes.

The NVIDIA H-series has been at the center of the US-China technology export control dispute. In August 2022 and again in October 2023, the US Bureau of Industry and Security (BIS) imposed increasingly strict export controls on advanced AI semiconductors, directly targeting NVIDIA's H100 and its derivatives.

The October 2023 rules created a "performance density" threshold that effectively banned the H100, H800, and A100 from export to China (and other restricted countries). NVIDIA responded by developing the H20, which falls below the compute thresholds while retaining significant memory capacity. As of early 2025, the H20 itself faced potential further restrictions as the US government continued to close loopholes with each successive rule update.

These export controls have had far-reaching consequences: they accelerated China's push for domestic AI chip development (Huawei's Ascend series, Cambricon, Biren), reshaped global AI supply chains, and created a bifurcated market where Chinese AI labs operate under significant hardware constraints compared to their US counterparts.