HBM raises the memory bandwidth ceiling for AI systems, with advanced packaging, thermal management, and reliability becoming critical to the next stage of scaling

Overview

This is a technical report on AI memory architecture published by Micron. It focuses on explaining why memory has become a core constraint for large models and accelerated computing, and how HBM alleviates the “memory wall” through high bandwidth, high parallelism, and heterogeneous integration. The report also reviews the evolution from HBM1 to HBM4 and discusses future challenges involving high-speed interconnects, advanced packaging, thermal management, and reliability.

Core views

The report begins with the scaling laws of large models: language-model performance improves smoothly as model size, dataset size, and training compute increase, but all three must scale in tandem, and a bottleneck in any one factor will limit the overall gains. As GPU compute performance continues to rise, the “memory wall,” caused by data not reaching processors quickly enough, is becoming increasingly prominent. Memory has therefore evolved from a supporting component into a core factor determining whether AI system performance can continue to scale. The report uses the Roofline model to explain this constraint. The model measures attainable performance in operations per second and arithmetic intensity in operations per byte. Applications with low arithmetic intensity operate in the memory-bound region, where performance is determined by memory bandwidth, while applications with high arithmetic intensity are more likely to be limited by compute capability. HBM raises the memory bandwidth ceiling, allowing memory-intensive AI workloads to achieve higher performance before reaching the bandwidth limit and thereby serving as a “compute-memory bridge” between computing capability and data-delivery capability. HBM achieves high parallelism through multiple independent channels, wide I/O, short-distance point-to-point connections, and 3D stacking. Each DRAM die contains multiple independent channels, and each channel has two pseudo-channels that share command and address buses but use independent data buses. The base die handles command and data interfaces between the host and DRAM, the micro-bump PHY connects the host to the base die, and the 3D TSV PHY connects the DRAM stack to the base die. HBM3E has 128 banks per die, increasing to 256 in HBM4. High-density interposer connections increase from approximately 1K I/Os in HBM3E to approximately 2K I/Os in HBM4. Generational specifications show that HBM continues to evolve in bandwidth, channel count, and capacity. The respective milestones for HBM1, HBM2, HBM2E, HBM3, HBM3E, and HBM4 are 2014, 2018, 2020, 2022, 2024, and 2026. Their channel counts are 8, 8, 8, 16, 16, and 32, respectively; pseudo-channel counts are none, 16, 16, 32, 32, and 64; and pseudo-channel widths are 128, 64, 64, 32, 32, and 32 bits. Burst length increased from 2 in HBM1 to 4 in HBM2 and HBM2E, and then to 8 from HBM3 onward. The prefetch width is 256 bits across all generations. The number of data I/Os remained at 1024 from HBM1 through HBM3E and increased to 2048 in HBM4. The corresponding nominal data rates from HBM1 through HBM4 are 1, 2.4, 3.6, 6.4, 8, and 11 Gbps, while nominal bandwidths are 128, 307, 460, 819, 1024, and 2800 GB/s. DRAM densities are 2, 8, 16, 16, 24, and 24 Gb, respectively. Stack height evolved from four layers in HBM1 to four/eight layers in HBM2 and HBM2E, eight/12 layers in HBM3, and eight or more layers in HBM3E and HBM4. Capacity per stack accordingly increased from 1 GB to 4/8 GB, 8/16 GB, and 16/24 GB, reaching 24 GB or more in HBM3E and HBM4. Each generation simultaneously raises the data rate, number of pseudo-channels, density, and stack height to improve bandwidth, access granularity, efficiency, and capacity, respectively. The system comparison further illustrates HBM's bandwidth advantage. An example system using eight DDR5 channels has a theoretical bandwidth of 307 GB/s at 4800 MT/s with two 32-bit subchannels per channel. Assuming two ranks per channel and 64 GB per DIMM, total capacity can reach 1 TB. Another example system consisting of eight HBM3 stacks has a theoretical bandwidth of 5.3 TB/s at 5.2 Gbps, with 16 channels per stack and two 32-bit pseudo-channels per channel. With 24 GB per stack, total capacity is 192 GB. The two architectures reflect different priorities: conventional DIMMs provide greater capacity, while HBM trades capacity for an order-of-magnitude increase in system bandwidth by being placed close to the GPU and widening the parallel interface. At the core level, 256 I/Os operating at 8 Gbps can provide 256 GB/s of bandwidth, whereas eight I/Os operating at 8 Gbps provide only 8 GB/s. The example bank-count comparison is 128 versus 32. At the same time, HBM performance is not without resource costs: the combination of architectural design, advanced packaging, and manufacturing complexity means that delivering equivalent HBM3E capacity consumes approximately three times as much silicon as DDR5. Consequently, HBM scaling is not merely a DRAM process issue; it also depends on coordinated optimization of packaging technology, I/O circuitry, interposer processes, and CMOS processes. The report summarizes AI memory requirements in four areas: performance, capacity, scalability, and low power consumption. Performance encompasses not only speed and bandwidth but also reliability and fault tolerance. Capacity is required to hold model parameters, files, and caches. Low power consumption affects energy efficiency and the deployment of AI edge devices. As higher-speed I/O and larger system-in-package designs emerge, future innovation directions include memory-optimized SERDES D2D PHYs, co-packaged optics (CPO), larger CoWoS-L- or CoWoS-R-type SiPs, and glass substrates. These technologies expand connectivity and reduce losses in high-speed links but also increase the difficulty of system co-design. Chip-package interaction is another key challenge. Complex, heterogeneously integrated HBM devices use multiple materials whose mismatched coefficients of thermal expansion can generate thermomechanical stress, affecting structural integrity and long-term reliability. Reliability, availability, and serviceability also directly affect large-scale training. The Llama 3 405B pretraining record cited in the report covers 54 days, during which approximately 78% of unexpected interruptions were attributed to confirmed or suspected hardware issues. These included 148 failed-GPU incidents, accounting for 30.1%; 72 GPU HBM3 memory incidents, accounting for 17.2%; 54 software defects, accounting for 12.9%; 35 network switch or cable incidents, accounting for 8.4%; 32 unplanned host maintenance incidents, accounting for 7.6%; 19 GPU SRAM memory incidents, accounting for 4.5%; and 17 GPU system processor incidents, accounting for 4.1%. These data show that AI cluster performance depends not only on peak bandwidth but also on the stability of memory and computing hardware during prolonged operation. To address reliability issues, HBM3 began providing two levels of ECC coverage. System-level coverage provides 16 metadata bits for each 256-bit access, enabling the system to apply CRC or ECC. Die-level coverage implements symbol-based Reed-Solomon ECC within the DRAM. The report argues that such end-to-end protection is an important component enabling HBM to support continuous training and high-availability computing. Thermal management will become more difficult as high-speed D2D interconnects, base-die functionality, DRAM die activity, and stack height increase. The report identifies liquid cooling and hybrid bonding as important response measures and further notes that advanced packaging must simultaneously advance areal-density I/O scaling, including micro-bumps, pads, and TSVs; lateral I/O scaling, including finer line widths and spacing; array and CMOS partitioning; and hotspot and power-delivery-network management. Stacking and bonding routes include CoS, CoW, C2C, C2W, W2W, micro-bumps, fusion bonding, and hybrid bonding, indicating that future HBM performance gains will depend on close collaboration across the memory, logic, packaging, thermal-management, and manufacturing ecosystems. The report concludes by noting that ChatGPT reached 100 million users in only two months to illustrate the speed at which AI applications are proliferating, and on this basis emphasizes that the industry is undergoing an unprecedented phase of acceleration. Its core conclusion is that HBM has become a foundational component of AI accelerated computing, but continued increases in bandwidth and capacity cannot rely solely on upgrades to individual memory technologies. Challenges involving high-speed interconnects, package size, material stress, thermal density, error correction, and system serviceability must also be addressed in parallel.

Analysis framework

The report proceeds in the following sequence: AI scaling requirements, memory bottlenecks, HBM operating principles, generational specification evolution, system-level comparisons, packaging and interconnects, and reliability and thermal-management challenges. It first uses large-model scaling laws and the Roofline model to explain why bandwidth limits performance, then quantifies the differences using specification tables for HBM1 through HBM4, examples of DDR5 and HBM systems, and root-cause data for training interruptions. Finally, it proposes the technology directions required for continued scaling from the perspectives of chips, packaging, materials, thermal management, and error correction.

Methodology notes

• (Method Outside the Taxonomy)

  Roofline Performance Model

  This model links attainable computing performance to arithmetic intensity and is used to distinguish whether workloads are constrained by memory bandwidth or computing capability. The report uses it to explain how HBM raises the bandwidth ceiling and improves the performance of memory-intensive AI tasks.

• (Method Outside the Taxonomy)

  Large-Model Scaling Laws

  The report cites the empirical principle that model size, dataset size, and training compute must scale in tandem to explain how memory becomes a bottleneck to further AI performance improvements if it is not upgraded alongside computing capability.

• (Method Outside the Taxonomy)

  Generational Specification and System Architecture Comparison

  The report compares the channels, I/Os, rates, bandwidth, density, and capacity of HBM1 through HBM4 and places an eight-channel DDR5 system alongside an eight-stack HBM3 system to demonstrate their differing capacity and bandwidth priorities.

• (Method Outside the Taxonomy)

  Root-Cause Classification of Unexpected Interruptions

  The report categorizes interruptions during prolonged large-model pretraining by GPU, HBM3 memory, software, network, host maintenance, and other causes to illustrate how hardware reliability affects the effective operating time of AI clusters.

Key data

• HBM Generational TimelineHBM1 2014; HBM2 2018; HBM2E 2020; HBM3 2022; HBM3E 2024; HBM4 2026Product evolution milestones presented in the report
• HBM Nominal Bandwidth128, 307, 460, 819, 1024, 2800 GB/sCorresponding respectively to HBM1, HBM2, HBM2E, HBM3, HBM3E, and HBM4
• HBM Nominal Data Rate1, 2.4, 3.6, 6.4, 8, 11 GbpsCorresponding respectively to HBM1 through HBM4
• HBM4 Channels and I/O32 channels, 64 pseudo-channels, and 2048 data I/OsHBM3E has 16 channels, 32 pseudo-channels, and 1024 data I/Os, respectively
• DDR5 Example System307 GB/s theoretical bandwidth and 1 TB capacityEight channels, 4800 MT/s, and two 32-bit subchannels per channel; two ranks per channel and 64 GB per DIMM
• HBM3 Example System5.3 TB/s theoretical bandwidth and 192 GB capacityEight stacks, 5.2 Gbps, 16 channels per stack, and 24 GB capacity per stack
• HBM3E Silicon ConsumptionApproximately three times that of DDR5The report attributes this to the combined overhead of architectural design, advanced packaging, and manufacturing complexity
• Hardware Share of Unexpected Training InterruptionsApproximately 78%Attributed to confirmed or suspected hardware issues during 54 days of Llama 3 405B pretraining
• Major Interruption CategoriesFailed GPUs: 148 incidents, 30.1%; GPU HBM3 memory: 72 incidents, 17.2%; software defects: 54 incidents, 12.9%The top three root-cause categories of unexpected interruptions presented in the report
• HBM3 System-Level Error-Correction Metadata16 metadata bits configured for each 256-bit accessThe system can use CRC or ECC; die-level Reed-Solomon ECC is also provided
• ChatGPT User Adoption SpeedReached 100 million users in 2 monthsUsed in the report to illustrate the speed of AI application adoption

Impact & implications

The report argues that AI system design is shifting from a singular pursuit of peak GPU compute performance toward coordinated optimization of computing, memory, interconnects, and packaging. HBM can significantly increase data-delivery speed, but higher bandwidth and capacity also increase silicon consumption, I/O density, stack height, and thermal load. Future performance scaling therefore depends on whether advanced packaging, high-speed D2D, liquid cooling, hybrid bonding, ECC, and cross-industry collaboration can advance in tandem.

Risks

• If memory bandwidth cannot scale in tandem with model size and computing capability, processors will be unable to fully utilize their computing power while waiting for data.
• Mismatched coefficients of thermal expansion among different materials in HBM heterogeneous packaging can generate thermomechanical stress, creating chip-package interaction risks.
• Increases in high-speed interconnects, base-die functionality, DRAM activity, and stack height will raise thermal density and make heat dissipation more difficult.
• Prolonged AI training is highly sensitive to hardware reliability; approximately 78% of unexpected interruptions in the cited 54-day training record were associated with confirmed or suspected hardware issues.
• HBM3E capacity requires approximately three times as much silicon as DDR5, while advanced design, packaging, and manufacturing complexity create resource and production challenges.

What to watch

• Whether HBM4's 32 channels, 64 pseudo-channels, 2048 I/Os, and 2800 GB/s nominal bandwidth can support the next stage of AI workloads.
• Progress in memory-optimized SERDES D2D PHYs, co-packaged optics, and larger SiP interposers.
• Improvements from liquid cooling and hybrid bonding in addressing thermal-management issues caused by greater DRAM activity and taller stacks.
• The evolution of advanced packaging technologies such as micro-bumps, TSVs, line-width and spacing scaling, and glass substrates.
• Improvements in AI training reliability and serviceability from system-level CRC or ECC and die-level Reed-Solomon ECC.