Global AI memory Report Interpretation

Overview

This primer explains where different memory technologies fit in AI systems. Bernstein organizes the discussion around workload-specific requirements and two complementary hierarchies: an architectural hierarchy based on where memory sits in a system, and a hardware hierarchy based on storage mechanisms and packaging.

Core views

Bernstein begins with transformer-based large-language-model workloads because their memory requirements differ sharply by task. Training is both compute- and bandwidth-intensive, making HBM critical to model size and compute speed, but it also needs lower-cost capacity for datasets, faster tiers for pipeline staging, and checkpoint storage because training can run for months. The report therefore treats system DRAM, local SSD and networked storage as necessary complements to HBM rather than substitutes. Inference separates into prefill and decode. Prefill processes the prompt and produces the first token; it is generally compute-bound because large matrix operations keep GPUs highly utilized, so HBM inside the accelerator is comparatively important. Its key latency measure is time to first token (TTFT): the report cites about 0.5 seconds or less as ideal and one second as acceptable for chatbots, while more than two seconds can frustrate users except for large tasks. Decode produces tokens autoregressively. To avoid recomputation, models keep previous tokens in a KV cache. Model weights are static and can be shared, but KV-cache data changes with each token and differs by user, causing capacity needs to rise with both token count and concurrent users. Bernstein therefore describes decode as memory-bound and argues that KV cache can exceed model weights in large deployments, constraining context windows, concurrency, deployment scale and potentially revenue capacity. Time per output token (TPOT) is the key decode metric; the report cites roughly 50 ms for text chat and less than 10 ms for demanding real-time voice, coding and agentic applications. RAG adds a different storage pattern. Creating a vector database from large unstructured datasets needs substantial SSD or HDD capacity and system DRAM for index construction, with relatively lighter HBM use; Bernstein characterizes this offline preparation as a one-off cost independent of user count and token volume. During search, a query is rapidly converted to a vector, typically using HBM, but system DRAM usually does more of the nearest-match work than HBM or SSD. The retrieved content then re-enters normal prefill and decode. Agentic workflows compound the issue: multi-step agents need memory for plans, branches and intermediate states, generate additional conventional CPU-and-system-memory demand when using external tools, and repeatedly feed outputs into subsequent inference stages, increasing prefill, decode and KV-cache demand. The architectural hierarchy runs from very low-latency on-chip SRAM at the top, through HBM, system DRAM, CXL memory and Storage Next, to local SSD, CMX and shared SSD/HDD/tape storage. HBM is placed directly beneath on-chip memory because co-packaging with accelerators provides high bandwidth and low latency for active weights and KV cache. System DRAM offers more capacity but longer latency. CXL is intended to pool physically separate memory and can extend to NAND products made more DRAM-like through DRAM or SRAM caching. Storage Next is presented as an NVIDIA-led effort to shift memory management toward GPUs and use NAND’s lower cost and greater capacity while improving latency, IOPS and access granularity; Bernstein places it between DDR DRAM and local SSD. CMX is a shared, SSD-based tier optimized for KV cache across compute nodes, with NVIDIA’s STX reference architecture intended to support interoperability. Local SSD extends node-level capacity, while shared SSD, HDD and tape handle durable or less latency-sensitive data. The hardware hierarchy explains the technical trade-offs underneath those tiers. SRAM offers the lowest latency but requires more silicon area than DRAM, making it useful for on-chip caches. HBM is DRAM packaged next to the XPU, while conventional DRAM also serves system memory and can be used in CXL products. Bernstein highlights continuing DRAM scaling via smaller process nodes, notes that EUV is strategically important, and discusses 4F² and 3D DRAM as Chinese research paths; 4F² may offer a one-time density benefit, whereas 3D DRAM could have a longer scaling runway but is more technically difficult and still years from commercialization. The report surveys several HBM and DRAM innovations. Samsung’s zHBM puts HBM stacks on top of the XPU to shorten connections, but Bernstein questions whether the DRAM can withstand heat from the processor and whether hybrid-bonding yields and costs are commercially ready. NVIDIA’s NVHBM moves the base-die design to NVIDIA, potentially reducing XPU die cost, increasing compute area, improving bandwidth and lowering power, but it also standardizes away some memory-supplier differentiation and shifts manufacturing value toward foundries. Intel’s XBM and ZAM, Qualcomm’s HBC, and module formats including MRDIMM, SOCAMM2 and LPCAMM2 are presented as alternative routes to bandwidth, thermal or power improvements. HBC trades some performance for lower packaging cost by avoiding the CoWoS interposer and uses LPDDR; Qualcomm targets fiscal 2027 for generation one and 2028 for generation two. For NAND and storage-class memory, Bernstein positions enhanced NAND products between DRAM and ordinary SSDs. Z-NAND and XL-FLASH use SLC or MLC to sacrifice some capacity and cost advantage for lower latency and higher IOPS; KIOXIA’s GP-series SSD is cited at up to 10 million IOPS in 2026 and more than 100 million in 2027. The report assigns SLC/MLC-based XL-FLASH to Storage Next-type applications, TLC to CMX products that balance cost and performance, and QLC to high-capacity products competing with HDD. HBF, led by SanDisk and SK hynix, aims to complement HBM with NAND’s lower cost and higher capacity while approaching HBM bandwidth, but Bernstein emphasizes that bridging several hardware tiers is technically difficult. HDD remains suitable for low-cost, infrequently accessed cold data, and the report says booming storage and KV-cache-overflow demand is benefiting it; it also notes reported growth in tape needs amid insufficient NAND and HDD supply. Finally, PIM combines processing and memory on the same silicon to reduce data movement, a key computing bottleneck. Bernstein sees the potential benefit but stresses that adoption has been limited because PIM departs from the established separation of processing and memory and would require broad redesign of processors, software, networks and the supply chain.

Analysis framework

Bernstein first classifies AI workloads by whether compute, bandwidth, capacity or latency is the binding constraint. It then maps the relevant memory types into an architectural hierarchy based on system position and purpose, followed by a hardware hierarchy based on cell design, packaging and storage mechanism. The report uses this framework to explain the cost, capacity, bandwidth and latency trade-offs of established and emerging technologies.

Methodology notes

• Industry AnalysisUpstream-Midstream-Downstream Transmission

  AI memory architectural and hardware hierarchies

  The report maps memory from on-chip SRAM through HBM, DRAM, SSD and shared storage to show how workload demands flow through the AI memory supply chain.

• Industry AnalysisSupply-demand framework

  Workload-specific memory requirements

  The analysis distinguishes training, prefill, decode, RAG and agentic workflows to explain which memory resources become constrained and why.

Asset mapping & comparison

Structured mapping from thesis to named assets (strengths, weaknesses, peers, risks).

• Samsung Electronics (005930.KS)

  Covered memory supplier and developer of zHBM, CXL memory modules and Z-NAND.

  Strengths

  The report highlights zHBM, CMM-H hybrid CXL memory and Z-SSD products.

  Comparison

  zHBM seeks to advance beyond standard HBM by placing stacks on top of the XPU.

  Risks

  Bernstein questions zHBM thermal performance and the yield and cost readiness of hybrid bonding.

• SK hynix (000660.KS)

  Covered memory supplier and HBF participant.

  Strengths

  The report identifies SK hynix as a leader in HBF alongside SanDisk and as a source for HBM examples.

  Comparison

  HBF seeks to complement HBM with higher NAND-based capacity at lower cost.

  Risks

  HBF must overcome a substantial performance gap between NAND and HBM.

• Micron (MU)

  Covered memory supplier and participant in Storage Next.

  Strengths

  The report cites Micron’s XTR SSD as an example of pSLC and includes Micron among Storage Next collaborators.

  Weaknesses

  NVHBM may reduce memory suppliers’ base-die differentiation.

  Comparison

  Micron is listed with Samsung among suppliers that manufacture the base die in standard HBM.

  Risks

  Standardization of the NVHBM base die could shift value toward NVIDIA and foundries.

• KIOXIA (285A.T)

  Covered NAND and SSD supplier; its GP-series SSD is used as a Storage Next example.

  Strengths

  XL-FLASH GP-series SSDs are cited for super-high IOPS and placement in the Storage Next tier.

  Comparison

  Its SLC/MLC-based XL-FLASH prioritizes latency and IOPS over the density economics of TLC.

• SanDisk (SNDK)

  Covered NAND supplier and HBF leader.

  Strengths

  The report identifies SanDisk as an HBF leader and uses its HBF design to illustrate increased total memory capacity.

  Comparison

  HBF aims to complement rather than replace HBM using lower-cost, higher-capacity NAND.

  Risks

  The required NAND performance improvement to approach HBM bandwidth is a high technical hurdle.

• Seagate (STX)

  Covered storage company linked to the lower shared-storage tier.

  Comparison

  HDD is positioned below NAND in the memory hierarchy but remains useful for low-cost cold storage.

• Western Digital (WDC)

  Covered storage company linked to the lower shared-storage tier.

  Comparison

  The report describes HDD as a G4 shared-storage option for durable, less latency-sensitive data.

Key data

• Ideal chatbot TTFT0.5 seconds or lowerThe report cites this as ideal for time to first token; 1 second is acceptable.
• Potentially tolerable TTFT for large tasksMore than 2 secondsThe report says this may be tolerated for tasks such as digesting a 100-page PDF, but can frustrate users.
• Text-chat TPOTUp to 50 msIllustrative acceptable time per output token.
• High-end TPOTLess than 10 msIllustrative requirement for real-time voice, coding and agentic workloads.
• KIOXIA GP-series SSD IOPSUp to 10M in 2026; 100M+ in 2027Cited for XL-FLASH-based high-IOPS storage.
• Qualcomm HBC timingGen-1 in fiscal 2027; Gen-2 in 2028Company target cited by the report.

Impact & implications

Bernstein’s framework suggests that AI memory demand is not limited to HBM: expanding context windows, concurrent usage, RAG datasets and agentic workflows can increase demand across DRAM, SSD, HDD and new intermediate tiers. The report emphasizes that the technology and commercial value of each solution depend on where it improves the latency, bandwidth, capacity or cost bottleneck.

Risks

• HBF faces a high technological hurdle because NAND must bridge a large performance gap to complement HBM.
• zHBM may face thermal, hybrid-bonding yield and cost challenges before commercial production.
• PIM adoption has been limited because it requires major changes across processor, software, network and supply-chain architecture.
• 3D DRAM is technically challenging and may be years from commercialization.