Global AI memory Report Interpretation

Overview

This primer builds a framework for understanding AI memory requirements and the technologies seeking to address them. Bernstein argues that AI does not create one uniform memory market: the workload and location of data determine whether high bandwidth, low latency, capacity, cost or shared access matters most.

Core views

Bernstein starts with the premise that AI workloads impose different demands on memory. Training is highly compute-intensive and often bandwidth-constrained, making HBM critical, but large datasets must be prepared and held in slower, cheaper storage, then cached and staged in faster tiers. Because training can run for months, checkpoints are needed to avoid restarting after hardware or software failure. The report therefore views system DRAM, local SSD and networked storage as essential alongside HBM rather than as substitutes for it. For transformer inference, the report separates prefill from decode. Prefill processes the prompt and produces the first token through large matrix-matrix operations; it is generally compute-bound, so accelerator performance and HBM matter relatively more. Bernstein identifies time to first token (TTFT) as the key metric: around 0.5 seconds or less is described as ideal for chatbots, 1 second as acceptable, and more than 2 seconds as potentially tolerable for large tasks but likely to frustrate users in many settings. Decode generates tokens autoregressively. Previous tokens are retained as key-value pairs to avoid recomputation, causing KV-cache requirements to grow linearly with token count. Unlike read-only model weights, KV cache is updated with every generated token, is user-specific and grows with concurrent-user count. Bernstein therefore characterizes decode as memory-bound: the combined growth in context length and concurrency can make KV cache larger than model weights, constrain context windows and user capacity, and ultimately influence the scale and revenue capacity of an AI deployment. Time per output token (TPOT) is the relevant performance metric; the report cites roughly 50ms, or 20 tokens per second, as potentially acceptable for text chat, versus less than 10ms for demanding real-time voice, coding and agentic applications. RAG adds a different storage profile. Database generation digests large volumes of documents, code or webpages into vector databases and searchable indexes. It needs large storage capacity—preferably SSD rather than HDD for faster access—and substantial system DRAM for index generation, while HBM is less important; it is also an offline, one-off workload independent of users or tokens. During search, query embedding can use HBM quickly, but finding the nearest match usually makes system DRAM more important than HBM or SSD. Retrieved data then returns to the ordinary prefill-and-decode inference path. Agentic workflows compound these requirements because they use iterative planning, tools, external environments and intermediate-state retention. They also add demand for CPUs and conventional server memory, while outputs passed between agents repeatedly create further prefill, decode and KV-cache load. The report organizes these needs into an architectural memory hierarchy. At the top is on-chip or G0 memory, generally SRAM embedded with the accelerator for immediate, latency-critical work. G1 HBM is placed directly below it, co-packaged with the accelerator for high bandwidth and low latency. G2 system memory is higher-capacity but slower DRAM managed by CPUs, although certain GPUs can access it directly. CXL is classified as G2.5 because it pools physically separate memories logically and can extend from DRAM to NAND products made more DRAM-like through caching. NVIDIA's Storage Next is treated as G2.6: a GPU-centric initiative seeking NAND's cost and capacity with more DRAM-like latency, IOPS and access granularity, often through DRAM/SRAM-plus-NAND hybrids. Below this are G3 local SSD, which expands node-local capacity but is not shared across a pod, and G3.5 CMX, NVIDIA's context-storage tier aimed at KV cache. CMX-compliant SSDs sit in data nodes and can be accessed by multiple compute nodes through Ethernet, Spectrum-X switching and BlueField-4 DPUs, with GPUs able to access the tier without CPUs. Bernstein describes STX as NVIDIA's reference architecture for integrating compatible CMX storage and ODM solutions. G4 shared storage consists of SSD, HDD or tape for durable data outside the immediate critical path; the report says HDD remains suitable for low-cost, infrequently accessed cold data and is benefiting from expanding data-storage and KV-cache-overflow needs, while tape may also see demand as NAND and HDD capacity prove insufficient. The hardware hierarchy explains the cost, latency and technology trade-offs behind those tiers. SRAM is faster but uses more silicon area than DRAM; it is usually embedded in logic dies for cache, with L1 closest to compute and smallest in capacity. Embedded DRAM can compete with SRAM in larger caches but has limited market share. HBM is DRAM and shares wafer capacity with conventional DRAM, while conventional DRAM can also participate in system-memory and CXL tiers. Bernstein notes that DRAM technology progression remains largely driven by lithographic scaling and argues that EUV is necessary for advanced DRAM; it sees China's lack of EUV as a long-term competitive limitation for CXMT. The report describes 4F² DRAM as a one-time density improvement by relocating peripheral circuitry, whereas 3D DRAM could offer a longer scaling runway but remains more technically challenging and years from commercialization. Several innovations seek to improve high-performance memory but carry distinct implementation risks. Samsung's zHBM would place HBM atop the accelerator to shorten connections, but Bernstein questions whether DRAM can tolerate processor heat and whether hybrid-bonding yield and cost are commercially ready. NVIDIA's NVHBM moves memory-controller functions to an NVIDIA-designed base die, which could reduce accelerator cost or free die area for compute while enhancing bandwidth and power efficiency. However, Bernstein argues that this standardizes away part of memory suppliers' base-die differentiation and shifts manufacturing value toward foundries, most likely TSMC. Intel's XBM is presented as an emerging back-end-of-line DRAM concept that may coexist with logic and potentially support compute-in-memory, but manufacturing compatibility and the eventual producer remain uncertain. Intel's ZAM seeks better thermal dissipation by orienting DRAM dies sideways; its stated practical-use target is FY2029, with wireless data transmission among the reoriented dies a challenge. Qualcomm's HBC instead accepts lower performance than HBM to avoid CoWoS and interposer costs, using LPDDR for lower power; Qualcomm targets fiscal 2027 for generation one and 2028 for generation two. Module innovations such as MRDIMM, SOCAMM2 and LPCAMM2 aim to raise system-memory performance. MRDIMM adds interface chips to increase bandwidth, while SOCAMM2 uses LPDDR, a 128-bit bus and a lower profile to reduce power, increase bandwidth per module and improve airflow in AI servers. PIM combines logic and memory to reduce data traffic, a central computing bottleneck, but Bernstein emphasizes that adoption remains limited because it departs from the prevailing separation of processing and memory and requires changes across processors, software, networking and the established logic-versus-memory supply chain. For storage-class memory, Bernstein places technologies between DRAM and NAND. MRAM, PCM and RRAM use mechanisms fundamentally different from DRAM and NAND, while enhanced NAND approaches seek higher IOPS, lower latency and finer access granularity through caching or cell-mode trade-offs. KIOXIA's XL-FLASH uses SLC and later MLC to support high-IOPS GP-series SSDs; the report cites up to 10 million IOPS this year and more than 100 million in 2027. Samsung's Z-NAND/Z-SSD likewise targets ultra-low latency. Bernstein maps SLC/MLC-based products to Storage Next-type use cases, TLC to CMX products that balance performance and cost, and QLC to large-capacity products competing with HDD. Micron's XTR SSD is cited as a pSLC example, using only part of TLC-capable cells for one bit to gain SLC-like endurance and speed. HBF, led by SanDisk and SK hynix, is Bernstein's key NAND-based attempt to complement HBM: it seeks HBM-like bandwidth with much greater capacity and lower cost per bit, and is placed conceptually at a G1.5 tier. The report stresses that achieving this would require NAND to leap several levels of the hardware hierarchy, making the technical hurdle high. Samsung's zNAND-O is instead positioned for on-device AI, using TLC for large capacity and advanced packaging near the processor. Overall, Bernstein's framework implies that AI growth broadens the addressable memory and storage stack, but the economic benefit will vary by workload, tier, implementation feasibility and where technology integration captures value.

Analysis framework

Bernstein first links each major AI workload to its performance bottleneck and memory needs, then places technologies into an architectural hierarchy based on where memory sits in the system and a hardware hierarchy based on cell structure and storage mechanism. It compares latency, bandwidth, capacity, cost, packaging and supply-chain implications to explain which innovations may address each bottleneck.

Methodology notes

• Industry AnalysisUpstream-Midstream-Downstream Transmission

  AI workload-to-memory-tier mapping

  The report traces how workload requirements flow through accelerators, DRAM, NAND, SSD, storage, packaging and suppliers, showing why different parts of the memory chain may benefit or face pressure.

• Industry AnalysisCost curve analysis

  Performance, capacity and cost trade-off across memory types

  The report compares high-speed, high-cost memory with lower-cost, higher-capacity alternatives to explain the rationale for HBF, CXL, Storage Next and storage-class-memory products.

Asset mapping & comparison

Structured mapping from thesis to named assets (strengths, weaknesses, peers, risks).

• Samsung Electronics (005930.KS)

  Covered memory supplier; the report discusses zHBM, Z-NAND and CXL memory products.

  Strengths

  Developing zHBM, CMM-H and Z-NAND/Z-SSD offerings across emerging AI memory tiers.

  Weaknesses

  Bernstein questions zHBM thermal management and hybrid-bonding yield and cost readiness.

  Comparison

  Competes with other HBM and DRAM suppliers; NVHBM may reduce base-die differentiation.

  Risks

  Commercial readiness of zHBM and technology-scaling requirements.

• SK hynix (000660.KS)

  Covered memory supplier and HBF co-leader.

  Strengths

  Participates in HBM and leads HBF with SanDisk.

  Comparison

  HBF aims to complement HBM using lower-cost, higher-capacity NAND.

  Risks

  HBF requires a substantial performance leap from NAND.

• Micron (MU)

  Covered memory supplier; cited for pSLC-based XTR SSD and DRAM module technology.

  Strengths

  Participation in advanced DRAM and NAND-based SSD approaches.

  Comparison

  Competes with Samsung and SK hynix in AI memory; NVHBM may standardize base-die differentiation.

  Risks

  Potential value shift from memory suppliers to NVIDIA and foundries under NVHBM.

• KIOXIA (285A.JP)

  Covered NAND and SSD supplier; its GP-series SSD illustrates Storage Next and XL-FLASH.

  Strengths

  XL-FLASH supports high-IOPS SSD products for storage-class-memory use cases.

  Comparison

  Its SLC/MLC-based approach targets faster tiers, while TLC and QLC address CMX and capacity-oriented storage.

  Risks

  Product economics require capacity and cost trade-offs against higher-performance cell modes.

• SanDisk (SNDK)

  Covered storage supplier and HBF co-leader.

  Strengths

  Leads HBF with SK hynix, targeting higher capacity and lower cost per bit than HBM.

  Comparison

  HBF is positioned between HBM and lower-tier NAND storage.

  Risks

  The report identifies a high technological hurdle to achieving sufficient HBF performance.

• Seagate (STX)

  Covered storage supplier linked to shared-storage demand.

  Strengths

  HDD remains a low-cost option for cold data and potentially KV-cache overflow.

  Weaknesses

  HDD is outside the immediate critical path and is much slower than flash memory.

  Comparison

  Competes with higher-capacity SSD and tape in G4 shared storage.

  Risks

  Its role depends on demand for low-cost capacity rather than latency-sensitive workloads.

• Western Digital (WDC)

  Covered storage supplier linked to SSD and shared-storage demand.

  Strengths

  Exposure to expanding AI data-storage needs across flash and storage tiers.

  Comparison

  Storage technologies are differentiated by latency, IOPS, endurance, capacity and cost.

Key data

• Ideal chatbot TTFT0.5 second or lowerBernstein cites this as ideal time to first token.
• Acceptable chatbot TTFT1 secondTTFT above 2 seconds may lead to user frustration or prompt abandonment in many cases.
• Text-chat TPOT50ms, or 20 tokens per secondA potentially acceptable output speed for text chat.
• High-end TPOTLess than 10msRelevant to real-time voice, coding and agentic workloads.
• KIOXIA GP-series SSD performanceUp to 10M IOPS this year and 100M+ in 2027Cited for XL-FLASH-based, high-IOPS storage-class-memory positioning.
• Qualcomm HBC roadmapGen-1 in fiscal 2027; Gen-2 in 2028A lower-cost, LPDDR-based alternative that compromises on performance versus HBM.
• Intel ZAM targetFY2029Stated target for practical use of the thermally oriented DRAM packaging concept.

Impact & implications

The report indicates that AI infrastructure demand extends beyond HBM: decode-stage KV cache, RAG databases and agentic workflows create roles for DRAM, SSD, HDD, tape, memory pooling and new intermediate tiers. It also suggests that system architecture and integration choices can redistribute value among memory makers, GPU designers, foundries, packaging providers and storage suppliers.

Risks

• HBF faces a high technical hurdle because NAND must make a large performance leap to complement HBM.
• zHBM may face thermal, hybrid-bonding yield and cost challenges before commercial production.
• PIM adoption remains limited because it requires broad changes to processors, software, networking and supply chains.
• 3D DRAM is more technologically challenging and is still years from commercialization.
• NVHBM could reduce the differentiation and value captured by memory suppliers' base-die designs.

What to watch

• KV-cache growth relative to model weights as context windows and concurrent-user counts rise.
• Whether HBM capacity can expand affordably or workloads shift toward DRAM, NAND and new intermediate tiers.
• Progress in CXL, Storage Next, CMX and GPU-direct storage implementations.
• Commercial execution of HBF, zHBM, NVHBM, XBM, ZAM and HBC.
• Adoption of high-IOPS NAND technologies such as XL-FLASH, Z-NAND and pSLC products.