Raptor Uses 3D-DRAM to Break Through the Memory Bandwidth and Energy Bottlenecks of Generative Inference

Overview

Starting from the tension among capacity, bandwidth, and power consumption in generative inference, the report explains how d-Matrix Raptor directly stacks logic dies on DRAM and jointly designs memory mapping, data transmission, thermal refresh, error correction, and redundancy mechanisms to use 3D-DRAM for low-latency, long-context inference.

Core views

The scale of data in generative inference is expanding from both directions simultaneously: model weights continue to grow, while KV cache grows with context length and the number of concurrent users. The report gives the example that, with 64 users and a context length of 1 million per user, the KV cache is approximately 935 GB. The prefill stage of inference is generally constrained by compute throughput, whereas autoregressive decoding is generally constrained by memory bandwidth; in low-latency, small-batch scenarios, decoding accounts for the critical runtime. High GQA and speculative decoding may make attention computation compute-bound, but MoE may remain memory-bandwidth-bound even with moderate batch sizes and speculative decoding, so capacity and bandwidth must scale simultaneously. The report argues that the two existing types of memory each have clear shortcomings. On-chip SRAM offers sub-nanosecond latency; an example pair of cards can provide approximately 300 TB/s of bandwidth, 4 GB of capacity, approximately 1 ns of latency, and approximately 0.5 pJ/bit of I/O energy consumption. However, its 6T storage cell is approximately 10 times the area of a DRAM 1T1C cell, bit-cell area from the N5 to N2 nodes remains approximately 0.021 µm², leakage for gigabyte-scale capacity can reach tens of watts, practical capacity is approximately 4 GB, and cost is approximately 100 times that of DRAM. HBM offers high capacity but is constrained by package-edge space, stack count, pin speed, and I/O width; the report estimates that the practical bandwidth ceiling of an HBM4 system is approximately 20 TB/s. If HBM were pushed to 100 TB/s at 2.4 pJ/bit, memory power alone would require 1.92 kW, excluding interconnect power. Raptor's core approach is to face-to-face stack logic and DRAM, replacing centimeter-scale interposer paths with PHYs with millimeter-scale vertical connections. The report gives 3D vertical I/O energy consumption of approximately 0.3–0.4 pJ/bit, roughly an order of magnitude lower than HBM; the specific interface estimate is 0.37 pJ/bit, versus approximately 2–3 pJ/bit for HBM4. Raptor uses a TSMC N4/N4P logic top layer, a 3D-DRAM bottom layer, and face-to-face stacking with a 36 µm pitch. Each chiplet provides 4 GB of capacity, 12.5 TB/s of bandwidth, and 840 DRAM microbanks; eight chiplets form a 32 GB, 100 TB/s card, four cards form a 128 GB, 400 TB/s system, and 18 trays totaling 72 cards reach 2.3 TB of capacity and 7.2 PB/s of bandwidth. The model-loading estimates assume 4-bit weights, 8-bit KV cache, and a 72-card scaling domain. GLM 5.2 contains 744 billion total parameters and 40 billion active parameters, requiring 44.7 GB of KV cache per user at a context length of 1 million; Kimi K3 contains 2.8 trillion total parameters and approximately 102 billion active parameters, requiring 11.7 GB of KV cache per user. The report states that 72 Raptor cards, each with 32 GB, can accommodate Kimi K3 at a context length of 1 million, with disaggregated deployment and multiple racks enabling further scaling; its conclusion further states that a single Raptor rack serving an approximately 3-trillion-parameter model at a context length of 1 million can sustainably deliver approximately 1,000 TPS per user. High bandwidth can translate into effective performance only when compute, memory, and interconnect are codesigned. Through sharding and collective communication, Raptor must coordinate approximately 8,000 virtual devices across a hierarchical network, compared with approximately 72 devices in a typical GPU system; low latency is therefore necessary to avoid limitations imposed by Amdahl's law. The report adopts an in-package full-mesh topology, using virtual diagonal links to reduce in-package latency while avoiding the increased distance, D2D power consumption, and package layer count associated with physical diagonals. Intra-card and inter-card protocols use device-initiated, unidirectional push transfers that require no acknowledgment from the source card, supporting multiple parallel short-message transfers. The first specific integration challenge is bank-to-channel mapping. Each compute-engine access requires 128 B of data, while each bank column access can return only 32 B, so each channel theoretically requires four banks. A chiplet has 256 channels and the DRAM has 840 banks; after subtracting 72 spare banks, only 768 usable banks remain, or three per channel. Directly performing two accesses reads 192 B for a single 128 B flit, causing 33% overfetch and wasting approximately 33 TB/s. A simple column-staggering scheme requires a 192 B shift buffer and increases the difficulty of timing closure and verification. Raptor uses “stream blocking”: the fourth 32 B partial access is shared across three flits, so that 4×96 B of input exactly forms 3×128 B of output, reducing overfetch to 0%, fully utilizing each column access, and reclaiming approximately 33 TB/s of bandwidth. The second challenge is I/O power consumption. At 100 TB/s and 0.37 pJ/bit, I/O itself consumes approximately 296 W. Traditional HBM data bus inversion relies on multi-cycle bursts, advance knowledge of the complete burst, and dedicated DBI pins; Raptor's 3D-DRAM interface uses single-cycle, 256-bit transfers without bursts or sideband pins, so this mechanism cannot be directly reused. Its “stream flipping” scheme compares adjacent flits by channel: during writes, it inverts the current flit when inversion results in fewer bit transitions, and during reads, it restores the data using a 1-bit tag stored together with ECC. The report states that the scheme incurs only 0.8% overhead, requires no PHY modifications, and can reduce I/O power by 20%. The third challenge is reliability at a junction temperature of 105°C. The standard DRAM retention time is 32 ms at 85°C but falls to 4 ms at 105°C, implying an eightfold increase in refresh frequency; high temperatures also increase soft errors, while the yield risks arising from 840 banks, channel asymmetry caused by a single failed bank, and contention among ECC, scrubbing, and refresh for throughput must all be addressed simultaneously. Raptor leverages its deep-bank structure, which has 16–32 times fewer rows, so the bandwidth cost of 4 ms refresh is only 1.37%, allowing it to sustain approximately 100 TB/s; the final eight column locations store interleaved [132,128] Reed–Solomon error-correcting codes and DBI bits. Seventy-two online redundant microbanks use a two-level bank chain to handle up to two failures at arbitrary locations, with spare banks filling in, thereby preserving symmetric channel width while adding negligible routing cost. In a comparison based on silicon area and effective bandwidth, Raptor's capacity density is 11.4 MB/mm², below 21.9 for HBM4 24 Gb, 26.3 for HBM4 32 Gb, and 21.9 MB/mm² for Rubin R200; however, its bandwidth density is 32.6 GB/s/mm², significantly above the respective 1.67, 1.51, and 1.39 GB/s/mm². Raptor's power consumption per GB/s is 2.96 mW, versus 40.0 mW for all comparison targets; this comparison assumes effective bandwidth utilization of 83% for Raptor and 85% for Rubin. The report therefore emphasizes that low-latency inference does not necessarily require maximum capacity per card, but is better suited to moderate capacity combined with extremely high bandwidth. The early-silicon materials further state that across models including Llama-3.1 70B, DeepSeek-V3, Kimi K2, GPT-OS, Whisper, and Canary, Raptor 3D-DRAM delivers 4.71x the throughput of HBM-based solutions and 2.44x that of SRAM-based solutions, while also being less sensitive to network latency and bandwidth.

Analysis framework

The report first decomposes the bottlenecks of generative inference into the prefill and decoding stages, then compares SRAM, HBM, and 3D-DRAM in terms of capacity, bandwidth, latency, I/O energy consumption, and packaging constraints. It subsequently estimates model capacity and bandwidth from the chiplet level through cards, trays, and a 72-card rack, analyzing bank mapping, I/O power consumption, high-temperature refresh, ECC, and redundancy issues one by one. Finally, it validates the design trade-offs by comparing effective bandwidth utilization, metrics per unit of silicon area, power consumption per unit of bandwidth, and multi-model throughput.

Methodology notes

• Industry/Sector Analysis FrameworkSupply-demand framework

  Matching Inference Data Requirements with Memory Capabilities

  The report treats growing model weights and KV cache as demand for capacity and bandwidth, then compares this demand with the capacity, bandwidth, and power consumption that SRAM, HBM, and 3D-DRAM can provide to determine which memory architecture is better suited to long-context inference.

• (Out-of-Vocabulary Method)

  Prefill–Decoding Bottleneck Decomposition

  The report divides the inference process into a prefill stage, which is generally limited by compute throughput, and a decoding stage, which is generally limited by memory bandwidth, to explain which portion of runtime high-bandwidth 3D-DRAM primarily addresses.

• (Out-of-Vocabulary Method)

  Hardware, Architecture, and Workload Codesign

  Rather than treating 3D-DRAM as external storage, the report jointly designs bank mapping, data flow, interconnect protocols, refresh, error correction, and redundancy so that the raw bandwidth provided by physical stacking can be translated into effective inference throughput.

• (Out-of-Vocabulary Method)

  Normalized Comparison of Silicon Area and Effective Bandwidth

  The report compares Raptor, HBM4, and Rubin R200 by capacity per square millimeter, bandwidth per square millimeter, and power consumption per GB/s, while incorporating effective bandwidth utilization of 83% for Raptor and 85% for Rubin to avoid comparing nominal specifications alone.

• (Out-of-Vocabulary Method)

  Amdahl's Law

  The report uses this law to explain that even if memory bandwidth increases substantially, the remaining serial or communication components will still limit overall system acceleration if intra-card or inter-card communication latency does not decrease accordingly.

Asset mapping & comparison

Structured mapping from thesis to named assets (strengths, weaknesses, peers, risks).

• d-Matrix Raptor 3D-DRAM Generative Inference Accelerator

  The report positions it as a 3D-stacked integrated logic-memory platform for long-context, low-latency generative inference.

  Strengths

  100 TB/s bandwidth per card, low I/O energy consumption, scalability from chiplets to a 72-card rack, and codesigned features including stream blocking, stream flipping, thermal-aware refresh, ECC, and bank chains.

  Weaknesses

  Per-card capacity is 32 GB, and capacity density by silicon area is 11.4 MB/mm², below the HBM4 and Rubin R200 figures presented in the report; the system must also coordinate approximately 8,000 virtual devices.

  Comparison

  The report gives Raptor a bandwidth density of 32.6 GB/s/mm² and power consumption per unit of bandwidth of 2.96 mW/GB/s, versus 1.39–1.67 GB/s/mm² and 40.0 mW/GB/s, respectively, for comparison targets; cross-model throughput is 4.71x and 2.44x higher than HBM- and SRAM-based solutions, respectively.

  Risks

  Integration risks include thermal management and power delivery for 3D stacking, microbump yield, bank failures, refresh and soft errors at 105°C, signal integrity across wide parallel buses, and multi-chiplet and multi-card communication latency.

Key data

• KV Cache for 64 Users at a Context Length of 1 MillionApproximately 935 GBIllustrates how long context and concurrent users jointly amplify memory-capacity requirements.
• Example SRAM Specifications300 TB/s, 4 GB, approximately 1 ns, approximately 0.5 pJ/bitOutstanding bandwidth and latency, but capacity, leakage, and cost constrain scaling.
• Practical HBM4 Bandwidth CeilingApproximately 20 TB/sConstrained by package-edge space, stack count, and I/O scaling.
• HBM Memory Power at 100 TB/s1.92 kWCalculated at 2.4 pJ/bit and excluding interconnect power.
• Raptor Single Card32 GB, 100 TB/sConsists of eight 3D-DRAM chiplets, each providing 4 GB and 12.5 TB/s.
• Raptor 72-Card Rack2.3 TB, 7.2 PB/sConsists of 18 trays and 72 cards.
• GLM 5.2 Model Estimate744B total parameters, 40B active parameters, 44.7 GB KV/userAssumes a context length of 1 million.
• Kimi K3 Model Estimate2.8T total parameters, approximately 102B active parameters, 11.7 GB KV/userSeventy-two 32 GB cards can accommodate the model at a context length of 1 million.
• Original Bank-Mapping Overfetch33%, approximately 33 TB/sWith only three banks per channel, directly reading a 128 B flit requires fetching 192 B.
• Stream Blocking Effect0% overfetchAfter sharing partial accesses, 384 B of input corresponds to 384 B of effective output.
• 3D-DRAM I/O Power at 100 TB/s296 WCalculated at 0.37 pJ/bit.
• Stream Flipping Effect20% power savings, 0.8% overheadThe tag is stored together with ECC, requiring no additional PHY sideband pins.
• Refresh Requirement at 105°C32 ms reduced to 4 ms, with refresh frequency increasing eightfoldThe deep-bank design limits the bandwidth cost to 1.37%.
• Bandwidth DensityRaptor 32.6 GB/s/mm²; HBM4 and Rubin R200 at 1.67, 1.51, and 1.39 GB/s/mm²Compared on an effective-bandwidth basis.
• Power Consumption per Unit of BandwidthRaptor 2.96 mW/GB/s; comparison targets 40.0 mW/GB/sLower is better.
• Cross-Model Throughput Comparison4.71x higher than HBM and 2.44x higher than SRAMCovers the language-model and speech-model workloads listed in the report.

Impact & implications

The report argues that competition in generative inference accelerators is shifting from simply adding compute units toward simultaneously addressing data capacity, bandwidth, energy consumption, and communication latency. Raptor sacrifices some capacity density per unit area in exchange for bandwidth density far above HBM and lower power consumption per unit of bandwidth; if its bank mapping, transmission encoding, high-temperature refresh, ECC, and redundancy designs perform as indicated by early silicon results, this combination of moderate per-card capacity and extremely high bandwidth could improve long-context, low-latency decoding and support scaling trillion-parameter models to a 72-card rack.

Risks

• Accelerator heat must dissipate through the temperature-sensitive DRAM stack, while hundreds of watts of power must be delivered through TSVs, potentially creating junction-temperature, voltage-drop, and thermal constraints.
• The 840 banks and 36 µm microbumps increase yield and alignment risks; even a 1% failure rate could disable an entire channel or cause channel asymmetry.
• At 105°C, DRAM retention time falls from 32 ms to 4 ms and refresh frequency increases eightfold; poor coordination of soft errors, ECC, scrubbing, and refresh could reduce throughput.
• Wide parallel buses face crosstalk, simultaneous switching noise, clock distribution, and cross-stack skew issues.
• Sharding and collective communication across approximately 8,000 virtual devices impose demanding requirements on network topology, protocols, bandwidth, and low latency.
• Scaling to multiple stacked layers still faces routing, power-delivery, and thermal-management challenges, while the hybrid-bonding path must also advance toward pitches below 2 µm and eight-layer stacking.

What to watch

• Monitor validation of the 100 TB/s effective bandwidth, power consumption, and throughput results of early silicon across more real-world generative inference workloads.
• Monitor whether sharding, collective communication, and network latency in 72-card and multi-rack deployments can maintain the scaling efficiency described in the report.
• Monitor the long-term reliability of 4 ms refresh, interleaved ECC, scrubbing, and 72 spare banks in a 105°C environment.
• Monitor the hybrid-bonding roadmap toward pitches below 2 µm and eight-layer stacking, as well as progress in routing, power delivery, and thermal management for multilayer stacks.