HBF Is Not a Cheaper Version of HBM: Low-Bandwidth, Capacity-Oriented Workloads Present an Opportunity, While Dense and High-Batch Inference Could Become a Cost Trap

Overview

This Hot Chips 2026 tutorial examines the applicability boundaries of HBF in AI inference from a system architecture perspective. Its central conclusion is that HBF derives its value from high capacity rather than high bandwidth: it can improve small-batch MoE, sparse long-context KV, and local expert storage, but HBM remains the more economical choice for dense models, large batches, or deployments emphasizing rack throughput.

Core views

The report first places HBF on a memory technology landscape described by β and α: β represents cost per GB, while α represents the bandwidth available per unit of capacity. HBF occupies a low-β, low-α, capacity-oriented position and can provide approximately 8—16 times the capacity of HBM at the same cost, but it should not be viewed as a cheaper version of HBM. The report uses the formula “$mem = β·max(C, I·b/α)” to determine memory cost, where C is the amount of data that must be accommodated and I·b is the bandwidth requirement. What truly determines economics is not how many inexpensive GB are purchased, but the larger of the capacity constraint and the data-supply bandwidth constraint. Therefore, low $/GB does not automatically mean low $/token. Workload characteristics determine whether HBF falls within its effective range. MoE models can store vast expert weights in full, but each token activates only a small number of experts. Thus, at small batch size B and lower access intensity I, the model is constrained primarily by capacity rather than bandwidth, and low bandwidth may be sufficient. Dense models require broader weight reads; as batch size, access intensity, or I·b rises, HBF's insufficient bandwidth forces the system to provision more capacity to satisfy data-supply requirements, and beyond the crossover point HBM becomes cheaper instead. The report therefore limits HBF's advantage to the “small B, low I” MoE subregion, leaving Dense and high-B workloads to HBM. The deployment comparison includes all-HBM, all-HBF, and a hybrid “2×HBF＋6×HBM” configuration. All three can achieve the same cost but offer different capacity and bandwidth. In short-context, low-batch simulations, HBF can win on cost, but approximately 85% of its capacity remains idle; HBM, meanwhile, may charge for performance beyond the capacity wall. In long-context cases, more capacity is likewise not necessarily better: additional capacity has economic value only when I·b remains low. The hybrid configuration attempts to use HBM as a hot-expert cache, but expert popularity tends to flatten under mixed queries, so caching is effective only at low batch sizes or when similar queries are grouped for processing. The report further conducts a 72-GPU, rack-level, decode-centric simulation using the Kimi-K2 1T model at FP4. The primary context consists of 1 million input tokens and 1,000 output tokens, supplemented by 32k/8k and 1k/8k scenarios. Batch size per data-parallel instance ranges from 1—512; tensor parallelism is used to meet capacity requirements, data parallelism fills the rack, and cost is expressed as a ratio after amortizing capital and operating expenditures over three years. The all-HBM configuration is TP8·DP9, the all-HBF configuration is TP1·DP72, and the hybrid configuration is TP2·DP36. Capacity per data-parallel instance for the all-HBM, all-HBF, and hybrid configurations is 2.3 TB, 4.1 TB, and 2.5 TB, respectively; total rack capacity is 20.7 TB, 294.9 TB, and 89.3 TB, respectively, equivalent to 1 time, approximately 14 times, and 4.3 times. The capacity expansion comes with a bandwidth cost. Aggregate rack bandwidth is 1,584 TB/s for all-HBM and 922 TB/s, or approximately 0.6 times as much, for all-HBF; the hybrid configuration delivers 1,418→279 TB/s. Corresponding per-GPU bandwidth is 22.0 TB/s, 12.8 TB/s, and 19.7→3.9 TB/s, respectively. Under comparable rack cost and power consumption, the report concludes that HBM can always process more tokens. HBF is better suited to deployments constrained by single-machine or single-instance capacity because it can fit a model that would otherwise require multiple GPUs onto a single GPU. In other words, “rack-level cost per token” and “whether the model fits on one machine” lead to different memory choices. HBF's hardware characteristics also constrain how it can be used. The three grades listed in the report provide maximum user bandwidth of 0.384, 1.536, and 3.072 TB/s, respectively, with interconnect rates of 8, 16, and 32 GT/s; capacity configurations include 8-layer 256 GiB and 16-layer 512 GiB. The interface supports reads of 64 B—4 KiB, writes of 4 KiB, and 4 KiB pages, while achieving maximum bandwidth requires 64 KB reads and 1 MB writes aligned to 64 KB. HBF performs reads and writes through DMA and is not part of the GPU cache hierarchy; when used alongside HBM, it requires independent memory management. Its powered-on data retention time at 85°C is approximately 24 hours, its lifecycle must be managed by the host, and writes must be carefully controlled to achieve a lifespan of approximately 10 years or avoid exhausting 100% of endurance. Overall, it is a read-optimized, write-constrained, host-managed medium, making data placement primarily a software problem. Regarding the software ecosystem, the report treats vLLM as the current default focus for production deployment, while also listing general-purpose inference engines such as SGLang, LMDeploy, and Modular MAX, as well as vendor-optimized stacks such as TensorRT-LLM, OpenVINO, AWS Neuron, and Google JetStream. Existing vLLM paths primarily target CPUs or LMCache and do not directly support HBF. Using HBF requires a new allocator or backend, hot/cold data placement policies, asynchronous prefetching, and endurance telemetry; otherwise, its hardware capacity advantage cannot be converted into production inference gains. The Kimi K3 example demonstrates a relatively clear data placement scheme. The 2.8T model has total weights of 1.56 TB, of which 93% of the bytes, approximately 1.45 TB, are MoE expert weights with write-once, cold-read characteristics that the report considers suitable for an HBF expert pool. The remaining 7%, approximately 110 GB of weights, are better suited to HBM. The KV Cache for each 1-million-token sequence is approximately 30 GB. HBF can also serve as a KV offload pool and prefix cache, replacing pinned host DRAM with greater capacity. This creates a tiered architecture: HBF stores vast but relatively cold expert weights and offloaded KV, while HBM stores active weights and high-bandwidth data. Sparse attention further expands HBF's applicability. For long-context or multi-turn KV, the system can retain the complete context at low cost while reading only the top-k portion. The report gives a read ratio of φ＝top-k/context at approximately 1%—2%, below the threshold φ*=α/l. Under this condition, HBF capacity can retain all KV, while the smaller read volume avoids hitting the bandwidth wall. However, this advantage explicitly depends on sparse-attention models such as DSA, CSA, and Kimi-Linear and does not apply to architectures that require dense reads of the entire context. HBF capacity may also reduce expert-parallel communication. A conventional example shards experts across 8 GPUs and performs all-to-all at every layer. If two nodes can each store experts locally in HBF, expert-parallel sharding can be reduced, significantly lowering or eliminating all-to-all communication. Capacity therefore not only addresses whether a model fits but may also buy back some communication overhead, although the benefit still depends on expert-access locality and prefetch bandwidth. The ultimate conclusion is that HBF is a precision tool rather than a universal replacement: it offers a cost advantage only for low-bandwidth-demand, small-batch/low-access-intensity MoE and sparse-KV scenarios with long contexts; outside this range, it may become a trap. To expand its value, read bandwidth α must increase, the per-GB cost gap β relative to HBM must remain sufficiently large, and write endurance, write bandwidth, latency, and software support issues must also be resolved.

Analysis framework

The report first establishes a memory cost equation using β, α, capacity C, and bandwidth requirement I·b, then segments workloads by MoE versus Dense, batch size, context length, and access sparsity. It subsequently compares all-HBM, all-HBF, and hybrid configurations and uses a 72-GPU rack simulation of Kimi-K2 to evaluate capacity, bandwidth, and three-year amortized costs. Finally, it combines Kimi K3 data layout, the vLLM software path, sparse attention, and expert-parallel communication to determine the deployment conditions and boundaries of HBF.

Methodology notes

• (Out-of-Vocabulary Method)

  β—α Memory Cost Equation

  The report uses β to represent cost per GB and α to represent bandwidth per unit of capacity, and compares the memory cost required to meet capacity and data-supply bandwidth needs through $mem = β·max(C, I·b/α), explaining why a low price per GB does not necessarily reduce cost per token.

• (Out-of-Vocabulary Method)

  Rack-Level Cost-per-Token Simulation

  Under the same 72-GPU rack, cost, and power-consumption assumptions, the report adjusts tensor-parallel and data-parallel configurations to compare the capacity and bandwidth of all-HBM, all-HBF, and hybrid memory, evaluating inference economics based on capital and operating expenditures amortized over three years.

• Industry/Sector Analysis FrameworkSubstitution Effect Analysis

  Workload-Based Boundaries for HBF Substitution

  Rather than treating HBF as a comprehensive replacement for HBM, the report determines the specific data pools in which it can replace HBM or host DRAM based on MoE versus Dense, batch size, context sparsity, and deployment scale.

Asset mapping & comparison

Structured mapping from thesis to named assets (strengths, weaknesses, peers, risks).

• HBF

  Serves as a high-capacity, low-bandwidth memory tier in AI inference and can accommodate MoE expert pools, offloaded KV, and prefix caches.

  Strengths

  At the same cost, capacity can reach approximately 8—16 times that of HBM; it is suitable for write-once, cold-read data and can reduce expert sharding and all-to-all communication.

  Weaknesses

  It has lower bandwidth, constrained writes, and requires DMA and independent memory management, while production software support remains incomplete.

  Comparison

  In the 72-GPU simulation, capacity is approximately 14 times that of all-HBM, but aggregate bandwidth is approximately 0.6 times as high; rack-level token throughput is lower than HBM.

  Risks

  If batch size or access intensity rises, or attention reads are not sparse, the bandwidth wall will turn it from a low-cost capacity solution into a cost trap.

• HBM

  Used for high-bandwidth, active weights and dense or high-batch inference, serving as the report's benchmark memory solution.

  Strengths

  It offers higher rack-level bandwidth and token throughput and is suitable for Dense, high-B, and frequently accessed data.

  Weaknesses

  Its cost per GB is higher, and when capacity is constrained, additional tensor parallelism or GPUs are required to accommodate the model.

  Comparison

  The all-HBM rack has 20.7 TB of capacity and 1,584 TB/s of aggregate bandwidth; the all-HBF rack has 294.9 TB and 922 TB/s, respectively.

  Risks

  In low-bandwidth workloads primarily constrained by single-machine capacity, users may pay for bandwidth beyond actual requirements.

Key data

• HBF Relative Capacity PositioningApproximately 8—16× HBM capacity at the same costHBF trades lower bandwidth for a capacity advantage and is not a cheaper version of HBM.
• Maximum User Bandwidth of Three HBF Grades0.384 / 1.536 / 3.072 TB/sCorresponding to Grade 1, Grade 2, and Grade 3.
• HBF Interconnect Rates8 / 16 / 32 GT/sCorresponding to the three hardware grades.
• HBF Capacity Configurations8-layer 256 GiB; 16-layer 512 GiBReflecting its high-capacity positioning.
• 72-GPU Rack CapacityHBM 20.7 TB; HBF 294.9 TB; hybrid 89.3 TBEquivalent to 1×, approximately 14×, and 4.3×, respectively.
• Capacity per Data-Parallel InstanceHBM 2.3 TB; HBF 4.1 TB; hybrid 2.5 TBThe simulation configurations are TP8·DP9, TP1·DP72, and TP2·DP36, respectively.
• Aggregate Rack BandwidthHBM 1,584 TB/s; HBF 922 TB/s; hybrid 1,418→279 TB/sAll-HBF is approximately 0.6× all-HBM.
• Per-GPU BandwidthHBM 22.0 TB/s; HBF 12.8 TB/s; hybrid 19.7→3.9 TB/sReflecting the bandwidth cost of high-capacity configurations.
• Rack Simulation ModelKimi-K2 1T @ FP4The primary context is 1 million input tokens/1,000 output tokens, with a batch size of 1—512 per DP.
• Cost BasisThree-year amortizationIncludes capital and operating expenditures and is presented as a ratio.
• Low-Batch HBF Capacity UtilizationApproximately 85% idle capacityAlthough HBF has a cost advantage at low B, a large amount of capacity remains unused.
• Kimi K3 Total Weights1.56 TBThe model size is 2.8T.
• MoE Expert Weights1.45 TB, accounting for 93% of bytesThey have write-once, cold-read characteristics, which the report considers suitable for HBF.
• Remaining Weights110 GB, accounting for 7%The report considers them better suited to HBM.
• KV Cache per Long Sequence30 GB/1-million-token sequenceHBF can be used for a KV offload pool and prefix cache.
• Sparse-Attention Read Ratioφ approximately 1%—2%The report states that this is below the threshold φ*=α/l, allowing complete KV to be stored in HBF while reading only top-k.
• Powered-On Data RetentionApproximately 24 hours at 85°CThe host must manage the data lifecycle.
• Target Service LifeApproximately 10 yearsWrites must be carefully managed; otherwise, 100% of endurance may be exhausted.

Impact & implications

The report argues that AI memory configurations should be determined jointly by workload and deployment scale rather than by comparing price per GB alone. HBF can accommodate extremely large models or KV pools on fewer GPUs, a single machine, or local nodes, and may reduce expert-parallel communication. However, when pursuing high rack throughput, using dense models, or increasing batch size, its low bandwidth raises cost per token. Actual deployments are more likely to use a tiered approach: HBF carries vast, cold data, while HBM retains active, bandwidth-sensitive data.

Risks

• As batch size, access intensity, or I·b rises, HBF will hit the bandwidth wall, and HBM may become the lower-cost choice.
• Hot-expert caching may fail under mixed queries because expert popularity tends to flatten.
• HBF write endurance, write bandwidth, and latency require strict host-side management; otherwise, service life and availability may be affected.
• Existing vLLM paths primarily target CPUs or LMCache and lack an HBF allocator, placement policies, asynchronous prefetching, and endurance telemetry.
• The advantage in sparse-KV scenarios depends on sparse-attention models; it does not apply when the context is read densely.

What to watch

• Monitor whether HBF read bandwidth α can increase and whether the per-GB cost gap β relative to HBM can be maintained.
• Monitor whether write endurance, write bandwidth, and latency issues can be addressed.
• Monitor whether inference engines such as vLLM add HBF backends, dedicated allocators, placement policies, asynchronous prefetching, and endurance telemetry.
• Monitor the locality of multi-agent access, prefetch bandwidth limits, and the match between memory configurations and workloads.
• Monitor whether sparse-attention models and batching of similar queries can keep the actual access ratio within HBF's effective range.