The Next AI Bottleneck Isn't Compute. It's Memory.

A 100 TB/s HBM4 system burns roughly 2 kilowatts on memory alone. That single number is why the next AI scaling wall is bandwidth and energy, not compute. d-Matrix, a Silicon Valley startup, says its new "Raptor" 3D DRAM chip, unveiled at Hot Chips 2026, can match that bandwidth while using roughly one-tenth the power. The catch: every major DRAM manufacturer is pursuing a similar stack, Raptor's density is half of HBM4's, and the numbers are still vendor-stated rather than independently validated.

That 2-kilowatt figure is the right place to start. Today's high-bandwidth memory (HBM), the stacked DRAM that ships next to every AI accelerator from NVIDIA's Vera Rubin to AMD's Instinct MI455, moves data at roughly 2.4 picojoules per bit. A picojoule is a trillionth of a watt-second; at 100 terabytes per second of traffic, that works out to about 1.92 kilowatts of memory power, before the GPU or any compute work happens. As models grow and their KV cache (the working memory a large language model keeps for every conversation) balloons, that number gets worse, not better. HBM4E on the roadmap raises the bandwidth ceiling but not the energy efficiency; the practical ceiling sits around 20 TB/s per package, capped by how many stacks can fit on a chip's edge.

d-Matrix's answer is a 3D-stacked memory architecture it calls 3DIMC, built in partnership with ASIC designer Alchip. Instead of placing memory next to the processor, the company stacks DRAM dies directly under the logic die, dropping the PHY (the physical-layer interface that connects compute to memory) into a layer that fuses the two. d-Matrix says this reorients the stack so heat flows away from the DRAM rather than into it, "mitigating" the thermal challenge that has made other 3D-DRAM designs difficult, though the company admits that mitigation, not solution, is the right word.

The numbers, all from d-Matrix's own Hot Chips 2026 presentation and Pavehawk test silicon, are the right ones to anchor the comparison. Raptor targets roughly 100 TB/s of bandwidth at 0.3 to 0.37 picojoules per bit, about one-seventh the energy of HBM4. d-Matrix frames the end-to-end claim as 10 to 20 times better energy per bit moved, and 10x faster inference than HBM4-based systems. For comparison, an SRAM-based accelerator card like d-Matrix's earlier Corsair can hit around 300 TB/s at about 1 nanosecond of latency, but it tops out near 4 GB of capacity and burns tens of watts just sitting idle. Raptor lands in the middle: 32 GB per single-high stack, enough to hold a meaningful slice of a model's weights or KV cache, at bandwidth that approaches SRAM and energy that approaches HBM.

That middle position is also the bet. HBM4 already ships today in production AI clusters; HBM4E is on every DRAM maker's roadmap. SK hynix, Samsung, and Micron are each pursuing some version of stacked-logic-plus-DRAM, and Samsung's ZHBM work, for instance, takes a different path through the same destination. d-Matrix's Raptor is one data point in that race, not a winner of it. The 10x inference claim is from vendor benchmarks on test silicon, not yet replicated on production hardware, and the company's own deck concedes that Raptor's density is roughly half of HBM4. Any system that needs more memory per accelerator will pay for it in chip count, board area, or both.

For the past five years, AI infrastructure has been sold on compute: more FLOPs, larger models, more GPUs. The memory wall has been the silent tax on every one of those deployments, the reason a 70-billion-parameter model needs a 2-kilowatt memory subsystem just to run. d-Matrix's Raptor, and the broader 3D-stacked DRAM race it joins, is one attempt to make that tax small enough to ignore. Whether the approach holds up under independent testing, and whether the DRAM incumbents reach the same destination first, is the open question the rest of 2026 will answer.