We recently argued that the current, extremely complex, multi-component chip architecture and networking technology required for microchips supporting AI was close to a tipping point, beyond which the very idea of chips would yield to wafer scale computing. Now it turns out that a bigger tipping point problem looms, suggests George Gilder, chief analyst at Gilder’s Technology Report.
Some of the most crucial of those multiple components are close to hitting a technology wall. AI chips use High Bandwidth Memory (HBM) – essentially stacks of DRAM (dynamic random access memory) – placed as close to the processor as possible to support the massive data flows required by AI. But now, a brilliant paper from Ed Sperling, editor in chief of Semiconductor Engineering, argues that HBM itself is hitting very physical limits on its performance.
Pretty good HBM can move trillions of bytes of data every second. It does this not by pumping power and raising frequencies but by working in parallel across many thousands of pathways. Those pathways are called “through-silicon vias” or TSVs.
To make a TSV, you drill tiny holes (really, really tiny) vertically through the silicon, insulate them, and then fill them with conducting metal (typically copper). Finally, align the TSVs in one die with the connections in the dies above and below it.
Individual TSVs, trying to keep up with AI processors, move data at a tremendous rate. Increasing that speed even further is difficult and risky, so manufacturers have been increasing the number of TSVs instead, now mounting into the tens of thousands per HBM stack. But every TSV steals silicon area that otherwise could contain memory cells.
Bottom line? Every successive increase in bandwidth makes manufacturing even more fiercely difficult. That cannot go on forever. The result, says Raghu Sreeramaneni, fellow for HBM Design Architecture at Micron, is that it’s “getting to the point where we are now architecting solutions around thermals instead of the other way around.”
Sperling suggests where the new paradigm may be headed. The “solution will likely be more distributed processing of that data, including in-memory and in-sensor processing at the edge, and less data being sent from the edge to AI data centers.”
