Give me 1TB RAM and 5TB/s or something that exciting.
Plenty of businesses want in local inference and are priced out.
Nobody helps China like American corps helps China. Domestic anticompetitive behavior spurs demand for international competition
I'm sure NVIDIA, AMD, Intel, Qualcomm, etc. would sell more hardware if they could, but they can't.
But this gives you significantly more memory capacity for a compute/vram bandwidth tradeoff (i.e. arithmetic intensity) that's ultimately comparable to Blackwell, perhaps even more skewed towards compute. This seems good if you care about being able to store larger sparse models and more KV caches because that ultimately bottlenecks your ability to do inference.
https://www.servethehome.com/the-amd-instinct-mi350p-is-a-hb...
Splitting the tensors is closer to n times the effective vram bandwidth but there are still overhead, sync waits, and reduce steps that eat away at the overall performance.
The thing you are describing is much closer to AMD and Nvidia latest generation server offerings.
See MI455X for the absolute cutting edge and MI350P for something you can use outside of a datacenter.
Speaking of... from [0]:
Introducing Qualcomm Cloud AI 100 Ultra
High performance and cost-optimized cloud AI inferencing for generative AI
Written by
Rashid Attar
Nov 13, 2023
Today we announce the Qualcomm Cloud AI 100 Ultra, a new member of our portfolio of cloud artificial intelligence (AI) Inference cards, purpose-built for generative AI and large language models (LLMs). ...
Edit: Ah sorry you’re right, the bandwidth is lower but the TOPS is a lot higher. I’m not sure how much effect you get from higher TOPS with such low bandwidth though.
Qualcomm Cloud AI 100 Ultra has 800 TOPS, almost 10 times more.
Raised arithmetic intensity. It would be quite good for LLM prefill, and decode would want to adapt by pushing parallelism higher (saving on memory bandwidth by reusing the same layer weights for multiple streams) or, if unable to do so (e.g. due to KV cache constraints) by resorting to speculative decoding (where the expensive 'validate' step handles tokens in parallel within a single stream). It does not make the platform inherently unviable. It would also do well at other tasks such as image/video generation, which is already quite compute bound.
But what is that amount of SRAM for? It simultaneously seems like a lot, because SRAM is so expensive, but still not very much, because what can you fit in 0.5GB? Is there a specific workload that needs something that looks like this?
SRAM wise it doesn't seem too insane. About the same ratio of memory:SRAM as the 5090 (IIRC). Still more than I would have expected. I assume they found it helps process the current layer faster or something.
I'm starting to believe in Qualcomm! The Snapdragon X2 Elite Extreme is alas only available in a single system, with it's tri-channel on-package (!) memory good for 228 GB/s. But it's a beast. The core is just good, even in the mainstream form in other systems: some of the best single core performance you can get, shy of Apple.
And they are upstreaming their stuff much better! They hired a bunch of really good known FOSS developers and it seems to be working!
Now they are talking up High Bandwidth Compute as their next big thing. And I want to believe, even if it makes me nervous that they too intend to leave consumer computing completely in the dust. https://www.qualcomm.com/news/onq/2026/11/qualcomm-hbc-valid...
This is, as commented, at M4 Max levels of GB/s. It is not impressive. It has some uses! But if this was their modern opening salvo I would not be impressed.
Someone drop the YouTube link of their robot presentation
Why is on-package memory impressive/better in this case? It's one channel more than your typical consumer x86 system (192-bit bus vs 128-bit) which is something but you could technically have quad channel with two LPCAMM2 sockets?
It's a trend I like.
Cirrascale (8-card Ultra server, 128 GB each, 512 GB system RAM):
$4,699/mo (monthly term) or $3,759/mo (annual term)
Cloud GPU rental (per-GPU): from $5.15/GPU/hr