I don't think that's what they are saying. In fact if they did the metric would be pointless. Rather they are saying by estimating that value on different architectures, one can find more efficient ones. They use open model to be able to remove unknowns. They aren't advocating for one model or another, only more efficient architectures.

> We propose miles per hour (MPH), distance travelled per unit of time

Stupid metric. It‘s not because you spend more time that you travel farther.

s/

Not a well-thought-out take.

Small models are much smaller than frontier models though, which is how they end up consuming less energy despite low batch count. (Though with local models growing strong agentic capabilities, batching becomes a reality with local models as well).

> Unless I misread it, are they saying local GPUs use less energy?

You misread it. From the abstract:

> local accelerators achieve at least 1.4× lower IPW than cloud accelerators running identical models

That's "intelligence per watt". They also have IPJ, per Joule.

So, they find local is 40% "dumber" than cloud for the same power or 40% more power for the same "intelligence".

Tables 13 and 14 summarize their IPW and IPJ metrics.

But, to your actual point, I think the "local is 40% dumber per watt than cloud" message is still an understatement. And maybe this is something I failed to find in the paper but they seem to ignore the "idle baseline" costs and talks about explicitly focusing on the power consumption of just the accelerator under load.

There is a large baseline power consumption just to support the accelerator. CPUs, memory, PS losses, network, fans, general environment cooling. This "cost floor" is different for data centers and a "random local computer" and I think must be in favor of data centers which are designed and built with efficiency in mind.

Idleness should also be considered. My local GPUs at $WORK and home are idle more than they are used. Idle time energy in real world scenarios should be somehow attributed to those brief, punctuated times when LLM functions are actually active on the accelerator. Actual, local LLM usage of a GPU is brief (assuming one user per PC). Even with my heavy usage developing s/w I'd guess I heat up a GPU about one hour per day total, sometimes much less. If that is local then one must pay 23 hours of idleness for that 1 hour of "intelligence". Of course a local PC is used for other things and the idleness penalty must somehow account for that. OTOH, data centers try to maximize utilization so their idle time penalty would be much less, perhaps close to zero, by construction.

You can most definitely batch local models and do unattended inference on a 24/7 basis to maximize utilization on local hardware too. The limits are usually set by some combination of memory utilization for KV cache (particularly on small dGPUs) and overall thermals/power limits (particularly on iGPUs with unified RAM/VRAM). (If you're not near thermal limits, the main alternative to batching is to use MTP or speculative decoding in order to raise arithmetic intensity and speed with the same memory utilization. But batching requests is generally viewed as preferable.) Newer models, especially from DeepSeek, do a nice job of reducing KV cache memory impact for any given context length and/or amount of parallel sessions, so batching on local hw really ought to be quite feasible.

> It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands.

This is only true if your local model is already resident in RAM / VRAM.

> During the winter time the GPU also doubles as a 300W in-house heater

There could be a service that works in reverse where if someone needs a heater for a few months, they could rent a portable server (e.g. using older repurposed GPUs) with a built-in 5G modem that would run inference on LLM queries. As an incentive perhaps renting itself could be free (or you could earn money?), but you'd still have to pay your electricity bill.

> During the winter time the GPU also doubles as a 300W in-house heater.

Just a reminder that heat pumps can consume 300W of electricity to provide 1200W of heat.

For sure but aren’t there more regulations and building codes around heaters? It’s been close to a decade since I took heat and mass transfer courses but this seems like saying you can use your oven as a heater for your house in the wintertime.

Please correct me if I am wrong.

What devices and models are people running locally?

> We've reached the point where local LLMs are good enough!

For some tasks, yes. For most of my deeper work they're not even close to my subscriptions.

> It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands.

I have some decent LLM hardware here and I strongly disagree with this. Claude responds quickly. Using Fable or Opus it will deliver a working result faster than my local models because it gets there in fewer tokens. That's just how it is.

> During the winter time the GPU also doubles as a 300W in-house heater.

This is a curse in the summer. I'm feeling it right now.

This is annoying but, can you point me to where to get started with local models? A repo, website, something?

Local LLMs make my laptop hot and are wrong about everything.

yea i see the potential of local slms too, it wont replace the frontier llms but for particular aspects they are generally really good, and at a certain point models like qwen 3.8 if people are able to set up proper api providers for it it can be extremely cheap and helpful. I am also working on trying to fix the memory and low context issue of such slms so they can actually be used locally for real use cases and not just small work

it really does feel like 'intelligence' is becoming so much more of a marketing term moreso than it is any actual definitive measure. I was acquaintances with a cognitive science post-doc many years ago who trained under the same program that Douglas Hofstadter was part of (and who he saw as a great popular science communicator but perhaps a bit over-rated when it came to his own research)

my very naive question to him back then was how close we were to understanding human cognition. we were both fans of grand strategy games (though the few hundred hours of Stellaris I played vs his thousands in EU4 paled in comparison) and I was asking if it was possible to map human cognition to the same array of interdependent logical chains-of-reasoning that games like that could be boiled down to

his answer, in short, was 'we are so, so, so far away and no, that's, at best, a reductive mental model of intelligence'

I keep that conversation in mind whenever I hear about all this talk of AGI - that realistically we're so far away from actual AGI in the same way that the inventor(s) of the wheel were from a gas-powered car, and there's many paradigm shifts to go in how we even understand what the nature of intelligence is before we get there

"We see a future where intelligence is a utility like electricity or water and people buy it from us on a meter"

- Sam Altman[0]

"Can you define intelligence?"

"Yes, it is this many moneys."

[0] https://www.businessinsider.com/sam-altman-ai-utility-electr...

> We have no dang clue what intelligence is, nor how to measure it.

Broadly in laymen's terms, no, we don't. We do have some ideas though. We have lots and lots of different tests of different facets of intelligence. How high can you count? (crows can count up to 30), how long can you remember things? (elephants can remember things from up to 50 years ago), Memory games, Reasoning. "The Science of Human Intelligence" , by Haier, Colom, and Hunt. is a good read.

Footnote on page 3.

>Total parameter count governs storage: at FP4, MIXTRAL-8X7B fits within 24 GB GDDR6 (Quadro RTX 6000), and GPT-OSS-120B fits within 128 GB unified memory (Apple M4 Max).