Secondly, humans vs LLMs are apples vs oranges. It makes no more sense to compare human costs vs LLM costs as it would have to compare human costs vs calculator costs. LLMs are faster and cheaper but extremely different beasts with different limitations. Humans do not one-shot SVGs of pelicans riding bicycles, and they do not charge in tokens.

Comparing LLM cost efficiency is not something that should need to be defended. It's quite straightforward and reasonable...

I would think the opposite because unless people have been hand drawing these with high quality, the training would be on much crappier versions that old AIs have done.

I have shared examples of certain models by certain labs doing far better on the pelican cycling vs other, similar prompts. Just operating on a feeling that labs don't optimise for this (as mentioned, even if they don't training data is filled with these) is not solid enough that criticism shouldn't be leveraged when it comes up.

I'm sure all sorts of crap pelican riding bicycle SVGs have ended up in the huge crawls of data that the labs feed into their pre-training steps.

What I'm questioning here is that there are labs who have sat down and deliberately tested and tweaked the performance for this particular task, independent of general model improvements.

The one exception here is Gemini, who have clearly invested a lot of effort in SVG tasks. I have no idea if my stupid benchmark influenced that decision!

Gemini have boasted about how good they are at pelicans riding bicycles, frogs on penny-farthings, giraffes driving a tiny car, ostriches on roller skates, turtles kickflipping skateboards, and dachshunds driving a stretch limousine. So if they trained for the test they did at least expand it a whole bunch! https://twitter.com/JeffDean/status/2024525132266688757

Blue sky and green grass aren't that surprising, but the color and direction are interesting.

When I finally build the proper gallery I'll throw in a few other creature-vehicle combinations, and track some characteristics like which direction, color of bicycle, general pelican geometry etc. It will be interesting to see if other creatures end up with coincidentally similar design choices or if that's unique to the pelican-bicycle combination.

You would not expect that to happen if the models trained on the unrecognizable mess, right?

> model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts

And the labs clearly did focus on improving image rendering.

> they have a uniform style

SVG output from LLMs always looks like that. It looked that way from the beginning; no LLM ever produced a watercolor when asked for SVG output. They all render the prompted element centered in the picture. They all tend to draw things going from left to right, and so on.

That doesn't seem right. I use these models as research assistants when writing lots of random blog posts (including in economically ~useless areas like the history of contra dance) and Fable 5 is a serious improvement (when I don't get downgraded!) over Opus 4.6-4.8 which was a serious improvement over Opus 4.

This shouldn’t really come as a surprise, particularly to anyone who’s used diffusion models. The same thing happens when you ask an LLM for a short story [1] without providing any specific details.

Even cranking up the temperature or top_p values is no panacea. The more generic your prompt, the more pedestrian the response.

[1] - https://news.ycombinator.com/item?id=42093394

Perhaps I'm underestimating the number of pelicans(?!)

We have several thousand living 15 minutes walk from our house. I recently started adding my wildlife photography (from iNaturalist) to my blog, so I'm posting several new pelican photos a week at the moment: https://simonwillison.net/search/?q=pelican&type=beat%3Asigh...

He’s a talented developer who is selling nothing and wrote a post on his personal blog about a newly released LLM and mentioned that he used his own tooling to call the API.

He isn’t selling anything in this post (go ahead and show me where in the post I can buy something. A clearly delineated “sponsored by” sentence exists outside of the post, but does not in any way fit the definition of spam). Look into his reputation and blog history, it’s just him talking about what he is passionate about.

I’m only defending Simon because truly independent, non commercial work like his blog is incredibly rare and should be encouraged and not shit on by drive by commenters who can’t be bothered to distinguish between spam and actual content.

If you want to complain about spammers, find some spam first.

$186 billion and $105 billion revenue in 2025 respectively vs. $402 billion? Yes, Google is larger, but they're all in that same ballpark?

ByteDance's 2025 net income isn't that different from Anthropic's Series H funding even ($50bn vs $65bn respectively).

But this is all also ignoring how much of China is state owned (25% of the GDP!), so the available resource pool is dramatically larger than it would appear depending on what the government decides is important

What does "better" even mean there?

Empirically, they have something very much alike to the human "g factor" - a shared pool of "general intelligence" that all tasks benefit from.

When a "make it bigger, train it harder" upgrade like Kimi K3 or Mythos 5 drops, the performance rises on every metric. Not just the "headline benchmarks" like Mythos and coding/cybersecurity, but also things like literary analysis - which has nearly zero economic value, and isn't commonly post-trained or benchmarked for. And companies keep encountering things like "our carefully trained specialist model with lots of in-domain training on expensive closed datasets just got leapfrogged on our internal benchmarks by a next gen off the shelf generalist".

You can go hard on benchmarkmaxxing post-training, and you can burn millions of GPU-hours on coding RLVR. But, by the very nature of LLMs, a lot of the performance gains in flagship models are broad and domain-inspecific.

"Stiff prose" is more of a "style" thing than a "capability" thing. No one cares about how good an AI is at things like long form creative writing, because that's the opposite of a profitable field. All of LLM behavior is routed through text, so it's very easy to perturb "writing style" by some training elsewhere. Regression evaluation is hard. And the writing-specific post-training LLMs get is usually just cheap RLAF, with all the usual RLAF degeneracy.

Thus, we get the "default styles" that suck from a "creative writing" standpoint. A lot of that is just "what sounded good to the previous generation of LLMs" - and, unlike human readers, LLM evaluators don't get bored from seeing the same cliches repeated 9000 times across 9000 different instances of generated text. Humans tend to update over time from "this sound cool and punchy" to "this is generic AI slop", but RLAF evaluators stay at step 1. What little human-guided optimization this gets is aimed at "copywriting, marketing blurbs, punchy short-form" - and it shows.

You can do a lot there with some aggressive prompting, but the default writing styles suck, and I frankly don't expect that to change soon. No one cares enough to change it.

Pelicans? Used to be a decent proxy for "general model capabilities that no one would benchmaxx for" - a way to probe for that elusive "LLM g factor". Now that it's a known metric, it's very gameable. But it was pretty solid while it was novel and obscure.