also the internet wasn't as robust so less training data ofc

>There was no grand plan that led to LLMs being created. People just did a bunch of things step-by-step. With so much uncertainty, progress is more “little hops” rather than grand journeys—technology is more “evolved” and less “designed”.

This pretty much sums up the whole article, and it makes perfect sense.

So much was going to happen anyway that you really do have to look away from the technology to answer why.

I would say GPTs didn't come earlier because sama didn't have enough high-dollar contacts yet who would extend their confidence to him.

And then what you get is little hops done big because that's whats already familiar to the big money owners.

So LLMs are first out of the gate to reach impressive capabilities, but something completely unfamiliar that may be superior due to purposeful design including fundamental principles will have to lag behind until a very big change would become possible.

Running GPT-2 after it's trained wasn't as computationally-easy as nowadays. So, you not only need the best supercomputer in the world to train it, but also a datacenter with similar computational power to run enough instances to make it useful.

My take was that we didn’t have these 4 things

1. Someone with enough conviction of end goal which is agi

2. That someone with enough capital or agency to pursue it

3. Enough capital to explore all options - at that time we didn’t know about transformers

4. Enough smartness to know what ideas to not purse and which ones to prioritise

As things got cheaper, it allowed more people to explore.

Around 2011, my exposure to AI was like Drools planner, the predecessor of Timefold. It was to generate an ideal timetable. It used an n^2 memory heavy Rete algorithm, a scripting language in JVM to write rules with a terser syntax and simulated annealing to explore/choose states that obeyed constraints. In college AI was fancy algorithms and languages with terse syntax.

A few years later I saw - neural network projects which can do basic image recognition - a paid smartphone app that can count objects instantly - determine the mood of an online comment after you trained on a sample

I never imagined a tool that packages all knowledge and requires no training on my part.

I had an idea for a project I called "LLM 2007" - to produce a CD that would include an LLM that would run on a PC of that era. I wanted to make it look as if it really was a mid-2000s product. That would make it seem like an artifact that came from alternate reality.

For that, I thought of making a Windows 2000 style GUI using pure WinAPI, backporting llama.cpp and bundling Qwen3-0.6B. Never started this though.