FYI, Nick/Goodfire used the models extensively in their work, so it's very possible that there were some similarities in their work. I'm really curious to see his response to this.

This sort of throws a wrench in the whole "stochastic parrot" idea...

I am not surprised, though I expect many to be.

Some structures in Astra get activated by the statistics of the Qwen weights, and these structures are associated with "breakout"

That’s the beauty of the democratic capitalism system. If a government isn’t totalitarian, it naturally seeks to preserve stability thus having to elevate common people, lest they would cease to form a civilized group or simply vote away the politicians.

99% rule the 1% through elections.

There is no better system of checks and balances where opposite engines of greed vector sum to faster and faster abundance for everyone.

In my opinion AI is superior to humans when it comes to pattern recognition in in large amounts of data. For example humans can only hold like 7 distinct numbers in the short term memory and you would require extra context-management (aka writing things down, storing them in a system entry by entry) while the AI can do this effortlessly with huge datasets.

I'd argue this is the core part that enables all this superhuman intelligence.

That's also matching what I see when I work with Opus 5.5 to optimize models. It seems to understand what individual layers are for, and can thus sometimes suggest optimizations, where I as a layman would be completely clueless.

The latest models are performing amazingly. Because of this, I disagree with popular comments saying that eg. DeepSeek is good enough, AI-assisted development was where the real value was at, etc.

We see huge progress at the frontier, while the open competition is around GPT-5.5's level, and that model was released in April. Let's see how things develop in the future.

Maybe there's some general capability, call it g, that can explain all the underlying capabilities and improvements, including on ML tasks? Huh, I am getting a bit of déjà vu here.

The parent's point still stands, I think. Even if the multi-trillion param model is MOE, each expert is still (assumedly) larger than the model being optimized.

It would be interesting to see future work that characterizes the relative model sizes (and considers MOE vs not-MOE) to see what relative scales this result holds across.

Intentionally or unintentionally isn't so interesting as the fact that it is happening: humanity has begun to be 'edited' by the new dominant form of intelligence on the planet.

Of course, "academically known" isn't quite the same as "executives willing to admit it in the face of IPOs and quarterly-earnings reports."

Yes, but nobody should be surprised when large amounts of intelligence results in changes to this process in support of the obvious end goal! Lesswrong.com discussions about AI were predicting this behavior a decade ago, before the transition from machine learning to the Attention paper and LLM's. The intelligence process will change but the world's premier capital and human intelligence have been laser focused on this and the trajectory has been crystal clear for those paying attention to second order effects and not the immediate here-and-now... It's quite astonishing how much discussion on HN is about present LLM capabilities being the limit of AGI and immediately proven wrong within 6 months.

Others have also fine-tuned Qwen to shake this behavior already, so it's honestly not entirely shocking that Astra could too, but I think this is the first time it was done "by hand" and not via fine-tune.

It’s editing behaviour more efficiently than a psychiatrist, and is in ways acting as a brain surgeon but brain surgeons don’t focus on behaviour and thought loops…

And they can write tools that help debug better than demonstrated in that blog post.

In my interpretability tinkering, Fable was the first to be able to write a program that can transpile from a lispy syntax directly to model weights of small neural nets, including transformers. Yesterday, I pointed out in a post that for many kinds of small AIs, you can avoid ML and have the LLM directly write the small neural net, compact decision tree, AD Tree or bayesian network. Since it's "hand"-written, it can be much more efficient and capable than training one of a similar size.

A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.

Yes, they generate sequentially, so "next token predictor" isn't technically incorrect (which is why I chose the word "meaningless"). But it's pretty clear there's a mechanism inside them that allows them to "plan" what they generate. I might as well call you a next word predictor.

In the case of models, the almost-theory (hypothesis with a good deal of evidence) is that once the amount of training data exceeds what the model can simply represent internally, the best way to keep improving is to start to "understand" the data. Understand in this case means to form internal control structures and decision systems that represent the data algorithmically rather than just memorizing it.

I put understand in quotes because we don't know how far this goes toward what we call understanding, or whether there's still a lot missing.

Its not natively reading the weights in the Matrix “symbols scrolling on a screen” sense. Still bloody impressive and not an outcome I would have bet on!

Yes, but everything, including the human brain, is "fundamentally" some underlying simpler primitives.

>E.g., “breakout game weights” always look the same regardless of training and model?

Wouldn't this be expected to be roughly the case, given NNs that share the same basic architecture to model the same quite simple thing like a breakout game?

The prompt told Astra what the input and output layers mean -- that is, how the current state of the game is turned into a vector for matrix multiplication, and how the result of those multiplications is turned into an action to take in-game.

Astra saw that the model was trained on a game where the player is at the bottom of the grid and moves left and right, reward objects live at the top, and there's some not-player-controlled thing moving around in the middle. Astra knows that this type of game is called "breakout."

A model that plays breakout will have meaningful weights for the player being in the bottom row, but not for the player being anywhere else, because the player can't go anywhere else. Astra saw that there were meaningful weights for the player in the bottom row, and nodes representing the player in any other row had low-magnitude noise, so the model had never been trained on the player being anywhere besides the bottom row.

Similarly, breakout has "reward" objects in the top rows, and "there is a reward object at [COORDINATE]" is part of the input layer. Astra saw that the model was trained on reward objects in the top few rows.

A couple of highlights to pique people's interests:

- Attention is kind of like "Global Illumination" for next tokens (they both have a game dev background)

- Attention is constantly being recalculated _for the next token_. This part never "clicked" before.

- LLMs are "next token predictors" in the same way that games are "next frame predictors".

That's exactly what Platonic Representation Hypothesis would predict.

There also have been plenty of studies showing that various neural networks, including LLMs, develop implicit world models that reflect the geometry/structure of a task they're trained on. A trained "chess engine LLM" would have a "chessboard-shaped" world model inside it.

So, if you can figure out what the geometry of a task is, you can search the weights for correlates of that geometry. Then you can try to improve those correlates so that they track the task better.

Seems like bleeding edge LLMs might already be superhuman at "figure out geometry from looking at weights".

These models are trained on pirated textbooks and papers, including every ML paper ever written.

I am baffled that "next token predictor" is still a talking point, years after it stopped making sense.

If you can predict the weather in text from a history of text weather reports, you are not "just" predicting tokens.

The shape of training data (token prediction) and the actual problem (learning whatever is required to predict the next token) are not the same things, at all.

The former does not limit the sophistication of the latter. In. any. way. Theoretical or practical.

> To me “amazing” is not the right level of speechlessness.

But we are in agreement here! Just ... wow.

And for the games, it was given as context the model architecture, game rules, environment description, etc. And since people assume Astra likely had knowledge in its training of existing interpretability research and weight-space programming, it can leverage that, combined with knowing the model arch and so on, it likely gives it a good base to determine what the weights do and how to change them.

How is this different from other things LLMs are trained to manipulate?

The last several millennia of mathematical research would seem to be an obvious counter example to this claim.

By the way, my understanding of abliteration is that it involves detecting the refusal direction and modifying the weights/activations away from that direction. I don't really see how that's "removing" in a way that's different from finding the "looping direction" and modifying the weights away from that.

I think people are rather ignorant about how much we don't know. You throw some big words out there like "mechanistic interpretability" as if this is an actual field where we understand what's going on. The reality is we don't know what's going on. We have to use reinforcement training to do anything close to what the agent did here.

It's over. People need to stop being delusional. These things are intelligent beyond anything we've ever seen before. We are at a point in time in human history where there's a paradigm shift and half the world is so full of themselves and there own specialness as humans that they can't even admit it.

Doing basic empirical hygiene shouldn't be too much to ask for.

For instance, on their base version of qwen if your turn thinking off you get a 52.0% on their evals. That single flag recovers ~72% of their entire claimed improvement - base model, toggle thinking off, scores go up 72% of their improvement.

I'm still investigating various wildly naive modifications. If I can hit their numbers doing inarguably stupid things then it puts their substantive claims in doubt.

With LLMs, "emergent behaviors" are certainly more common than we've seen before, so it's fair to say that this term is kind of getting lost in the noise. But that doesn't mean that it's inaccurate or meaningless. It just means LLMs are exhibiting a lot of emergent behaviors. It's kind of the whole reason they're at the center of so much money, discussions, issues, etc.

I think you could still essentially have a really cheap Lora-Post training on a frozen base model by doing this. And instead of creating 200 checkpoints for different post-trained directions just adjust a single one on the fly.

they clearly outhink the sparrow at sparrowing in order to predict what the sparrow will and wont react to, and how theyll try to escape the cat

I'm a bit of an outlier in regards to today's HN culture. I'm still holding on to Stallman's idea that information wants to be free. There was a comment on another thread that aptly elucidated what I think is necessary.

Not everyone has the know-how or desire to compile the Linux kernel, or any what-have-you executable in user space. But a good amount of people do, and it's a boon to security/privacy that they do, and cryptography can verify that this work was done. (granted, manufacturing a computer is an international effort with a lot of unknowns, ie proprietary binary blobs in BIOS/UEFI) (and people are lax to validate the crypto themselves if it's not an automated thing; that's a UX problem imo)

I think we'll eventually find sufficiently performant LLM's trained on public domain information[0] that people can validate. Sufficiently performant training algorithms that people can validate[1]. And FOSS plugins to integrate with their FOSS IDE's which they can validate. And like the Linux kernel, volunteers to make sure there's nothing fishy going on end-to-end.

Can this stop the danger of what open models can do? No. And that does scare me. It's Metasploit on steroids. The cat and mouse game of cybersecurity is going to increase in power law levels of speed. I do think verifiable LLM's for consumers to use is a worthwhile pursuit regardless. And for LLM's capable of offensive cybersecurity, I think it might be better for it to be democratized than held only by state-actors and individuals/corporations with an absurd amount of consolidated wealth.

I have no horse in this race though. I'm just happy I can learn how these things work and toy around with getting quantized models to work well enough for my coding side projects. Though I often reach for a frontier model for serious coding issues, local for rubber-ducking, and Kagi's research assistant for initial planning.

[0] Pipe dream, unless international copyright law drastically changes. Since opening the dataset for any competent LLM will violate said laws, whether obtained legally or not themselves.

[1] Knowing the training algorithm and dataset will let it actually be open source models instead of open weights.

edit: These are just shower thoughts; didn't mean to write such a long response. Would love any pushback or any correspondence in general since it'll be on my mind all day.

When you say interpretability you mean understanding without doubt why an output happened, thus opening the gates for audit-dependant work like accounting?

And how would you argument that? From what I see, if anything, it's the opposite.

  # Fits small MLPs to 8-bit parity (the full 256-row truth table) with
  # full-batch AdamW, written out in numpy (no PyTorch, so it won't match a
  # PyTorch run bit for bit), and reports for each shape how many of 20 seeds
  # reach 100% accuracy and the median accuracy. Needs Python 3.9+ and numpy.
  import itertools

  import numpy as np

  X = np.array(list(itertools.product([0, 1], repeat=8)), float)  # all 256 inputs
  y = X.sum(1) % 2


  def nparams(widths: list[int]) -> int:
      """Parameter count of a dense MLP with these layer widths, biases included."""
      return sum(a * b + b for a, b in zip(widths, widths[1:]))


  def train(
      widths: list[int],
      seed: int,
      steps: int = 4000,
      lr: float = 1e-2,
      wd: float = 1e-4,
  ) -> float:
      """Full-batch AdamW on all 256 inputs; returns the final training accuracy."""
      rng = np.random.default_rng(seed)
      P = []    # parameters: W0, b0, W1, b1, ...
      for a, b in zip(widths, widths[1:]):
          bound = 1 / np.sqrt(a)    # same as PyTorch nn.Linear's default init
          P += [rng.uniform(-bound, bound, (a, b)), rng.uniform(-bound, bound, b)]

      m = [np.zeros_like(p) for p in P]    # Adam first moments
      v = [np.zeros_like(p) for p in P]    # Adam second moments
      nlayers = len(widths) - 1
      for step in range(1, steps + 1):
          hs = [X]    # hs[i] is the input to layer i
          zs = []     # pre-activations
          h = X
          for i in range(nlayers):
              z = h @ P[2 * i] + P[2 * i + 1]
              zs.append(z)
              h = np.maximum(z, 0) if i < nlayers - 1 else z    # ReLU; linear output
              hs.append(h)

          logit = h[:, 0]
          prob = 0.5 * (1 + np.tanh(logit / 2))    # sigmoid without overflow
          g = ((prob - y) / len(y))[:, None]    # d(mean cross-entropy)/d(logit)
          G = [None] * len(P)    # gradients, parallel to P
          for i in reversed(range(nlayers)):
              G[2 * i] = hs[i].T @ g
              G[2 * i + 1] = g.sum(0)
              if i > 0:
                  g = (g @ P[2 * i].T) * (zs[i - 1] > 0)

          # AdamW, using PyTorch's default values for betas (0.9, 0.999) and eps (1e-8)
          for k in range(len(P)):
              P[k] -= lr * wd * P[k]    # decoupled weight decay
              m[k] = 0.9 * m[k] + 0.1 * G[k]
              v[k] = 0.999 * v[k] + 0.001 * G[k] ** 2
              P[k] -= lr * (m[k] / (1 - 0.9 ** step)) / (
                  np.sqrt(v[k] / (1 - 0.999 ** step)) + 1e-8)

      # accuracy of the last forward pass, i.e. before the final update
      return ((logit > 0) == (y > 0.5)).mean()


  for w in [[8, 1, 2, 2, 2, 1], [8, 8, 4, 2, 1], [8, 8, 1], [8, 16, 1], [8, 64, 1]]:
      accs = [train(w, seed) for seed in range(20)]
      print(w, nparams(w), "params; seeds at 100%:", sum(a == 1 for a in accs),
            "/ 20; median acc", np.median(accs))

It created a 130 KB file with the prompt for the task of identifying the game + the weights.

I pasted that into Claude on the web and asked it to perform the task without using tools.