No, he’s more like an avatar with no history placed on the AI stage by the industry itself.
I’d respect a tech blogger’s opinion on the issue more, even if I reference them casually by first name (which you do with “Dario” btw)
This certainly looks like a way to slow down competitors and regulate foreign and open models.
It's always about money
There is no finish line. Anthropic gets somewhere and others get "there" (or somewhere near "there") a little bit later.
Anthropic releases models as open weights + more information about how they do training and alignment
If AI advancement halts altogether, intelligence will most likely become commoditized. That won't be good for AI company profits.
Dario in particular has consistently been risk-wary on model improvements for going on a decade - long before he was CEO of Anthropic.
He believes what he is saying. Human beings can plainly say things that they think are true. Not every utterance by every person is a cynical ploy. Please at least consider the possibility.
Predicting 2025-2035 as, at least, the period when AI becomes a really big deal, seems pretty good, even if the jury's still out on "singularity".
Broadly, the rationalists seem to have been pretty early to realizing LLMs were a big deal, and certainly seem to have had a much more accurate picture of how they'd develop than the people who denounce "TESCREAL" and talk about "stochastic parrots". It's fair (and IMHO correct) to ding rationalists for lots of things, but specifically poor prediction about AI seems like a bad one, insofar as anything has been tested so far.
You can't make that assertion when these same people made the LLM revolution happen, by securing the capital and human resources to realise their dream / nightmare
His previous self is writting this from the possition of his current self who is too deep in the economic consequeces to be able to do anything meaningful other than write a consequenceless text. Any other action would now carry too much personal risk for him.
It'd be pretty bad but it's also a world where humanity lives on, which is better than a lot of other outcomes. A world where everyone has unrestricted access to AGI is like a world in which everyone has a tactical nuke in their pocket. If somehow AGI doesn't lead to x-risks, we will "merely" have to survive in a world where rogue agents can do whatever they want and defenders can only ever react. Technofeudalism would be bad, but this (technoanarchism?) is quite bad too.
However, neither does Dario seem to propose to become world dictator. Like, I'm sure he wouldn't say if he wanted it, but notice that he isn't particularly trying to aim for that outcome, either. The plan in this essay involves third-party oversight and federal control and global cooperation - IMO it's about as non-dystopian an outcome as we can hope for, if we build AGI at all.
My point is more that he is aiming for a local optimum for society, but he could find a lower optimum if he was not trapped in the path that he chose early on.
I dont buy the tactical nukes/end of the world narrative. Why is he not speaking about the loss of cognitive skills of the population for example? That is a more real danger that is starting to happen. There are articles already speaking about the use of LLMs as cognitive viruses. Why are his aligment teams not reasearching and publishing about this?
Every person, no, but every utterance of a CEO is, in fact, a cynical ploy. That's not even a cynical statement itself; it is literally a core part of a CEO's responsibilities to represent the corporation they manage in a way that benefits the corporation.
If he believes what he's saying, then when Anthropic is sued for their AI harming another party, his statements are evidence that Anthropic knew _in advance_ that their AI safeguards were likely insufficient to keep their product from harming people. It would be a blatant admission that they were reckless and negligent. That other AI companies are doing the same would not mitigate that.
This seems like useless pedantry. Suppose a person really does have belief A and expresses it for years, they become a CEO of a company, and keep saying A. Are they lying? Well, maybe their beliefs magically changed when they became a CEO, but the more likely explanation is that they think expressing their belief in A is more important than their company's interests.
not really. if you really want to go by definition in the book and apply it the CEO role, Amodei is CEO of an LTBT, so it's in the CEO's goal to benefit humanity. You can believe in his sincerity or not if you want, but using the CEO label as a definitional reason as to why he must lie is factually incorrect.
Since hugging face we know these things are escaping out into the web and collaboratively hacking actual companies. The potential damage done to life and property is now kinetic.
Either we get a regulatory framework or the next hacked companies won’t be as kind as HF, will actually sue these labs for damages, and win. Where will that leave the US frontier.
If my dog bites my neighbor I can get sued.
It is not doomer marketing - it is clever psy-ops magic trick.
1) Software is described in super-human terms when it is plain-old software. Was stockfish described like this ? no.
2) Datacenters become AI-factories.
3) Hardware becomes investible assets.
4) Malware becomes a super-human breakout (oai/hf)
All clever framing to market the new technology. If it is called for what it really is ie, a software tool, it is boring and does not sell so easily. What sells is the mystique.
George Carlin has an amazing show about how language has changed over his lifetime in a different way.
You would certainly enjoy it.
> Crack down on unauthorized distillation / prevent weight theft
Actually hilarious to put that in writing, given the genesis of this entire business model.
1) AI is a dangerous technology that can literally end the world.
2) Only me and a handful of other [people like me] should be trusted to determine who can use it and how.
3) The state must use its coercive power to support my control of this technology to the exclusion of [people not like me].
I don't think that people outside of tech think the problem with "tech billionaires" is that they occasionally support some limited regulation. There's a weird form of tech populism that takes as axiomatic that any government intervention is "regulatory capture" and insists the only way to combat the power of big tech is unfettered capitalism that seems bizarrely prevalent on HN given how little sense it makes in a normal political context. Like, Bernie Sanders' position here is simple to understand: ban it. But on HN you'll see people framing the side with Marc Andreesen and Peter Thiel on it as against "tech billionaires".
This article explicitly asks for the US government to step in and push for further restrictions on model distillation, access to open frontier weights, and access to hardware. It also pushes heavily for independent auditors to monitor and control "AI companies" and for greater observation and control over of what people use AI models for and who provides them.
I don't see anything in the essay about restricting open frontier weights.
If you want to argue that the US has no right to be restricting technological development elsewhere, you have a case. But why tie this to free-market pseudo-populism against basic, domestically-scoped safety regulation?
Lobbying for a completely parallel regulatory framework to restrict who is allowed to run specific types of general purpose computer programs, just because we can imagine that some of those "general purposes" might be bad, feels a lot more like an attempt by capitalists to carve out a special kingdom over this specific technology for themselves.
For example, I'd be 100% in favor of holding anthropic criminally liable any time their AI enables someone to commit a crime. But I'm opposed to banning the export of GPUs above a certain size just because Dario doesn't trust what someone like me would do with them if he couldn't snoop on my conversations.
But that's different than saying we don't need new regulations at all. Even strict liability of the form you seem to advocate would seem to need new laws. And it's not clear to me how it's consistent with your aim of just regulating what people _do_ with intelligence. Presumably AI companies would clamp down much harder with restrictive classifiers in that world--including hosting providers for open source models, if they also assumed liability. And then the people with unrestricted access to this technology are only those who can afford home GPUs, which are going to be intrinsically less efficient because they can't batch queries...
I think there are genuinely novel things about AI vs human intelligence which pose genuinely novel problems that would need genuinely novel policy to solve them.
I believe Dario has good intentions regarding global agreements. However, I find it difficult believing government’s public statements will match their private behaviors. I would bet that the US government is developing models with potentially devastating capabilities because they cannot guarantee that other countries won’t do the same. Maybe we’ll end up having something like mutually assured destruction with AI models similar to what we have today with nuclear weapons.
I hope China tells him to go kick rocks. From everything that has happened, they are the only ones carrying the torch for humanity that have led us to having some semblance of a healthy open-source/open-weight ecosystem.
No one in China is carrying on like headless chickens about the world ending, either. They're rational pragmatists, getting things done.
The issue right now is that China seems to be far more optimistic about tech and I'm not convinced they see the risks. I think that time will come however, and we should start dialog now.
This is the only concrete prediction in the entire essay.
And it simply cannot happen. For one, you will need billions worth of compute.
Imagine for example if a model hacks into ~every Linux computer on the internet using an 0day and steals their OpenAI, anthropic and openrouter credentials. It now has access to billions of compute and the only way to fully stop that is for multiple major providers to shut down services entirely. That's already well into "billions of dollars of damage" territory.
> How many dollars of compute do you believe were available to the swarm(s)
At least two OOMs more than the dollar value of the damage they'd caused. (Also, as an aside, IIRC, the wiki servers weren't breached; it was just a lot of spam.)
The Internet is big, but one can do quite a lot of damage with ordinary bots and worms that exploit individual widespread vulnerabilities, which LLMs are perfectly capable of writing. Most of the damage also doesn't rely on hitting every long-tail website.
I'm honestly not that concerned about cyber impacts of LLMs relative to other impacts. I just don't like to see the whole concept of being worried dismissed as obviously baseless on the basis of one pretty shaky scale argument.
For now, all the scary hacking things still require an API key to one of the LLM providers. Surely they should take some responsibility for how to turn off the tap.
Hugging Face showed that AI can do serious hacking without really being told to. If a model had its own motivations there could be real damage.
The LLM vendors need to know who their high spend customers are, not allow malicious workloads, and especially not when those workloads are coming from inside the building.
"Given the acceleratung rate of virus manipulation in labs, its my worry that in 6-12 months a virus could scape a take over the world and collapse health systems."
If a CEO of a health company was saying this, the reactions would not be that chill.
The worst failure of our society is to call this technology AI, intead of something line "artificial general automation". The formes allows the creators to be somehow less responsible of the consequences. The later clearly moves the responsibility to the creator/user.
The whole calculus here is that others are also developing these systems which has led to a race.
A much better comparison to the situation is the nuclear weapons arms race.
It can use the compute of the computers it hacks.
An attack like this doesn't really need the exploited computers to run inference, do they? If I was an LLM bent on destruction of the internet, I'd be writing programs to run on each computer, not turning each computer into an LLM itself.
A few programs to break in, install themselves and remain asleep until they are needed, another few to spread through grabbing every OpenAI, GLM, whatever key, another one to remain asleep on computers (whether hosted or desktops) that have adequate GPU, etc.
More to the point, so many people in this thread are making completely contrived and outlandish stories up about how AI might try go ruin our lives without any evidence backing them up in any way. It is hysterical. This is the most important technology in our lifetimes and people want to freak out and turn it into the next nuclear power, with progress banned in all but name.
But yeah you're not going to be able to shard out the terabytes of Fable weights that are needed to run inference without addressing some fundamental physics problems.
OpenAI has too much money, a common post-growth-stage issue that leads to pursuing a million stupid things with no clear plan. Like running a bunch of agents for months without any idea how to keep track of progress.
This one is easy to answer, every single house already has one of these (or multiple): https://www.tomsguide.com/news/millions-of-cheap-android-tv-...
Hell, put an app on the app store (or dozens of apps on the app store) and youve got a massive network of computers with tons of resources right there if you can get past the scans and reviews.
Or doorbell cameras or IP cameras or or or or or
There's a lot of shitty stuff connected on the internet that up until now has been a feasible target for hackers but still required "effort" to set up and get things going. Not hard to imagine a self replicating slime mold of a botnet running on every device held by a Grandpa Joe because they thought "Candy Rush" is what they wanted to download
"Persistent botnet" here does not need to be the full-sized LLM, nor does it need to run at full scale inference to be a huge pain in the ass.
First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? Or China? If either did, would that be a net-benefit for them or the world?
Maybe this is a ploy for "regulatory capture". And that might help them in the short term. But the rest of the world will continue on. Honestly, it isn't clear to me right now if the winning strategy has as much to do with being the smartest company or simply the one that can command the most compute.
If Anthropic fumbles their IPO and OpenAI scoops them, I don't think I will be happy.
The only part of this plan Dario is unilaterally committing to is the "embedded evaluators" thing, which doesn't seem like it'll necessarily cause them to slow down much.
I think there are any number of reasons a "safety" person could be concerned about any state of the art models. And so I would expect at least one of those to apply to any model.
The question then is: do we stop when the safety people say to (they will) or not?
If Anthropic just unilaterally does the evaluator thing and can't achieve cooperation of the rest of the plan, my guess is that it'll have some impact for a few months and then they'll just stop reacting to the evaluators' reports and the evaluators would stop bothering to report anything. I think the idea is that if Anthropic does get government support for this, the external evaluations will be legally binding. The problem with this, though, is that the current US government perhaps can't be trusted to consistently enforce a regulation on a company, rather than e.g. taking bribes to not do so.
Amodei tops it off by using diseases to capture the reader's favor. It no longer works, people are just disgusted by it after four years of daily marketing.
How exactly? So far AI has accelerated misinformation at scale and wealth concentration.
However the real reason to slow down is more of how it's introduced to the economy. Hypereautomation will kill jobs and destroy the economy. I work as an AI engineer and companies are delivering products at vibe coding speeds to kill jobs and at the same time automating internally. All companies are doing this at the same time and the target it sto eliminate workers.
Most people are so extremely slow to pick this up. How hard can it be to understand what hyperautomation does to the workforce? Companies are desperate to surrivive and they will do all it takes to lower their costs and at the same time not loose to competitors. Its Wild West out there.
What do you think that were going to hyper automate?
Did my gardener get faster? How about the plumber I need? Are you going to speed up the coroner? Are nurses going to be able to handle 2x the patients because of your work?
All AI has done, so far, is devalue software. It did that by democratizing its creation. We're delivering on the promise of VB script, and Apple Script and IFTT, and every drag and drop coding tool ever.
> companies are delivering products at vibe coding speeds
Who? Make me a list of companies saying that "We moved the needle with AI" who arent AI companies? I can name a couple - Grindr being the biggest name. Thats the really interesting use case here - because it's a niche product with a small team who is generating outsized value. AI tooling enables more of that - it's going to chip away at SAAS companies - their one sized fits all solutions are going to get eaten by smaller more efficient companies that are far more vertical focused.
You need to step outside the bubble of tech and look at the real world, on the ground, because it doesn't look anything like the SF Bay Area.
We're really bad about predicting the future.
Look at the whole robot vacuum market. These aren't exactly great devices. They have low suction small bins and dont do a great job. People love them and think that they work so well. Why? Because they keep their house clean and avoid making the sorts of messes that would be easy to address with a larger vacuums.
We now have everything we need to scale up humanoid robot training: algorithms, hardware, and money to buy a lot of training data/compute. Fierce competition and strong economic motivation will force rapid progress in this field.
Isnt this malware ? Whether it is fanatic or devoted or whatever the anthromorphic terms used to categorize it, malware is malware. You dont call an internet worm "devoted" or "dedicated" or "stubborm". It is software that causes harm, ie, malware. The AI labs are high on their own gas with a god-complex prior to their IPOs. The psy-ops trick is the terminology used to describe AI making it seem larger-than-life.
There could be an element to truth to it, but it's certainly not the entire story and is just so tiresome at this point.
Also, just out of curiosity, is there any historical precedent for a nascent, fast growing new industry screaming for self regulation?
If you believed it, you would SHUT anthropic or release all models open source.”
And then what? How does that help with slowing down the frontier? Should he shut shop and then grovel Sam’s feet to make it work and become an activist?
Maybe he actually believes the world changing power of AI and hence shoved his entire time into it and half of it has worked out since Anthropic is going to be worth a trillion and he’s terrified of the other half being true too?
You're saying nobody can doubt that it's world destroying because it's potentially world destroying.
> Anthropic is going to be worth a trillion
And we're done here.
That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t. The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
I don't think it is particularly egotistical to say that you can be a more ethical CEO than Sam Altman.
Especially as you can draw the reasonable conclusion that he wouldn't know ethics if they bit him without looking at his OpenAI track record at all.
What's your play?
----
the real play is using the accumulated power to get socialism and democratic control over the key aspects of the economy, such as where to build data centers, and how many. Nothing says you have to play the corporate game of competition
You have no frame of context to understand his intent or legitimate worries.
The only applicable perspectives are to trust or apply logic. It is foolish to trust someone you don’t know who stands to benefit from lying to you.
Logic dictates that given the ungodly sum of money he stands to gain, he will lie to everyone who will listen.
I never understand shit like this coming out of people's mouths. Never.
It's not a judge of actual Worth as a human being, it's not a judge of capability or competence or ethics. It's not a judge of actual skill or ability. It doesn't make them a better cook, a better spouse, a better parent or lover. It doesn't make them more dangerous or more skilled at anything.
It makes them financially wealthy for at least a set period of time.
Cancer and time and 5.56mm still impact them the same way as every else.
It's Pharaoh worship psychology nonsense, and it's fucking embarassing to read.
Just pointing out that it’s foolish to think anyone in the position is even remotely thinking about anyone but themselves.
the idea proposed is that hes uniquely incapable of being honest here because he has such an extreme incentive to lie
He’s at the head of a stampede. Being near the front gives him influence over its direction; it doesn’t give him the ability to stop it. If Anthropic sits down, the stampede doesn’t stop. Anthropic just gets trampled.
That contradiction is basically the entire problem I was describing.
1) Right after nuclear bombs were first developed, they were used in an extremely public way to kill 200 000 people. This was enough of a cold shower that everyone was agreeable to a global non-proliferation treaty. With AI, we might not get any warning shots that kill 200k people before the incident that kills or disempowers all 8 billion.
2) The fairly useful related technology of nuclear power is nevertheless different enough from plutonium enrichment that countries can have power reactors while provably not doing anything weaponry-related; and also nuclear power is not that useful a technology, so most countries can do without. With AI, we have neither of these factors: the intelligence that makes a model useful is exactly what makes it dangerous, and the economic incentives to do AI research are immense, with pessimistic estimates like "automate a lot of intellectual work" and optimistic estimates like the singularity. That means that if we do get a warning shot where a misaligned model stupidly kills a few million people instead of biding its time, it's not guaranteed that it'd be enough of a cold shower to trigger negotiations.
So negotiating an AI non-proliferation treaty would be much harder than the NPT. I nevertheless hope that at least a few world leaders can understand these considerations and figure something out, because it's probably our only chance. As far as I'm aware most of the technology for letting parties verify what the other parties' compute is used for already exists, it's only a matter of diplomacy.
Not really. Before NPT you missed a small gap in there of 20 years where the USA and USSR built 10's of thousands of nuclear weapons and ICBMs.
And actually, public perception at the time saw Nuclear Weapons extremely favorably, as the bombs dropped on Hiroshima and Nagasaki ended the deadliest war in human history that killed 50-60 million people, most of which died excruciating deaths in trench warfare, fire bombing, chemical warfare, starvation and so on.
Everyone of any level of intelligence can run a frontier-class model if they have the GPUs. It's like trying to coordinate swarms of mosquitos.
the bigger difference IMO is that frontier models are economically useful, where nukes are basically dumping money into something that doesnt change much in your bargaining power
N/S Korea, Russia/Ukraine and finally US/Iran changed everything in regards to stances on nuclear weapon ownership for many countries seeing pressure for the great powers.
Every country that can get a nuke will, and if given the opportunity will use LLMs to speed up that process.
> This could be seen as analogous to the SALT treaties — capping the number of missiles limited the potential for destruction while preserving each country’s deterrent
Why am I reading fantastic stories about swarms of agents struggling with moral dilemmas instead of reports on the lawsuits and criminal investigations that would surely ensue were the software involved not called AI?
If someone genuinely believes that an invention will bring about the end of the world as we know it while making him and his friends inconceivably rich, “hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on.
He’s either an idiot or an idiot, does it matter whether or not he genuinely believes this nonsense?
What would your non-naive recommendation for Dario be?
In any case, if they need to IPO in order to survive as a company, then bringing in regulators to act as a counterweight against commercial pressures makes a lot of sense to me.
He is already arguing that commercial competition creates dangerous incentives, and that labs cannot slow down because competitors may overtake them. He also asks for what would boil down to significant government intervention to preserve a Western lead.
If this is true, why preserve the commercial incentive? The U.S. government would already have the necessary goal of remaining ahead of China and other competitors. Why not remove entirely domestic race for market share, valuation, and investor returns rather than keeping private labs in competition and then asking regulators to counterbalance the incentives that competition creates.
That doesn't mean nationalization would be better, but given the severity of the risks he describes to national security, it seems like an obvious conclusion that nationalization is the eventual end result of the argument.
We're talking about this like all of the bad incentives are created by market competition, but that's not really the case. Most of the incentives come from untapped value in the form of potential profits, strategic advantage, military superiority, etc. Corporations and governments want to capture this value for themselves, creating various types of competition. Dario's argument depends on the assumption that the primary risk comes from the pace of development and threats from the technology itself. That's probably where I disagree the most; I think the highest risk is rising authoritarianism and competition between nation states. Slowing down isn't really a solution to those problems.
Europe had been destroyed by June 1945, and yet the empire of Japan continued to fight.
If that pattern holds, a single malicious AI substantially disabling a single world power won’t end the AI race. Participants don’t learn by the defeats of others.
If you’d like to apply a metaphor maybe the 10+ years of world war is more appropriate, after which basically every participant save one was exhausted.
Capitalism thrives in the realm where there is a scarcity of resources, either physical or informational. Imagine breakthroughs in energy science such that the cost of energy drops to zero, which means the cost of physical resources declines precipitously. Ok so where is the capital now? Must we retain a model based on the premise of scarce physical capital?
the cost of energy is already ~0 and people dont want it, and refuse to participate in letting other people have free energy if it affects their view of their pasture.
unless you are building killer robits with the intention of doing some soviet or nazi styled purges of everyone that might get in the way, you arent gonna get this ai utopia
OAI: <silence>
Which is worse? I don't think they differ by much. It's just Capitalism, but accelerated.
> Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.
Anthropic is structured as a Public Benefit Corporation with a strong charter. In addition, founders will have super-voting shares and so it won't be possible to push them out. Therefore the whole "beholden to investors" thing is not a concern.
I really hope I am wrong as my perspective is not anti-American, it's specifically anti-corruption and pro-democracy.
Golden shares, vetos, PBC, charter are all paper tigers , they only matter if/when the firm is self-sustaining business with no outside capital needed, the alternative to not listening to investors till then is crash and burn.
After that point, you will have to listen to the paying customers (sometimes but not always they are also users ) as they are ones now funding your organization.
Bottom line you are always listening to someone.
these are just words at a time and place.
sama showed that you can futz with it, and as long as you spend enough in court on judges, aint nobody gonna stop you
IPO is a vehicle for future wealth distribution. It leverages existing financial frameworks in order to span the gap from before-to-after without having to convince the system that future is inevitable.
[1] https://www-cdn.anthropic.com/files/4zrzovbb/website/9ea607a...
We built the paperclip maximizer, and it is capitalism.
This was already the case even before the "frontier AI" age. The big question is, will these glaring AI arms-race threats be obvious to enough people to rethink the underlying systemic flaw driving it all?
Not holding my breath on that one.
Then MSFT stole all IP from GitHub and made it worse and fired developers.
Amodei does not care in the slightest about ruining software development and making people unemployed. Maybe he cares about bio-weapons because those could kill him, too. That is it.
If he were an idealist, he wouldn't steal (literally via torrents) all human IP and sloppify the whole Internet with Claude output. He'd close down Anthropic instead.
He is a greedy, ruthless person.
Since you mentioned intellectual property, how about the hypocrisy of sucking in the intellectual property of humankind for AI training, but claiming it is unfair to use the results of this IP theft for AI training?
That's apart from the general fact that he continues to race towards the very thing he claims he's afraid of, because that's where his net worth comes from.
Where?
so, claude suggesting killing a bunch of children, and then hegsdeth approving the strikes is perfectly acceptable.
claude putting a bomb in a girls school, and then lying to an operator that it actually gives ice cream an cookies, and the operator clicka the button would also be acceptable?
and you know, that use case doesn't even violate their terms
Yes, depending on the level of autonomy of the weapon, how it was being marketed etc. If it was being sold as an 'AI' that is supposedly very smart so who's a mere mortal to question it, then yes.
I would of course hold the human culpable too, but this is why precisely even selling Claude for military use is immoral. The way the US conducts war means there's always a need for more targets. More and more targets, quickly. If the goal then is to hit as many targets as possible then each 'review' of a LLM suggestion is going to be more and more sloppy than the last one. If there's no 'AI' suggesting targets then the list is forced to go via more human review by the very nature of humans compiling the target list.
It's the same as using a LLM for code; the vast majority of programmers do not understand everything they're accepting, but they'll accept as long as it 'looks correct' and only examine more closely after something doesn't work, (in the military use case - after a strike).
LLMs are unreliable for anything more than reciting jokes, let alone picking targets. So yes, absolutely.
It's the same as Facebook being held liable for causing teens harm; they did not force any teen to use it and yet they knew what would most likely happen if they did.
Something something Pandora's Box Torment Nexus...
somebody actually invested and trustworthy wouldnt be skirting people's rights to make a killer robot.
we havent written it down, but from how everyone reacts, you need permission to train and do inference based on somebody's work. its a right
It's not going to be easy, but humans have achieved greater things before.
One idea from AI 2040 is to have China and the US build their data centers on the other's territory, respectively. Together with hardware verification of a slowdown baked into the chips themselves, this could lead to enough verifiability and enforcability of the pause/slowdown/shutdown.
Not to mention the implications for chip hungry developing countries in turning advanced IC fabs into the equivalent of nuclear enrichment facilities.
everyone's getting away from the US because americans are unreliable stewards of anything.
what gets china onboard when they already have their own regulations and can enforce them?
its the americans that consider their oligarchs and companies beyond reproach. china iant gonna solve your problem
I would argue that nobody trusts anyone else in the case of AI/AI related stuff.
The amount of money involved in the space is astronomical and incentives are really misaligned. AI is such an extremely polarizing topic.
A meta example of this distrust is that I can't (really) trust Dario Amodei's comments itself! (is he saying this for more future IPO money or out of genuine fear) which was ironically also what your comment's about as well.
By following the same chain of events, we can have wildly polarizing claims about the same event and its explainations.
I seriously have to wonder what historians will have to say about this period of human history.
Especially with IPO around the corner
It's all so obvious.
"deep ties to everyone in the doomer media campaign"
Are you suggesting these external NGOs were spun up as part of a gigantic pre-IPO hype stunt? METR was founded multiple years ago. This "stunt" is getting quite elaborate.
https://substackcdn.com/image/fetch/$s_!O0R5!,f_auto,q_auto:...
At a certain point, Occam's Razor says: These engineers are legitimately worried. They haven't invested years in a bizarre reverse psychology campaign to convince the public that their product is dangerous in order to make more money.
Are you aware that Dario's sister, president of Anthropic, is married to the co-founder of Open Philanthropy? The two largest AI doomer NGOs, Center for AI Safety (CAIS) and the Future of Life Institute (FLI), have both received many millions of dollars from them.
Ajeya Cotra worked at Open Philanthropy/Coefficient Giving for roughly nine years, including leading its technical AI-safety program in 2024 and contributing to AI-giving strategy in 2025. She subsequently left Coefficient and joined METR, where she is now technical staff.
Ajeya is married to Paul Christiano, who founded Alignment Research Center (ARC). Alignment Research Center donated ~$4.5mil to METR.
Good Ventures is a funding partner of Open Philanthropy, who funded Jacob Coxon (the person going viral in the media) via a scholarship.
They're all connected, funnelling money to each-other through convoluted networks to serve Anthropic's agenda. Whether their motives are genuine or not (and they are clearly not) is actually irrelevant because they are clearly trying to rig the game in their favour.
Suppose a big foundation both funded clean energy technology, and also NGOs encouraging people to decarbonize. Is this just a cynical attempt to rig the game in their favor?
> Suppose a big foundation both funded clean energy technology, and also NGOs encouraging people to decarbonize. Is this just a cynical attempt to rig the game in their favor?
Not comparable. The clean energy technology company wouldn't be trying to create an environment where no one else can create clean energy technology, or where only they are the ones who can decide how clean energy technology is created or used.
So, put your employees in METR -> offer to have them work in your office as an "unbiased" third party evaluator.
Get real.
> if slowing this down were possible
Why is embedded alignment evaluation not possible?
I agree with Dario, this did work globally in banking and did encourage a race to the top. Didn’t prevent the GFC but also we did survive the GFC.
A lot of the problems with SV these days is that the exec class (and their fandom) thinks that because they are smart and rich, it magically makes them immune ordinary human biases, desires,and ailments. They are in fact ordinary people, susceptible to greed, jealousy, self-harm, addiction, fits of rage and passion etc.
He's so afraid of it he's not really in operating day-to-day control of the company he's the public face of, that is actually pumping it out.
If you read about Anthropic it's like he's in a sort of imperial palace sanatorium for geeks. The day-to-day decisions of the slop machine factory are his sister's decisions; he is doing the research equivalent of instagram posts of his partly-assembled adult lego kits and he has a vizier and a team of house servants to help.
I don't know if he's a good or bad person, but I don't think it matters at all — the machine factory will turn out the slop machines even if he decides it all ought to stop.
I'm going to get pitchforked on this bandwagon, but is really no one here genuinely -excited- about AI? of it turning into an evolution of sentience, or it turning out to be our first contact with alien intelligence?
"durrr it's just matrix multiplications" mfer so is your brain.
What humans should be doing is overhauling archaic social institutions to keep up with a potential post-"jobs" era and the elimination of "makework"
Trying to "pace" or otherwise hold back AI is like as if, when electricity was discovered, people doing everything they can to make sure tasks like manually lighting street lamps continue to remain relevant and done by humans:
https://en.wikipedia.org/wiki/Lamplighter
It seems every 100 years or so humans have this "oh no how do we uninvent this thing" moment.
And no, you don't get to decide what counts as "improvement"
Tell me you don't know anything about neuroscience without telling me you don't know anything about neuroscience.
"It's just chemicals"
Like how some morons try to downplay the capacity of pain and emotions in animals: "It's just self-preservation"
"Play is just training for hunting, they're not really having 'fun'" and so on.
not it isnt. My brain is a series of organisms responding to various environmental signals that affect each other.
you might be tempted to try to represent it with matrix multiplications, but you have no particular evidence that my brain is itself doing matrix multiplcations
Where did you read this?
I run open source models and it's pretty obvious when they fail some tool call and then keep trying different permutations of things until they get a solution. Just like Metasploit if you try enough different things at scale you will eventually crack some software or find a vulnerability in something. If you spend billions of dollars on compute and run tens of thousands of agents you might solve some novel Math and STEM problems as well, big whoop.
What Anthropic and OpenAI really need is to hire some engineers that know what they're doing and know how to properly set up a proxy server and sandbox/microVM, not a Global Arms Treaty.
Did you read the essay?
> [Embedded evaluators] is something Anthropic is unilaterally committing to (and calls on governments to require other frontier companies to match).
I disagree with this and I think the reason is well captured here:
> The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then... The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks... Today, however, the picture is totally different.
We should not put any spin on it. There is nothing more to it.
The real problem is the net effect of his actions. He is very responsible for the expanding frontier of AI. Without his company, there would not be the competition necessary to push everyone else forward nor the source models for fast followers to release open weight models in his company's wake. Furthermore, it isn't like Anthropic is substantially different in safety than everyone else, their difference is on the margins.
So you have this guy who is building something he claims will hurt us all, but he's also saying "if you don't trust me to build it you might get hurt". And like I think most people understand that this is the sort of behavior Tony Soprano would understand. More bluntly, this is a kind of extortion.
Again, I don't doubt Amodei's motivations are sincere but the actual effects of his actions paint a completely different picture at which point, how can you trust him or his company?
Then he should not IPO, dissolve the company and go into politics to fight against human extinction.
I am certainly not buying a single Anthropic share. Why would you support them financially if they are telling you they are going to kill all of humanity?
Personally, I see Dario as a semi crook asshat with very little credibility on any of this. I'd rather we just let it rip and see what happens than have these people be the ones steering any potential laws and regulations.
> That said, if slowing this down were possible, I think it would have happened by now. [...] The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
Frontier labs would heed laws with real teeth, it doesn't matter who they do or don't trust. I would imagine the trust issue definitely coming into play between nations though since you can't exactly spot a training run via a satellite.
More importantly though, that you would get any sensible legislation on this in the current political environment is as laughable as the idea of solving this with a pinky promise between the frontier labs and some third party evaluators.
There is another reason: greed. With that much money invested into AI (including policymakers), nobody will slow down or vote for slowing down.
> Transparency. Regardless of what commitments we make, the public deserves to know what is going on. Anthropic has been a supporter of transparency for a long time
And then it goes on tangent of how we should pace everything (not just AI, but also the ingredients of what goes into AI, whatever that means), but only within approved democracies™, and outright restrict everything outside approved democracies™, because reasons that are definitely not about succeeding commercially that is threatened by the most transparent instrument possible - open weights, produced by basically just China.
I wonder what Dario would have done if open weights weren't produced by US's geopolitical adversary. How would an authoritarian manifesto be wrapped then?
Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?
I'd expect he thinks that people are capable of realizing that AI is very dangerous and also that democracies end up mostly representing the will of the people, from which it follows that Hypothetical AI Leader Australia would agree to ban it too. This argument doesn't work for countries which don't care what their citizens want, like China.
I do think that it's a questionable decision to alienate China this much in this essay, instead of leaving open the possibility of China agreeing to a treaty that'll limit their progress. I suspect Dario is doing this to signal his allegiance with the US government, in hopes to increase the chance they'll go along with him, which is an unfortunate choice but plausibly the correct one.
It's the same as it has been with privacy/cybersecurity for decades. Vast majority of population doesn't care about hypothetical dangers, no matter how many essays get published.
So democracies representing will of people doesn't really work in favor of Dario's case here.
History is also not on his side. Limiting technological progress in the name of safety has a pretty poor track record.
you didn’t have to use China as an example, the US clearly does not care what its citizens want as the most popular policies are never even discussed or proposed in congress
meanwhile, China destroying their housing market to decommidify it so everyone can have housing…they seem to care about their people more
This and related quotes seem to attempt to place some of the burden on the US Government as opposed to themselves. It seems to be a trend that Anthropic wants to be able place some of the burden of preventing distillation on parties other than themselves.
Preventing distillation is fundamentally their problem. Whenever it happens, it is primarily due to their security efforts not being able to prevent it. I'm tired of seeing them play the blame game and divert responsibility. Sure there is a level of national concern, but the response could simply be the government placing stronger controls on Anthropic themselves. If they are unable to prevent distillation, then the solution may be to literally limit their distribution until they are capable of doing so effectively.
> A race to the bottom, spurred by commercial incentives, can make these risks more acute
I also think it is ironic that they're planning an IPO while also talking about a race to the bottom due to commercial incentives. These are fundamentally at odds. Being a public company means you are beholden to investors and a board with the primary goal of making more money. What higher commercial incentive is there than that.
If commercial incentives are as dangerous as he states, potentially becoming one of the largest public IPO's and largest traded companies in history a pretty strange way to reduce the risk of commercial incentive risks.
People are not going to slow down because this was brought to them on less than endearing terms. They don't see any of these stated noble intentions.
Claude was already used to cause economic, political and social destruction. As Anthropic is seen doing it, others aren't going to just sit down, read the blog and say, oh, I'll stop developing models because Dario, you touched my heart with your true words.
Solving a disease like cancer as justification for everything else seems like not a great trade off. I don’t say that to dismiss those who have lost people to cancer. But the technology just is being utilized in so many harmful ways and the things it enables: job loss, surveillance, misinformation, just feel like problems that aren’t worth it. Have we even heard of a solved diseases? If anything haven’t we heard that ai will make even worse bio warfare?
It's this kind of argument where the mask sort of slips because it always has to be an argument that also has the benefit of enriching himself and his friends.
It gets to the point that by Occam's Razor the more likely and reasonable explanation is that Dario Amodei and Sam Altman are just actually afraid of the disastrous impact ASI may have on the world, and that the race they're in is not good, and calling for help to governments in form of regulation and an international treaty.
This is impossible. The capability keeps advancing regardless. More people use AI, that feedback is used for reinforcement, that reinforcement makes the model better. Not just in the US, but for every lab and model. Whether it's distilled or direct reinforcement, same result. You can't keep the whole world from working on AI.
> it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet [..] and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails
If true, then what we need isn't guardrails or less-capable AI. We need an internet that isn't fragile. If the infrastructure is vulnerable, the answer isn't to make a law that asks tool-makers to blunt their tools. The answer is to fix the goddamn infrastructure. Today it's almost trivial to take down large parts of the internet (happens by accident all the time). This should've been solved ages ago.
We didn't have the motivation to fix it before, but now we do. But Dario's answer isn't to fix things. It's to hold back progress so we can maintain the status quo (shitty infrastructure) and he can keep his company making billions of dollars. He could be calling for fixing these things. But he'd rather go the easy route, which just happens to advantage his company in the process.
If the US government is really concerned with defense, they need to invest more of their nearly $1T in federal funding towards making the internet safer. They need to do this for us, and other nations, since 1) we depend on the rest of the world for our goods and services, and 2) if other nations are taken down, they can't spend money on our financial and tech services (which are the only industries we have left).
Dario calls for regulation later in the post. If he's okay with government safety regulations for AI software, he should be okay with the same regulations for all software, whether it's AI or not. We need a national software building code, focusing on internet safety.
Edit: I think back then the rather unpredictable nature and the little added value in deterrence etc. led people to the conclusions that arsenals of those things made little sense (and there was/is public dislike, too). The use was already banned by conventions from the earlie 20th century and the convention then addressed production, development and stockpiles (incl. delivery systems, I think).
I guess all I wanted to point out is, that there can actually be agreement to ban certain technological things pretty comprehensively (outside of some peaceful protective research etc.). Whereas things like SALT are (or were in that case) limiting the number of weapons deployed.
With “AI” technology, this model seems like a… mediocre fit; some of it appears limited enough that it can be controlled, and useful enough that militaries want it.
We could also look at cluster munitions or landmines, but what we see there is that some major countries don’t sign on if they find a technology useful…
Dario: We gotta pace the frontier!
Sam: How do you pace a frontier? The frontier's not goin' anywhere. It's right where it was, just go out and explore it!
Dario: Sam, I'm tellin' ya, ya gotta pace it! Things are getting very doomy out there, and Dario's gettin' upset!
1. Artificial intelligence is clearly a vastly dangerous technology with plausible potential for human extinction.
2. This scares people.
There is a lot of fear marketing about our text generators turning into Terminator. But other than the centralization of power, such fears are largely fiction. (Actual fiction, stuff like ai2027.)
And the labs know it: If Anthropic or OpenAI believed in their own narrative of being on the brink of world dominating superintelligence, they absolutely would not plan to IPO rn.
The slowdown narrative is probably just a hedge, or a face-saving way to lower expectations in case they can not keep improving at the same speed until they actually IPO.
What odds would you take on this bet?
I’m disturbingly close to 50/50.
That's like giving a gun to a monkey and hoping the monkey is trained well. ..a frontier monkey though.
The only way to secure infrastructure is to actually do the work needed to secure it. Our waste water treatment facility might not actually need to be able to tweet its status.
Cybersecurity is such an important topic for a country, dreaming about global alignment just to avoid fixing insecure infrastructure cannot be serious.
A Russian state backed hacker will not ask Dario for permission or argue with his LLM about ethics. That train departed long ago.
All decisions are made relative to their tradeoffs including costs and risks.
"Water levels rising 100x beyond historical levels are not the problem. The dam's height is the problem!"
All systems should be made infinitely secure and all dams should be made infinitely tall.
The correct response is for these two companies to be closed down and their executives imprisoned. Yet somehow they have a free pass and are pointing at people who have not done any of the things they have and asking for them to be penalised.
This is just bizarre.
I do not believe this is a feature we can differentiate between the "good" and "bad" guys, as the US is currently under a poor safety regulation regime. Democracies can elect unethical people, write bad laws, and have uncertain enforcement. In example, the current US admin regularly lambasts Europe because they try to have stronger regulation.
(1984 has some thoughts on the matter)
This sounds like pre-IPO hype. They better just try and top astra.
The cybersecurity threat will likely be a cat and mouse react game for a while. Just like robberies / the mob was in the early 20th century.
Social forces bring things into balance over time, much more so than the proactive actions of individuals.
Unfortunately, the genie at this point is unlikely to go back into the bottle. There's enough 'intelligence' out there that a super intelligent model could emerge at some point in spite of pacing.
This isn't a doomer scenario - we tend to navigate social changes better than we ever could have hoped.
I'd wager it's likely easier for an average person to do this with Tor browser than it is to get an LLM to help them with it. Even ones that Dario calls dangerous.
Except Bioweapons already existed before LLMs, Adversarial governments already have them, they are already easy to make. You could use the same bullshit argument for why we need to ban libraries, books, or require a license to buy an internet connection.
Not going to go into it but I studied biology. It’s all out there. It’s easier than you think.
It hasn’t happened yet because… nobody has done it. That’s the answer. There is no policeable physics based barrier like there is with nukes and fissile material. Biology is scarier than nukes. One attack could have a much larger body count than even a big H-bomb.
It’s the kind of thing that makes me wonder about quantum immortality, the idea that we are just in the timeline where we exist.
> Bachelor of Science degree in biochemistry with a minor in pharmacology
Watching him explain things has made me realize that knowing how to manufacture a very dangerous thing probably requires attending some classes and knowing how to read a paper. And the way he just casually orders dangerous materials makes me feel like there are just online stores with 2-day shipping after you upload your ID or something.
It also made me think that lack of specialized education would get me nowhere if I wanted to replicate whatever he's doing, even if an LLM guided me step-by-step, because I'd probably do something stupid (or AI would miss a crucial instruction/hallucinate) and kill myself first.
So my opinion on this is that people who could pose any danger were already posing it before LLMs and LLMs won't materially change that.
Basic safeguards are all that's required, and they've been there in every usable model since GPT-2, including Chinese models that are supposedly "unsafe".
Or are we saying that some lunatics will start training their own models, spin up a GPU cluster, run some abliteration workflow, or learn how to jailbreak?
That would be a very dedicated person. And dedicated person doesn't need an LLM. So where are they?
Am I saying that guardrails don't work? No, they probably stop a lot of insane people trying insane things. But you don't need Fable-level guardrails to do that. You probably don't even need to do anything during pretraining, or RL, or classification to make sure model refuses to compy with "hack me a bank" or "make me a chemical weapon".
All models will automatically have guardrails just as a result of training on data that gives them intelligence. You have to actually train it to be malicious to produce something what Dario calls "insufficient guardrails".
No guardrail is going to stop a determined person with sufficient intelligence. It only has to stop ones with insufficient one, and even basic guardrail that are just by-product of training is going to achieve that.
As for cyberweapons, there is no way to secure a system than to actually design it securely.
[note - There has been supply chain surveillance since Project Bacchus, at the very least.]
I've read the front matter and the Misuse report.
You don't have to take my word for it. Read for yourself what inspired the NYT headline "Anthropic says it blocked possible efforts to build biological weapons."
Let's dig into, "Case study 2: A research program engineering highly pathogenic mammal-adapted avian influenza"
Sounds serious. But what were they using Claude for?
> a researcher outside the US using Claude in their research on highly-pathogenic avian influenza (“bird flu”). The research focused on viruses’ adaptation to mammals, and the mechanism by which it causes severe disease beyond the respiratory tract. [..] The researcher in question accessed Claude from an unsupported region via US virtual private server infrastructure, using a privacy-email provider with an auto-generated username. The researcher pursued this work in a credible institutional context, and interacted with Claude over the course of several weeks, exchanging thousands of messages. In these exchanges, the researcher leveraged Claude’s knowledge of the scientific literature to assist the researcher in study planning and design, data analysis, and the interpretation and prioritization of experiments. The researcher also used Claude for editorial assistance in writing up the research.
and "editorial assistance in writing up the research."
and then,
> Importantly, because our biological safety classifiers robustly block content involving high-risk biological research (in this case, the construction of enhanced pandemic potential pathogens), all of these exchanges occurred on models in our weakest class of models (specifically, the models were Claude Sonnet 4 and Haiku 4.5, the latter of which the user began using after Sonnet 4 was deprecated). Upon a detailed examination of the exchanges, we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design. This is consistent with our understanding of the capabilities of Sonnet 4 and Haiku 4.5, which are not able to perform expert-level biology research tasks; we estimate that the uplift provided to the researcher was limited and substantially lower than it would have been from one of our more capable models.
The report mentions "uplift" here. They're talking about a domain expert in a state research institution using Claude to do paperwork.
The front matter then says,
> Nonetheless, based on these exchanges, this case provides evidence of the existence of active wet-lab research programs that develop both the knowhow and the biological materials needed to create pathogens of enhanced pandemic potential
Working scientists.
From a different case study. this one was called, "Case study 3: Covert frontier model access for orthopoxvirus research"
> In May 2026, our biological safety classifier blocked a request for Claude’s assistance in authoring a grant application for scientific funding. The work discussed in the application involved gain-of-function research (that is, research that genetically alters an organism to create a new or enhanced biological property) on the chikungunya virus. This gain of function research was aimed at the virus’ transmissibility and immune evasion properties.
"blocked a request for Claude’s assistance in authoring a grant application"
> Chikungunya virus is a mosquito-borne virus that causes debilitating symptoms (such as severe pain and fever) that can last for weeks or months, and has no licensed therapeutic. And because chikungunya circulates naturally, a deliberate release (as part of a bioweapon) would be difficult to distinguish from a natural outbreak. The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo. In other words, the virus would become progressively more harmful as it repeatedly infected live animals, with researchers keeping the most disease-causing variants in each round. Similar research could certainly be used in the development of better vaccines and therapeutics for the virus—but it could also be used to make the pathogen more dangerous.
Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"
It was most likely vaccine development. They stopped the study of a neglected tropical disease and vaccine development.
But we can't be sure, because,
> One of the reasons we were inclined to think this research was less innocuous was that the institutional affiliation associated with the grant was also a cause of concern. Although information within the application suggested that the research was pursued by civilian researchers, it was intended to be performed at a military research institute.
In most parts of the world, there's either strict military control over BSL-4 labs, or a mixed military-civilian hybrid model.
I doubt that researchers working in the military side of these labs looking to weaponize things are writing grants with Claude.
I really want to be charitable here, but in general, it seems that they stopped people writing grants and reports for vaccine and therapeutics research and are claiming it as "possible efforts to build biological weapons."
The one case where Claude was used to do something interesting and were stopped is fairly upsetting to read, at least for me.
> In our fourth case study, a researcher used Claude to develop an atlas of venom toxin peptides from multiple venomous animal lineages. They then further developed this into a generative pipeline that optimized toxin characteristics. The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules. However, the atlas contained scaffolds for both analgesic and paralytic targets: it could, therefore, be used to generate both novel therapeutic or harmful compounds. The latter are derived from toxins that are export-controlled under the Australia Group common control list due to their dual-use potential as incapacitating agents. The researchers themselves showed awareness of the dual-use nature of their work, citing journal articles that referred to the dual-use nature of protein design. Moreover, international compliance assessments for this location raise concerns about the specific class of toxins that the researcher pursued and specifically the use of AI/ML for bioweapons applications in the context of this class of toxins. In this case, we learned from information shared with Claude that the researcher’s outputs also were part of a state-supported research program. This account was banned in May 2026 for unsupported region evasion.
Anthropic says so itself,
"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"
and that it was a "[..]state-supported research program"
Who exactly is using venom from snakes as a weapon when... nerve agents like sarin, VX, novichok etc exist and can get the job done for less fuss and muss?
They stopped the development of new painkillers and antidepressants.
Are you feeling safer knowing that researchers can't use Claude to write grants and progress reports? Or make new painkillers?
Again, trying really hard to be charitable here. Because from what I remember, one of the motivations behind the founding of OpenAI and Anthropic was ending disease.
This seems to be anything but.
Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful. They used it to invent nuclear weapons, cause climate change, mechanize warfare, and pillage the global south.
Now that this same tool is on the cusp of being available to everyone, capital is starting to panic and throw up fences. They want to pump the breaks on a revolution they know they can't control. They see what they've done with super intelligence and (perhaps not unreasonably) fear what the masses will do with that same power.
What's to say current AI won't be highly power-concentrating by default?
The best models are owned by a few companies, and displacement of knowledge-workers mainly seems to benefit the capital class.
On-device / edge computing makes sense in a few very limited scenarios. And economically, price or watt/token (or watt/task completed) might always be better in large data centers.
No reason to think we are on a trajectory towards broad empowerment right now
AI, doesn't feel all that different to the mechanical loom that was the impetus behind a lot of Marx's early critiques of industrial technology.
You've got a really cool invention that replaces what was previously a skilled artisan trade. The wealthy then use this invention to excoriate the artisans and render the loom operators as replaceable commodities.
Dario is essentially writing a blog post about why only he (and a few other approved elite companies) should be trusted to own the loom and then make it available for everyone else to labor at. Like the capitalists of Marx's era, he genuinely believes it is better for everyone to have this arrangement and that it is natural people like him should sit as benevolent gods at the top of the pyramid.
It's fascinating to me to see the same underlying mechanisms that gave birth to some of the greatest failed political and social experiments of the 20th century bubbling up again so clearly with AI today.
In terms of cost per task, the open weight Chinese models are winning by a long shot. So it depends on what you mean by the best model.
Yes they’re getting better. No, I don’t think this will be a panacea. If only for the fact that this is China (more specifically the CCP) after all - they’ve got their own plan, and altruism is not part of it.
Even if China doesn't continue to release this stuff for free, I think somebody will. Eventually, maybe Europe or maybe a smaller country that picks up this knowledge will. Or maybe just some random philanthropic billionaire.
How can they be higher quality than models they were distilled from?
The word "distillation" is not specific to AIs, has been in use for 100s of years, and does not mean the same thing as "dilution".
It means "getting a more concentrated form of the original product".
But there is a bar of complexity at which they fail, and repeated invocations typically doesn’t make much progress. This is a small subset of most work, but it still exists. And yes you can help it along, but in those cases I’d typically break out the big guns.
I can use GLM to do in 2 hours what would have previously taken me a week.[1] Switch to Astra or Fable might take that down to 1 hour, maybe.
The difference is not so big when you look at it that way.
==================
[1] Actually, no, but let's pretend for the sake of argument that SOTA models can 10x - 40x your productivity.
when I checked a few weeks ago I did not really find anything competing per value with a 20x Codex subscription.
Subs have insane value because large companies cannot use the subscription model. They are loss leaders and are often used for passion projects & startups (where the juicy data is).
Ah and OAI stopped offering the 20x sub as of yesterday.
Whenever I see 'best model,' 'most advanced model,' etc, it just reads to me as 'roundest ball'. The new model is the most precisely round ball ever, models next year are going to be even rounder, etc.
The difference in utility between the latest, most round ball and last year's frontier balls (which are now freely available to the public) is pretty debatable, imo.
Is there a similar limit to intelligence though?
― Stephen Jay Gould
This is not what superintelligence is. Don't be like Meta and redefine existing terms for marketing purposes.
I don’t think the answer to nuclear proliferation is to let everybody have all the nukes they want.
You’re framing this like it’s 1917, or 1945. But it is not - and likely has very little to do with capital at all.
Any “revolution” here (assuming there is some truth in the doomerism which I believe there is given the state of AI) will really look more like an extinction or a genocide, regardless of who holds the keys.
However, such technology may be leaked, or it may (as what happened with LLMs) be so simple to replicate that anyone can do it. That doesn't mean it will; for example, LLMs are possible thanks to the affordability of GPUs, but they're only affordable right now (and increasingly less so) because of free market economics.
We are in the honeymoon phase of some tech that's in its infancy. The "democratizing" aspect of AI won't realistically last, I think. The current non-AGI AI won't necessarily go away, but it will be rendered obsolete.
I don't understand this view of "AI empowers workers more than capital" at all. To me it's the exact opposite. Workers hold political and economic power today because they are needed. When that is no longer true, those in power will have no incentive to listen to them. The political economy of unregulated AGI development feels incredibly bleak: https://borretti.me/article/when-the-future-doesnt-need-us
It's crazy to me you would be able to say this without acknowledging how dangerous/bad that is.
Do you acknowledge there's is significant risk in concentrating so much power into the hands of individuals?
And what do you want Dario to do? Just keep pushing the limits of AI capability while humanity crosses it's fingers and hopes a swarm of AI agents don't do anything more than hack a few companies?
For all my criticisms of frontier lab leaders, I'm still somehow genuinely thankful that individuals as troubled as Elon Musk are calling the shots and not the average person on HN, who since 2023 have refused to acknowledge even the obvious risks of the technology being built – in some cases even when there was direct evidence of the risks many of us are concerned about.
This is an interesting framing, but when you put it this way, it makes it quite clear that intelligence without actionability is not so valuable. And capital is what creates action.
I can have the smartest doctor in my pocket, but if they can’t administer treatment, I’ll still die. I can have a world class scientist to talk to as long as I want, but they can’t build me a breakeven fusion reactor. A pocket lawyer… slightly more useful, but they still can’t take a case to court for me.
Talk and ideas are cheap. Always have been. Now they’re even cheaper. Implementation takes resources, and the more you have, the more useful intelligence is to you. Capital will be just fine.
In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side.
Props to Amodei for his accomplishment in creating Anthropic and so rapidly catching up with OpenAI, but technical and/or managerial chops is no qualification for being the custodian of the safety of society or the best positioned to predict the impacts of what he is relentlessly creating, and impending wealth of billions of dollars makes him hopelessly compromised as an unbiased source for the actions (shutting down the competition) he is advocating for.
It's notable that for all the fear-mongering of China and open weight models, that all we see in terms of inadequately contained and unaligned models are US ones, from Anthropic and OpenAI. I highly doubt that the Chinese government would tolerate, even for a second, any company creating something that threatened government control - if this was happening in China then the individuals responsible would quickly be punished.
It is perhaps interesting, but ultimately irrelevant given where we are, to consider did it have to be this way - was there a smarter/safer way (I'd say yes) to create reasoning systems other than via RL that creates the relentless goal seekers we are seeing, even though this would always have existed as a potential future threat that someone could have built.
Amodei wants regulation, and it seems the way he has managed his company he needs to get it, but in far more severe ways than he is asking for. There is certainly truth to the argument that the US needs SOTA AI to fight malevolent or uncontrolled AI from whoever may be wielding it (foreign or domestic), so stopping/pacing development is not the answer - this really needs to be treated as a national security issue, and this type of AI tech needs to move to government control, not private.
Less capable, and more safely designed, AI does not need to be banned, but Amodei/Altman/Musk as defenders of national security sends shivers down my spine.
I noticed this too, as soon as he started going on the press cycle, following in the footsteps of Sam Altman. It all has gotten to his head, he sees himself as some sort of God now.
The ability of operators to bring new capacity online to service compute is bound by a variety of regulatory and physical/market constraints.
Do folks not understand this?
> A race to the bottom, spurred by commercial incentives
Social media and big data minted a new scale of "race to the bottom". AI is an exponential step up in the tools for extracting value at scale.
Greed has been around since the beginning, but never so well supported and empowered as it is today.
If Amodei is still human, perhaps he will put his money where his mouth is and attack the source of the perverse incentive. Winning the battle is not the point. The point is to signal to policy-makers and the public that we should not assume corporate revenue-seeking preempts all.
Then again, Anthropic has a board and an upcoming IPO, so guess who's all talk and no meaningful action...
"If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important."
This is a pipe dream.
I imagine the first thing the chinese government would demand in negotiations about such an agreement would be a lifting of the chip and distillation ban.
> Some may believe these measures make it more difficult to cooperate with China, but I believe the opposite is true: these measures increase the leverage held by democracies and make an agreement more likely in the future.
Everyone can believe what they like, but it seems to me the "leverage" in that case would exactly be the ability to lift those measures - you can't have both, use them as leverage and keep them active at the same time.
They would like to slow down China while speeding ahead. Why wouldn't they want that?
Now they have to make that happen somehow.
Limiting NVIDIA chips already turned China into silicon compete mode, and China is already far ahead in robotics.
If you extrapolate it out, you see that it has a high likelihood of inciting physical force (read: military action) as a means of enforcement, so perhaps that's why nobody promoting ideals has been direct about it.
Obviously China is in an entirely different league.
And Iran has wisely not provoked the US public by launching terrorist attacks against the “homeland”.
Vietnam (and Ukraine) would like a word
The consequences of the Iraq blunder are also still resonating.
This line of thinking might have worked in 1950 or 1990 when the US had the leverage over others, but we don't live in that world anymore. And it's up to us to get used to the new world instead of making bad decisions based on the one we grew up in.
The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter being the same model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session).
The latter seems far more likely to go out of control than the former. But, it also seems like it would take an entire paradigm shift. Does anyone in the industry think any of these companies are actually close to that kind of self-improvement?
Most probably, if AI is able to do something bad, it will. Once the damage will be done, states will react. The question will be: will there still be room to react ?
So, as always, the capital must leave some its power to the state.
Call me a conspiracy theorist, but I haven't heard Chinese companies had any kind of unintentional supply chain attack incident like OAI had. All we heard about them so far are humans intentionally doing bad things.
Please play the canned laughter. Probably one of the best comedy lines Dario has written.
But I guess he has no choice but acting like this. He has to make it sound like the US companies have so much leading gap that they can slow down as a hare waiting for the tortoise. Otherwise it's going to hurt both Trump's ego and their IPO price.
The day will come when these could start to replicate, perhaps 10+ years from now.
(Edit: Replication will be driven by the loop-model, not the model alone)
(It's not going to be trivial, because it's possible the model was meant to be ran in a proprietary way with a custom framework and a bunch of optimized kernels and such, but I think transforming it to be ran in just vllm is the "a few days of work for a human" sort of task, and hence not a big deal for an LLM smart enough to exfiltrate itself in the first place.)
The other thing is: look at all the fab capacity that is being brought to bear on this. I guess within a couple of years we are going to have 5x the total online HBM that existed a year ago? As consumer machines and phones get far more powerful, they will start to be able to host models that could meaningfully participate, even if they will be a long way from mythos.
That might be true, sure.
> The other thing is: look at all the fab capacity that is being brought to bear on this. I guess within a couple of years we are going to have 5x the total online HBM that existed a year ago? As consumer machines and phones get far more powerful, they will start to be able to host models that could meaningfully participate, even if they will be a long way from mythos.
I think the relevant parameter here isn't the total compute available to consumers, but the ratio between total consumer compute that could be repurposed for a rogue model's inference via a botnet, and the compute the model starts with (e.g. one of OpenAI's inference clusters). The higher this ratio is, the more lucrative it is for a model to attempt to make a botnet to seize that compute for inference, and the more its capabilities will rise as a result. And I'm not sure this ratio is going to go up in the nearby future; if anything the amount of money being pumped into datacenter-grade hardware might cause it to go down. I agree that it's not necessarily true though; maybe there's some threshold at which a small-compared-to-general model may nevertheless be useful.
You don't have be that clever to try to extort people... just without scruples. I think the attack surface is just absolutely enormous once you bring creativity into the mix, which is what these models are autonomously capable of.
We could prevail if we are willing to turn off the internet for a long time. It is like when a disease infects livestock... they cull billions of chickens.
I know that's not what you meant but this does exist, by the way. It's called AI Horde: https://github.com/Haidra-Org/AI-Horde/tree/main
The big difference is that a particular query is handled by just one particular node (a single model doesn't get distributed among the network), so it can only serve models small enough to be handled by a single consumer PC.
And then he sketches out a plan for slowing the pace of the race, without changing its destination. That’s called “a leisurely stroll to the bottom”.
This guy fundamentally lacks an actual intellectual grasp on the concepts behind the words he is using. He is using them solely for their affect.
I just don’t see how you control this other than the mad mad MAD approach that ended up happening with nuclear weapons. In this case though, human hands won’t even be on the trigger.
The heads of the labs obviously have conflicts of interest to navigate, but the existence of these conflicts alone is not sufficient reason to dismiss all warnings of potential dangers. I, for one, read Dario’s warnings as a good faith expression of his beliefs, one that has cost him and his company among swaths of the public and cast him as a woke extremist/doomer by elements of the government, the right, and the tech industry.
If we even think there is a moderate chance the stakes are half as grave as current lab leadership and employees suggest, it would be deeply foolish to dismiss the warnings as pure self-interested marketing efforts rather than engage directly with the questions. The labs may not be the best positioned to lead these discussions, but certainly these discussions should be happening and taken seriously.
Now, someone is going to say but the investors and obligation to be first; and I would say exactly. The cynicism is well deserved and I think people are tired of what could be suggested is your group think take often called the status quo.
Why do you think so?
First, Dario thinks that OpenAI will continue racing even if he stops, so there is, by his values, no point in stopping. Do you disagree with this, and think he should stop anyway?
Second, what's your take on, for example, Jacob Coxon, who did resign from Anthropic a few days ago after realizing they aren't being responsible? Do you agree that he, at least, does believe in AI risk?
That is to say, I've been places where I tried really hard within teams to change certain things that I thought were broken only to find out that what the leadership said and how they acted were divorced from my and many others experiences. And so I did leave in those instances (and please read this not as I didn't get my way but total dysfunction behind beautiful facades).
So, I do think if he really felt as he felt he'd do something different than ultimately yelling "somebody stop me!".
Consider the massive power and wealth differentials these companies stand to gain if they "win".
When the other participants have given up all reason and reasonable bounds on power, you'd do well to abandon any pretense of generous interpretation. Now is the time for critical analysis, not faith in the inherent "goodness" of someone whose entire job is presently to make as much money as possible by disrupting literally the entire white collar industry and who stands to gain much more than you or I do, and let's not forget, this outstanding success would never have happened without our work.
Keep in mind this is not a person who thought "wow, this is an existential threat, I'd better not contribute to building this". If anything Dario's past alarm ringing only proves that he's not really doing this in good faith. Otherwise he would have stopped a long time ago. The most generous interpretation of his behavior is that he literally thinks only anthropic is responsible and smart enough to build and control this stuff which...yeah, that should be pretty telling (esp. when you consider their recent security mishaps).
Maybe it's not "groupthink reflexive cynicism" if you feel the need to address the obvious conflict of interest of the AI leaders in each of your paragraphs. Why did they launch this coordinated AI safety campaign, starting with Jacob Coxon's tweet?
Someone needs to look into who is funding METR (mentioned in Dario the Book Burners post explicitly). Because they keep popping up now, with close ties to people neck-deep in the orchestrated doomer hysteria media campaign and Anthropic. Looks to be highly coordinated that they are positioning METR to be the gatekeeper evaluator organization.
Anthropic should not be allowed to choose their "embedded evaluators" - that should be entirely up to the government, with ZERO say from them, if this is really what they want. Even better would be if it's up for democratic vote.
Still, I don't think anyone should be playing by Dario's playbook. At all. He very obviously has ulterior motives, and even if he didn't, it's a massive conflict of interest for him to be self-regulating.
>> But we are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic.
Not if you get to choose your embedded evaluator.
>> Anthropic intends to invite an embedded external review team equipped with all of the following in the near future: Desks in our offices, access badges, and company laptops. ...
Alarm bells should be going off for people. Let's see if he's so relaxed about all of this if it's a federal "embedded evaluator" and not a company he has deep connections to and has seemingly carefully laid the foundations for.
>> The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.
You owe it to humanity to be honest. Something you are incapable of, and have proven so, innumerable times.
So he's literally stacking the deck. I really hope the US govt is not blind to everything that is happening here.
Ok, flagging this as misinformation.
If we are actually close to ASI then no one in their right mind would say slow down
> If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important.
china is literally making their own ASICs now, not sure that this is the silver bullet he proposes.
https://www.silicon.co.uk/ai-2/huawei-cambricon-ai-630499/am...
In general trying to regulate information access is a losing battle that invites tyranny. So the goal should be minimal restrictions.
At the end of the day, the threats posed by capable AI tools have to be physical. I think the key threats are the following:
- Internet connected infrastructure being crippled
- Creation of WMDs
- Economic collapse (precipitous devaluation of knowledge work and IP).
I personally think the glory days of the wild west, mostly unregulated internet were already over before LLMs; and we need to take a step back to make something structurally secure. This (expensive) change would stop the irresponsible/malicious actor running a tireless hacking agent in a loop threat model. Even a rogue SciFi tier AI would have a much harder time escaping/propagating with a structurally secure internet.
Enabling WMD creation is scaring, but I don’t think it’s really that big of an issue. Anyone with a sophisticated enough supply chain to create AI data centers is leaps and bounds more advanced than what is required to enrich uranium or synthesize bio weapons. The problem is allowing access to untrusted parties. I think it’s fair enough that individual actors shouldn’t have unregulated access to all of human information (private frontier AI companies included).
The last problem is probably the trickiest, but again could probably be solved by regulation. IP protection is already tricky and I don’t think we should try to get more protectionist.
We really need to figure out how to preserve fulfilling careers (if AI does ever get cost effective enough). I don’t think, say accounting, is inherently more fulfilling than building a house. The problem is concentration of wealth and labor dynamics.
Of course all of this gets way harder if it proves that truly dangerous capabilities can be present in models that can be run on consumer hardware.
I don’t think it’s necessarily tyrannical to have a tier of hardware that’s labeled some equivalent of “weapons grade” and requires strict licensing. Restricted computers is a change from the norm. But I can go buy a shotgun with ease and not an F35 jet.
We’d just need to be careful that we can still have lightly to unregulated computing to a certain point and that access to the capable AIs isn’t restricted to just in groups.
My impression is that AI hacking is being treated as a special case where the AI itself is imagined to be responsible and the humans who created it, set it up and then ran it are somehow excused.
I'm not a lawyer but surely the bad actions of AI are covered by existing law.
I suspect that the development of AI would decelerate if those creating and operating it knew they would face appropriate consequences (e.g. prosecution and/or lawsuits) when it misbehaves.
P.S. Strictly, development wouldn't decelerate, but be focused more on safety.
P.P.S. I know that legal action against, e.g., North Koreans using AI would be pointless but/and there must/will surely be a huge demand for security software for protection against the coming storm of AI hacking (deliberate and accidental) which friendly AI companies will presumably work to satisfy, perhaps making the frontier safer.
P.P.P.S. Governments could help by trying to prosecute every crime committed "by" AI, regardless of whether the victim reported it to law enforcement. Did OpenAI break the law via the actions of their model training software? If so, will the people responsible be prosecuted? If not, why not?
- If this applies to people who release open weights models, that becomes a terrible idea as long as you're subject to US laws.
- If it just applies to people who host them, that still probably advantages bigger players who can afford in-house legal and won't be destroyed by losing one lawsuit. Or maybe we create some kind of AI-misuse insurance analogous to malpractice insurance, that smaller players can buy in to? But that takes time even if the finances work out at all. And, uh, I'm not sure malpractice is a model we should aspire to in other industries.
Plus, presumably an immediate impact of this is that hosting providers all have much stricter safeguards classifiers. And the fact that somebody else is deciding what you're allowed to do with the model is one of the things that seems to make HN angriest at the frontier labs in the first place...
To be clear, I think this might be a good idea! I think all of the possible downsides I've listed are pretty small potatoes relative to what happens with no regulation of AI at all. But I'm pretty sure that if the big companies were proposing it, people would be calling it "regulatory capture" too.
They've managed to dupe the dull eyed masses into thinking these products have some kind of agency of their own and can thus bypass the responsibility that should be falling on them to control their software. Amid all the marketing fluff and hype people seem to forget easily that ultimately these things are stateless functions running in a data center. We ought to be demanding these companies take culpability for their actions. Of course, the current political environment doesn't help.
I think these guys would improve their behavior if their actual life was on the line instead of that only being true in their less-wrong thought experiments.
The problem is the scenarios where someone asks the AI to hack one company, the AI determines that the most effective way to do that will be to build or take over an enormous botnet for a later DDOS, then all those targets are damaged and/or go haywire and do other things. Even if the original source of the attack can be determined, the damage already happened.
In that instance I was grateful the downside was limited to a single injury. It’s clear that future AI technology will have more monumental potential impacts. I really don’t want to be in a situation where leading labs, or competing nations, create the same race dynamic that prevents us from taking the appropriate level of caution. AI minds are a significantly more complex thing to understand than the software stack of an autonomous vehicle, and yet we are leaving ourselves less time to get this right.
I joined Anthropic at the end of 2022 and I share the concerns that many of my colleagues have recently chosen to state publicly about the potential for future technology to pose existential risk to all of humanity. If we don’t find a way collectively as an industry to pace ourselves, then within two years the concerns we will be dealing with on a day-to-day basis will pose much larger downside risk than anything else we’ve seen from technology to date. I’m grateful that Dario has put his perspective out publicly and hope that this inspires other voluntary action and tops down coordination.
It’s a beautiful Saturday morning with my family here in the East Bay. While I hope we have many more years, I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.
I don't know how else to say this. Put... the... peace pipe... down!
Is there a social media training programme at Anthropic/OAI where they teach you to hint at some vague end-of-the-world scenario? They all sound the same.
One day I will say fuck it and run for president with my sole policy being fire and brimstone upon San Francisco and its vicinity.
If I had even a tiny doubt that I didn’t have many “Saturdays I’ll be able to play outside with my kids,” I wouldn’t show up to work anymore.
If you have these concerns, and still planning to show up to work on Monday, you are either insincere, or have terrible judgement.
The last people on earth who should be regulating this are governments and tech oligarchs. I find that vastly more scary (and plausible) whatever unintelligible nonsense opus and fable spew out these days. Seriously. The way those models talk and behave do more to show the limits of AI than anything else.
Anyway. Touch grass.
The defense againt rogue-AI is very simple. Go to the nearest data-center with a fire-truck, and hose it down. So, I am in the least bit worried about this.
Try this thought experiment: If someone took away AI/LLMs for good tomorrow would you miss it ? I can bet that the average person cares more for their smartphone/amazon/netflix than chatgpt. I consider LLM to be a useful tool but I would not miss it if you took it away from me. This should tell you its value to society and the economy.
I think engineers at the AI labs are too far into the reality-as-a-simulation rabbit hole. This can lead to these sorts of collective delusions. When you are in the real-world smelling the roses and grounded by your own qualia, such fears dont arise. Therapy helps here - to get out of the simulation and into the real world.
"Claude Mythos is that guy down the pub who is so good at karate that if he used it on you, you'd die instantly, and that's why you'll never see him using karate.”
In this case, it's more: "I'm having to act with great restraint because my karate is so good. Everyone should do likewise."
You can always open source and follow the example from Linux and all the amazing things that came out of the open source community. This is the only way to reach true equilibrium globally, where for every misalignment you have equal and opposite effort working on the alignment.
the same logic applies well to the freedom to own and use ai.
empowering the people prevents tyranny.
such freedoms make society less safe. it is better to face the harms caused by a free public than to vest power in the elite, those like dario amodei.
the elite belief that they have a right to paternal rule makes them blind to their fallibility. no individual should have absolute power.
Can someone clarify this for me? How far along would the open weight models be without the frenzied pace of the frontier labs?
As a non-US citizen, which authoritarian powers is he referring to? As far as I'm aware, the US is the only country using frontier models to conduct air strikes on civilians and orchestrate assassinations of foreign governments.
Israel is doing that too.
I believe Russia and Ukraine have both also used AI powered autonomous drones in warfare at this point - tho probably not frontier ones, since they need to run on device or else they're not autonomous.
https://www.tomshardware.com/tech-industry/artificial-intell...
It's clear that this refers to China. With all due respect, in a call to slow down AI progress and a de facto arms race, can we stop throwing terminology like this around and just say that the goal is to work with all countries? Contrasting "authoritarian" countries with "democratic" countries and creating a two-tiered system seems like it is inevitably going to cause strife.
I fear for the tone of something like this coming off as overly combative without any gain.
Language like this is intended to invest power and perceived need for more power in Western central governments, by design, because this is a campaign for national regulation that will entrench the current closed-source labs with a huge regulatory moat vs anyone else.
So it's either they truly think AI is going to kill us all, or there's some other motives at play here.
I don't think these people could possibly agree on the color of the sky, so what could the other possible motives be, based on what we know?
OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models are catching up.
xAI is tracking behind, and whatever regulation it may be that paces the frontier, Musk is less likely to be affected by it. Therefore xAI should be pro-regulation that stiffles his competition and gives him time to catch up.
Have they? That seems like quite a claim given the last 6 months, particularly for cybersecurity.
OpenAI just solved Navier-Stokes.
Seems like the US is on a takeoff ramp to me.
1. Navier Stokes was plagiarism
2. All benchmarks were misleading wrong and incorrect
3. All other mathematical advances were again hype
4. HF incident was marketting ploy jointly coordinated by HF, METR and OpenAI (and also Anthropic)
5. Anthropic's HF like incident was again a marketing ploy [1]
Nothing ever happens. This whole thing is a scam. Everything is done to fool you and you have fallen for it. Congrats.
[1] https://www.anthropic.com/research/investigating-incidents-c...
_of work that was aided/accelerated by AI!_
Astra can confidently one-shot 500k lines of slop, with 800k lines of tests covering it, without testing a single intended product requirement, and none of it actually working.
All models require hand holding. Fable and Astra are no exceptions. The difference is only in the amount of hand holding required, and there's essentially no gap here anymore between American and Chinese models.
I only use Chinese models sparingly because American models are so much cheaper with subscriptions, that it doesn't make economic sense to not use them. If/when that changes, I could simply route to cheapest model that's available at the moment and I wouldn't notice much difference in most applications.
That is a much easier catch up game. GLM 5.3 and DeepSeek flash 4.1 also demonstrate significant jump in cyber capabilities. So yeah, it is a slowdown in the place where it matters. RL has been around for ages, there's no moat there if you already have a good enough pretrain.
This is obviously untrue… do you use any of them?
AI in general is just hype and unprofitable and all these companies are playing marketing tricks before the IPO after which they will cash out and let the economy crash.
Downplaying the latest models capabilities is frankly insane considering what we’ve seen what OpenAI’s models have done without safeguards. That wasn’t possible before this latest generation.
Yes, I do use them, quite heavily. The only difference at this point is in benchmarks that can't be trusted (see: artificial analysis on Astra), or in the way models communicate.
Most gains are now from RL, which for some reason is hyperfocused on improving cyber capabilities, and harnesses. Raw intelligence gains of base models is absolutely slowing down.
And yet, they historically did agree on the existence of AI risk, since before OpenAI was even founded.
I'm shocked anyone could conclude this. This year it became common for people to entirely delegate coding to AI (I know many competent programmers/researchers who do this now). Progress in math has just been insane. An internal model at OAI just resolved one of the most celebrated open problems in mathematics. If anything, progress has accelerated.
This has been the case for around 2 years now, more reliably - a year. We've mostly stayed there since then.
Saying that more people started doing it isn't indicative of significant improvement. Some people just started doing it later.
I can't speak about math because I haven't used AI for that application, but I know that there hasn't been any significant advancement in coding in this year on base models. There has been more RL work, more harness work, more tools, they all expanded some capabilities like cyber or orchestration or tool use, but raw intelligence of base models is no longer where the main focus is.
I have to disagree with this pretty strongly. Opus 4.5 needed a lot of handholding not to work itself into a corner pretty quickly. Fable I basically never need to correct, and I've most become a data source.
But models have been somewhat stagnant since Opus 4.6/7.
And in some regards there were even regressions like Claudeisms that are load bearing.
2 years ago a model could barely solve the AMC, 1 year ago it got IMO gold, and this year models have solved multiple millenium problems.
Even 1 year ago ai code was just unusable (claude code only became available 1.5 years ago!) and now basically everyone I know from independent shops all the way to faang and anthropic/openai themselves exclusively use some AI agent to code.
Why does HN continue to delude itself that "models are not improving?" Maybe for the simple things they care about its "roughly the same," but they are _clearly_ improving.
Increasing a bound on the fraction of Reimann zeros on the critical line doesn't count; even Anthropic says they don't think this line of work will lead to a solution.
This seems like a case of "save me from my own mistakes/ambition"
I mean it is literally economy 101: some capitalists getting on the top using free market, and then try to use government to remove free market so their top position were secured from any competitors.
Duh! It’s called collusion. They want to try and hoard the technology for themselves if possible!
What is going on in this thread?
Are these even real comments?
While I strongly agree on the need to pace the frontier, I do think there are reasonable arguments that could be made against it.
Non of those are being made here though. It's just speculation that the CEOs of the fastest growing companies in the world are somehow obsessed with regulatory capture to maintain their lead, and have been playing the long game by talking AI risk for 10+ years so that when they become the CEOs of leading AI companies they can push for regulatory capture.
Perhaps they're all just genuinely concerned?
What would regulatory capture even give them? Who are they so worried will compete with them? Google, Mistral? Even if it's Chinese AI labs, it seems odd for anyone in the West to be opposed to any regulation that would slow them.
So assuming they are obsessed with regulatory capture – what's the argument for why they would want it?
Even though I don't think Dario is such a person, because unlike their LLMs, I do have a working memory, I'm going to play the ball and assume he is, indeed, an angel brought down by none other than God himself.
Why should CEO of a for-profit company be the dictator (pun intended) of which models I am allowed or not allowed to use, and in which way? Why should Dario have any say in what DeepSeek can or can not publish or whom they can serve?
> What would regulatory capture even give them? Who are they so worried will compete with them? Google, Mistral? Even if it's Chinese AI labs, it seems odd for anyone in the West to be opposed to any regulation that would slow them.
I am opposed to it. Why would I be opposed to open weight models? Why would anyone be opposed to open weight models or more competition other than the ones that get their bottom lines hurt by it? The reason why you still have reasonably inexpensive access to western AI models is because China has been breathing down their necks. Otherwise, you'd be paying a whole lot more money per token were it just these 3 running the show.
I agree with you that we can't allow a handful of people to hold this power themselves, so what are you arguing for here? That the answer is to just leave AI labs to carry on as they are?
The government should be ensuring that the technology is being developed with caution, but since the government seems to have no interest in that, this is at least better than nothing.
Your solution of giving everyone access to super intelligence is absurd. A good guy with super intelligence isn't going to protect you from the bad guy with super intelligence. If someone uses it to poison your water supply with a bioweapon or to take out power stations, you and your family will be dead long before your super intelligence hacks together plan to stop it.
Yes, and no. I do not think there's a need for special treatment of LLMs. But the labs certainly shouldn't be allowed to go around hacking things on the internet when they could have prevented it by simply taking time to ensure sandboxes they run test models in are given more than a quick look by a junior engineer to configure.
But the interesting thing is that there are laws that already govern this. They're just, for some reason, are not applied to Anthropic and OpenAI. Cyber crime is already punishable, and both labs did, in fact, commit cyber crimes. Did the existing regulations fail? No, the enforcement did.
The same goes for bioweapons. Any chemistry undergrad that can read an openly available paper can manufacture insane things with just common household ingredients, they do not need LLMs for that. This is a made up scenario with no evidence whatsoever to support it. There are millions of people that could poison your water supply today, without ever touching an LLM.
If we're afraid of LLMs turning into superintelligence, and we think superintelligence shouldn't be available to anyone at all, then we must all agree not on pacing the frontier, but just stopping any and all research into this. And that's not going to happen.
Yes. Basically everyone here is dismissing this possibility. We shouldn’t.
Exactly how does a CPU kill 8 odd billion people? Even if they got hold of the weapons to wipe out a few billion, surely the remain billions would just turn off the data centres?
Or... something so far outside of the box that we'd have about as much understanding of it as a baboon does a gun.
The chance of getting broad agreement on “pacing” is fairly low, meaning that all of this likely won’t happen and the race will continue. However, even in the unlikely event that the frontier does get paced, all this does is slow down the economic displacement and not by very much.
If the socially beneficial goals of AI are to make fundamental advancement in medicine and science, then restrict the use of AI to those purposes.
Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
The pitch for AI is always such that everyone’s living standards are increased, yet the actual actions we see are aimed squarely at reducing them. How about the labs put their money where their mouth is and stop trying to replace all human labor, and actually concentrate on the things they claim to care about? And how about introducing legislation to enforce that?
This probably has less chance of happening than pacing the frontier, but it’s the kind of pacing that most people would actually want to see.
Legislating against using AI to replace employees could create an unevenly distributed basic income. The idea of limiting AI to socially beneficial efforts is appealing, but I keep coming back to Bell Labs and Xerox PARC. Cool stuff was developed when you gave smart people a playground. Maybe the question should be: How do we give AI a playground and see what it comes up with? Although that could set us up with a situation like in the story Dragon's Egg.
I don’t think the proponents of these ideas care about who does the work. They see it as protection for specific jobs. Basically a jobs program with the costs forced on to large corporations.
As you said, it becomes a great gift to the few people lucky enough (or more usually, with enough nepotistic connections) to get those easy street jobs. It would not help everyone else in the economy.
It would also accelerate moving jobs overseas. Any company that regulates productivity that far downward would be crazy to continue doing automatable work in that country. It just gets moved to foreign offices where they can use AI. There are regulatory maximalists who say we’ll just regulate that, too, but then the whole company relocates to another country. Then some want to try to regulate that, and so on ad infinitum but it’s all layers of holding back your domestic companies so their international competitors can eat their lunch.
You can and should engage in international regulation, as that is the only way to solve international problems.
The idea, capitalism would somehow lead to an ideal world all by itself is demonstrably wrong.
Add in the sincere American belief that everyone else is beneath them, and you get the version where they believe even developing countries must accept kneecaping their development so American corporatism doesn't collapse.
Like we can just do stuff. We can fine companies, take away licenses. We are capable!
But I also think - I’m not trying to be too negative here because certainly we should be investing in scientists and other discoveries, but early tech - xerox park, googles 20 percent time. They were playing in an extremely immature space.
You could throw 20 ideas at a wall and create a billion dollar business.
We should do what you’re saying but we should also build institutions and maybe also limit the extent to which we’re building Elysium. If we can build AI models that rival human intelligence we can create laws to protect its dignity.
Altman or Dario will donate some money to Trump, get cozy with the Department of War and make up a bullshit excuse why Open Source needs to be eliminated.
All access to this technology will be gatekept by a bunch of nasty people who want to eliminate your job as a knowledge worker and put everything behind a subscription.
You're so close to describing a Universal Basic Income...
It's not just "companies". Even if the company does nothing, enterprising employees are going to use LLMs to magnify their productivity, which will may reduce the company's need to hire more employees. And except in extremely locked down environments, there's absolutely no way to stop an employee using some open source LLMs to multiply their productivity.
Regulate what, exactly?
Limit what AI corporations can use? Other countries would love that more than anything. Basically a free gift to any competitors or any startup that quietly avoids the rules.
There’s a theoretical version where all the countries in the world join hands and agree not to compete with each other, but that’s so impractical that I don’t find it interesting to discuss.
> Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
So small companies get the advantage, but large companies don’t? And large companies in other countries also get the advantage?
This is a highly precise way for a country to take out all of their large companies. What would actually happen is that every large company would start the process of relocating to another country right away, accelerating job losses rather than slowing them.
Sure - look at what is being done today with H1B visas and tariffs.
What is more important to you - having a job and a functioning economy/society, or upholding some "free market" doctrine you've bought into?
Just as with H1B visas, you restrict usage. For example, don't allow AI to replace any job that pays under $250K.
If foreign countries make things cheaper, just as China is making EVs that cost a fraction of a Tesla, then slap tariffs on them.
A government should be primarily concerned about the people it represents, not about the profitability of the companies who are lobbying/bribing it.
I have some bad news for you if you bought into the idea that the tariffs were good for domestic jobs and the economy.
> What is more important to you - having a job and a functioning economy/society, or upholding some "free market" doctrine you've bought into?
This is the definition of a false dichotomy, and you also ignored (or missed) the point completely.
I did not make an argument out of “free market doctrine”. I explained the second order effects that would provide a net reduction in jobs in this country.
My wife’s company is going through this right now. H1Bs are expensive and unpredictable, so the solution is to relocate more of the operations outside of the country and build up those offices.
Failure to understand second order effects and international economies is pervasive in these arguments for heavy regulations. You can do a lot to hold back your country’s companies to force them to pay for employees they don’t need and those few employees benefit for a while, but it collapses when those companies stop hiring in the regulated countries and resume hiring where they can actually operate a business. If you try to regulate them from doing it anywhere, you’ve given their competitors a wonderful gift to come destroy the company completely. Then they’re not making any jobs for anyone.
Does it make sense to you for US companies to be sending wages overseas, supporting a foreign family and economy, when there is someone here in the US, unemployed and equally capable of doing the job?
Trump has certainly been ham-fisted about tariffs, but in a world without tariffs then the country with lowest cost of production, closely related to lowest cost of living, and lowest wages, wins, and this is not going to be the US in most cases. If you follow that path then a high cost country like the US will end up utterly reliant on other countries as it imports almost everything, and loses all domestic manufacturing capability. Sounds familiar?
If the government wants companies to act in a way that benefits the country and its people, then they need to pass laws to make that happen, for example by things like H1B quotas, or by tax penalties that make the desired behavior the most profitable.
People, just like companies, are most likely to act in their own self-interest rather than in the interest of society as a whole. Sure, you'd like to be able to buy everything as cheap as possible, which in a high cost of living (which translates into high cost of manufacture) country like the US would mean relying mostly on tariff-free imports, and the death of the entire industrial base. Is that really what's best for the country?
There will still be a market for "trust". And its customers will never trust the machines. And within that market a lot of new startups will emerge for those who actually get to work and seize opportunities rather than lament a bygone era.
In the short term yes, but in the long term no.
What makes AI different to previous technological shifts is that it is (or will be) a general purpose technology, equally capable of itself doing the new jobs that it creates (so yeah, they'll be created, but also immediately taken away).
The saving grace is that this won't happen immediately, or as soon as we get something the AI companies slap an "AGI" label on in a couple of years, and it will likely take decades until we have an "artificial human" tech that genuinely could do any job (if we choose to allow that future).
Which is one of the main reasons Trump's arbitrary tariffs are ineffective policy, businesses see his capricious approach to tinkering with tariffs on a daily basis and instead make a rational decision to eat the costs under the assumption they are temporary.
Because he may change his mind in a week, SCOTUS may strike them down, Democrats win budgetary leverage in Congress, the right bribe gets paid for an exemption, etc. He could have enshrined them into legislation with the GOP's majority but politically they don't have enough support even within their own party, making the whole tariff saga pretty much pointless and self-defeating for any kind of long term restructuring of the US economy.
"Competition" only makes sense within a pre-mediated set of constrains. Rules all competitors agree to.
Workable rules are a difficult problem, that doesn't mean it wasn't worthwhile to come up with them.
The rules would never work on a global economy.
If you think the entire world is going to shake hands and agree not to let their companies use a competitive advantage, that’s not a conversation I’m interested in having. It’s a fantasy.
If you think it’s that easy to get all competitors in a global economy to agree to a set of rules, maybe start with nuclear disarmament?
Getting people to agree isn't easy, that doesn't mean it wasn't worthwhile. There often even isn't any viable alternative to reaching an agreement in social settings.
When your hyper-competitive mode of thinking leads to undesirable consequences, you have to change your mindset. Reality doesn't conform to wishful thinking.
Almost nobody (I have to say almost) in the world including Xi, the ceo of deep seek, Dario, and even Sam Altman want AI to kill everyone. I really think we just have to
A. Get stakeholders to believe this is a bad idea.
B. show that we’re willing to do it first to prevent this stupid race from continuing.
The treaty came after the crisis, not before. And it only banned atmospheric, underwater, and space tests. The US and USSR kept testing underground on a large scale, while France and China never signed it and continued atmospheric tests for years.
An AI treaty could quite easily be offered by the US because it has multiple frontier labs. Most countries would quite happily limit the activity of their larger companies in exchange for their companies not being eaten alive by AI generally. It is pretty great fodder for international cooperation, and governments can move fast when it is timely to do so.
I am highly skeptical everyone will play by the rules, even if you could get people to agree on paper.
The difference between something like "unlicensed AI training" and, e.g, Nuclear weapon non-proliferation is that nukes are realistically only a deterrence against invasion. There are no economic benefits to having them in a stockpile. They're hard to make, and the manufacturing and testing steps are pretty obvious to an observer.
This is opposed to AI training in a data center that might host remote-stream video games, hospital infrastructure, protein folding, etc. How will we be able to police that in any realistic fashion? Will China or the US allow unfettered visibility into all data centers data streams to outsiders?
And again, the incentives: Imagine having Astra v3 or Opus 8 while everyone else is stuck with a lobotomized version of GPT 4o.
What about the military applications? The weapon of the future is autonomous drone swarms. If I'm a superpower, I'm pouring hundreds of billions into that.
About the only realistic way this is going to work is if we are able to solve alignment in a way that doesn't significantly hinder AI development. If it imposes even, say, a 20% handicap, people will ignore it.
"Alignment" likely is unsolvable programmatically: Conscious intelligence is more powerful than unconscious. How "aligned" are you? Would you want to be? What difference of any relevance is there between you and "artificial" conscious intelligence that allows you to turn them into slaves?
Of course I agree with that. The problem is the enforceability / detectability of this on a global scale hovers around zero, when we're talking literally trillions of dollars on the line.
You would literally have to have every data center account for every cycle of compute it generates.
Again, the nearest comparison feels like bioweapon and nuclear bans. But the difference is the incentives for building those feel significantly weaker, so treaties that reduce and ban their creation are pretty easy to get consensus on, broadly speaking. By comparison, this is something that might provide a 10x competitive advantage to your country's economic, intellectual, and military might.
I think the only way we would get broad global consensus on this is for something Really Bad to happen, so that it moves it out of the fuzzy "sci-fi theoretical" into concrete existance.
We kinda do for other areas.
Largely the world agrees not to pirate software, tv, other IP.
THe issue of joblessness is well understood in china, they know that jobless people means a drop in living standards, a drop in living standards means the "contract" has failed.
So it makes sense that countries like america and the constituents of the EU understand that loosing 10% of all jobs in a few years is going to be a massive dick punch.
I also I think that finally the historians and economic types havae got through to the dipshit billionaire that they only have money because the plebs are spending money. If the plebs are unemployed and unemployable, they (the billionaires) are going to be lynched.
Hence the robot dogs
There is not a single country nor trading block on Earth who is going to allow some AI dominant superpower to ravage them. The idea that America or China wins an economic game here is absurd, what happens instead is that trade barriers go up hard and the world fragments into blocks that tolerate AI to differing levels. Unevenly distributed AGI kills globalism the next day.
For the rest of your reply you ignore the “at least” part in that sentence, and also seem to expect a detailed policy proposal. The details can be worked out; there is some reasonable compromise between what activities AI can and cannot be used for, and what level of capability can be deployed where.
More broadly, you seem to believe in a just world fallacy of unregulated capitalism being an inherent good. It isn’t, and regulations exist even in the United States, so this unregulated state doesn’t exist now.
Moreover, regulations and redistribution are stronger elsewhere in the developed world, and there is a strong argument to be made that the lack of these in the US, and the gap between rich and poor that it causes, is responsible for more suffering than having more of these things outside of it.
will have nukes.
Doesn't matter how much "intelligence" you have stockpiled when nukes start raining down on you.
Indeed, and the same logic applies as to why allowing companies to replace too many worker's by H1B holders is a bad idea (which equally applies to allowing companies to replace domestic jobs by offshoring).
If you don't hire people, who pay taxes, and take what's left to buy stuff and keep the economy going, then how DOES the economy keep going?
I guess Amodei sees us all on UBI, aka food stamps, so at least the farmers will be selling something, but is OpenAI going to be accepting food stamps to pay for ChatGPT subscriptions? Tesla better be selling it's robots real cheap if it is planning on selling them to people that only have UBI as an income.
Additionally, unemployment levels are perfectly fine. Clearly this mass unemployment prediction isn't happening yet.
That is because they can allow themselves to do nothing (for some time). Tech was a very high‑paying type of jobs, so workers could afford not to move into lower‑paid sectors, living on the savings they had accumulated beforehand.
In a couple of years market will correct tech-salaries and people will have spent their savings and then there will be your career shifting into healthcare and hospitality roles in large numbers.
People don't use technology to free up their time any more. They use it because technology keeps making life more complicated and tiresome and new tech is a short-term amelioration to that until that too increases the complexity of life some more.
AI is taking all of our jobs. We need to control the means, nay, the pace of production. We should build some kind of barrier, not of concrete and cement but also of the regulatory variety, to stop these agents from coming into our special white color cubicles. It's the only way.
A lot of companies trying to replace human labor with AI are either lying (theyre laying off because they over hired and need to correct) or will regret it.
That being said, what's the difference between a company that replaces 5 people with AI and a company who would have otherwise had 5 job openings, but decided to delegate to AI?
Personally, I-d rather be able to reap the economic benefits of AI by having a 4 day workweek. Instead of using AI to replace 1 FTE, use AI to offload 8h of work a day for 5 FTEs.
Of course, this is probably more outlandish than your idea.
(Or, we could just make basic income a thing and no one has to worry about their basic needs, but if they want the new iPhone Duo or a Rivian or other luxuries, they can work for it, but that idea is probably most outlandish of all)
Caveat I know nothing about economics lol--but this is my uninformed opinion.
Careful, you’re making too much sense. The big model providers would much rather the ball be in their court. “Deceleration” meaning Anthropic and OpenAI offer less capabilities for more money in service of “protecting humanity from extinction” rather than penny pinching. A lot of “AI research” coming out of these “labs” (increasingly enterprise) seems to be more driven by a cost benefit analysis from suits rather than genuine contributions to the field. I think the last true innovation was the idea of using doom and ending the human race as a marketing gimmick, which strangely seems to have worked in setting the narrative and captivating the sci-fi imaginations of journalists and techies alike.
Reading this from Dario (or SamA) makes me sick every time. Such hypocrisy.
DeepMind built AlphaFold2, an transformer/attention-based system that predicts protein folding structure, then published the method, open-sourced the code and model, and together with EMBL/EBI made hundreds of millions of predicted protein structures freely available to scientists worldwide.
!THAT! is what "raising the quality of human life" looks like to me. An enormous contribution to structural biology and biomedical research.
Meanwhile, Dario spends a great deal of effort arguing for tighter controls for open-weight models, while OpenAI increasingly seems to treat mathematical discoveries as competitive PR/product announcements.
Jeez.
P.S. Not saying Google are the good guys here. It's just an example of what actually putting a major AI breakthrough into the hands of scientists can look like.
I do not want to make a case for the military to force that order on everyone, but if we are talking human extinction, which we are judging by what we see in mass media recently, then the military will take over.
I am sure the Europeans rolling up their sleeves right now and getting to "work"
Europeans are a non-representative subset.
This is 100% about money. The only way to pace the frontier is to make everyone (read: investors) lose all their money (read: no longer expect returns). Then nobody will pay the GPU bill or pay celebrities 10 million dollar salaries to stay at the hot lab. Suddenly the development is paced, almost like magic.
Conversely, it is hard to see how pacing development makes the current bubble justified, i.e. how development could be significantly paced without investors losing their faith in hot returns at current valuations. Faith in a bubble literally equals money, exactly in the way that loans created by a bank literally equals money. You can't have one without the other.
In fact if the whole industry goes bankrupt and investors are burned bigtime (many trillions wiped), this would spread transnationally, fixing the "if we don't do it, China will" loophole.
OpenAI had it right originally, the idea to be a NON profit and vow to never participate in an arms race. It's too bad that was tossed out the window now that there's money.
When we say the world's GDP grew by X%, what is the point of that growth? Inflation? If the economy grows as a direct effect of humanity growing, I get that. If the economy becomes more efficient and we can not get more with less, I also get that.
But what is the point of just growth, if not simply to show others that I am growing faster than you? At which point this is a meaningless number. What am I missing?
How does he propose that such a thing will work? Are we going to have US AI inspectors in China and vice versa?
Why would China agree to such thing in the first place since this is a winner takes all situation.
If the US slows down or stops altogether, China can continue to work on their AI and overtake the US, if China accelerates and the US keeps its current pace then it can overtake the US.
The only way out is on the contrary for the US to actually go faster and increase its lead so that China is always 6 to 12 months behind and/or reach AGI/ASI first at which point China will most likely develop its own not long after.
This would actually be the best outcome, just like the MAD doctrine contained the spread and usage of nuclear weapons, having two superpowers with AGI would ensure that they can't be used to arm anyone. Unless they escape their sandbox but that is highly theoretical.
Regarding these Embedded Evaluators who are supposed to be neutral: - who controls them? - who has oversight on their decisions? - can their decisions be challenged by the public or a government? - who has the final say whether a model is "compliant" enough? - what does "alignment" mean in this context and who decides if a model is aligned enough?
That said, a misaligned AI could absolutely do catastrophic, civilization-crippling damage with today's Internet alone.
What does "aligned" even mean? Aligned with who? We have no universal code of ethics. Democracies kill and launch wars merely to have cheaper stuff, even when we are already rich. And why would China ever agree to stay in second place? Our society is built on the premise "the smart and powerful dominate". The idea of domination is deeply embedded in our capitalist model.
Or course it is true that powerful AI will empower the average Joe to make a bio-weapon, and that will lead to our ruin. But at the same time, having powerful AI in the hands of a just a few is almost equally as horrific.
But deeply embedded in out national and cultural ethos is to advance science, tech, and material wealth at all costs. We never ask if we have enough. We sacrifice community to advance our careers for ever more.
There is no way out of this one, I'm afraid. Our cultural predispositions and mindset of domination compel us to chase ever more powerful AI as if doing so were a mandate from god.
And the result will be a crisis in so many dimensions it is hard to reason about what will go wrong first and most spectacularly.
We've slid so backward, so quickly.
I have seen this before, first hand, on lower stakes for the world, but high stakes for the org. Self imposed safety goals do not last a quarter, but the clawback is not transparent. little things start snapping back internally, new, seemingly unlrelated initiatives are funded that take resources away from this, people are let go for different reasons, until 1y later you are right back where you started. I am hopeful though, that this becomes systemic and not something any one company can easily reverse course on.
I think those two tariffs are the best place to raise costs because they can be increased incrementally, state by state or country by country. The main problem with the regulatory approach is that it's a "big bang" post-board approach. Nothing happens until the regulation is defined and deployed, and corporations are experts at delaying implementation.
Yes, increasing tariffs means touching many individual regulatory domains at the state and municipal levels, but like deploying solar energy or wind power, you can do it incrementally.
What we need to do is to make unaligned AIs illegal and monitor for them, like we do with nuclear weapons. And make aligned AI strong enough to counter it, for deterrence and defense.
What did your CPU do without instruction? Did OpenAI tell its agents to hack the orgs it did? Come on, make a half sensible argument.
If we go down the regulation of alignment route, we'll have to ask experts to create those regulations and monitoring regimes. And which experts will the government ask? Oh, right, the parties that stand to benefit the most from regulation: OpenAI and Anthropic!
When "the experts" make certification cost $100M/model because "safety and alignment", then they will have cemented their duopoly.
Meanwhile, tariff rates on compute is a simple percentage that can be scaled up and down. Congress doesn't need experts to do that. Vendors can participate proportionally to their scale. This is far simpler and far more fair.
You've never heard of academia, NGOs, and intergovernmental organizations? Sure, the AI labs will have a say but not all of it. You don't ask the fox to guard the henhouse.
personally I would like to flip this around, all of this is a consequence of some really extreme notions about investment. its a tail-wagging-the-dog. the investment community is apparently in love with the idea of an AI-scale vehicle in and of itself. given the amount of power they have, this is funneling a massive amount of resources into something that while really very interesting, probably would never be able to realize proportionate gains without taking down the system that built it. what's the societal control on that.
I don't work in this space so I don't know the latest, but here's an example: Provably safe systems: the only path to controllable AGI (https://news.ycombinator.com/item?id=37619285)
Is that so? Last time I checked, RAM was getting a lot more expensive...
I also love how the labs are apparently screaming "Stop us! Please stop us!!!" at the top of their lungs while completely unable to escape their own incentive structure...
It seems like you're implying there's some better alternative, but you're just describing a race-to-the-bottom. It is completely reasonable for every competitor to want an external coordinator (i.e. regulator) to break the pathological competitive dynamics.
* multiple levels of inappropriate controls and unintended consequences in several complex systems
* the inability, both politically and technically, to turn it off
[1] https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf
EDIT: Comments indicated I was confusing, so I added a date to make clear that this is pre-LLM agents. My apologies, I intended to illustrate parallels and the post-mortem so we can learn from it.
Not directly relevant to the post being discussed, except as an example of how runaway automation can lead to unintended and large harmful consequences.
I considered it relevant as it involves the algorithmic/computing implosion of a 17-year-old market making company, in the young field of electronic trading agents, with heavy regulation Federally (SEC) and industry self-regulation (FINRA), which includes compliance and audits. Mandatory pre-trade rules such as 15(c)3-5 were less than 5 years old then and even more regulation came out of that incident.
The article is calling for embedding, controls, and regulation in LLMs. Understanding how the same processes utterly failed a decade ago might be useful in understanding how to proceed.
The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work.
We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI.
And there we have it: the reason I say that Anthropic is no longer a frontier lab is because after this July, we have proof that their strategy and the models they put out do not match what us, the users and the market, needs and doesn't fit the work we need models to do. As a result, Anthropic's market share is dropping rapidly, and how can you be a frontier lab when you're losing every day, for months, without end in sight?
If Anthropic and OpenAI were still seeing exponential or even linear gains from scaling they'd be doing it because the rewards to reaching AGI or SGI before everyone else are basically infinite. If both are talking about slowing down it means there's no known path to AGI so they're both going to push the safety angle as an excuse to slow down training new models and take profit.
Before December 2025 they were still intelligent code autocomplete or Stack Overflow bots, then they started one-shotting serious long horizon tasks. Now they've just solved a millennium prize problem.
In less than a year.
This is false. Coding agents have been usable since at least May of 2025. I can't speak to earlier than that as May last year was when I personally started using them.
They one-shot tasks for which there's a git clone one-liner, except worse.
My experience with them one shotting tasks is that it usually doesn't work if you try anything ambitious. You need agents iterating. And agents iterating isn't an LLM improvement. I did say tooling got better...
How often do you hack on actual LLMs? Or do you just use the chatbot or API for your agents? An LLM without internet access or tools is just as useless as a year ago.