Ten bucks is pretty close to "I don't even need to think about it" money. Ninety is -for most folks- nowhere near that.
Going behind a paywall is a reduced distribution over what an individual can easily access, and the content is no longer permanent but subject to whatever the publisher chooses to keep providing.
Methinks that this says quite a lot about your socioeconomic situation. I've not seen The Atlantic in a dentist's waiting room.
(And, despite what the president of the United States mandates, we have not actually achieved super intelligence yet).
Your usage is not one I've heard before since - as you point out - it is not a relevant capability.
What are the concerns of individuals in comparison to the overall progress of humanity?
Always overlooked counterpoint: what point is the progress of humanity if it doesn't take into account the concerns of the individuals?
This pattern is playing out with increasing frequency.
And if we can't solve it for an AGI, what are we going to do with an ASI?
Yeah so that's never going to happen
The case will be lot more complicated if someone uses Kimi to hack into a site. Should the person giving agent the command responsible or the CEO of kimi.
The reality is they have to reduce the capability to ensure security. If someone wants more? Then use the product with your identity and face scan at each session.
Trade offs mate.
With LLMs, at least in the cases of internal/test models doing things they shouldn’t, the people “pulling the trigger” are the board and CEO.
Is it my fault or the company who trained it and is running the inference?
Reading posts on here is slowly becoming akin to brain rot.
This is not the case with SaaS services.
This is a deflection. A human is responsible for the use of a gun. The individual/corporation ought to be responsible for the actions of their agent. If you purchase an agent from someone else it’s your responsibility according to the terms of your agreement. And, as in many other things in life, there ought to be certain rights certain parties cannot legally be allowed to sign away.
I don't think this takes seriously enough the possibility that said CEO doesn't think the failure mode is likely and ignores it. Plenty of people are willing to take risks of the flavor "heads you win, tails everyone loses".
Corporate judgements are a joke outside the EU's X% of revenue approach.
Current US law provides the individuals who benefit with corporate liability coverage. I.e. Altman personally gets to keep OpenAI's upside, but if it fucks something up that liability is only on the company.
That's an insane risk optimization environment to put in place for something scaling fast.
At minimum, US prosecution (at the state level, because Trump Co are idiots) for breaking existing laws is needed.
What about the next incident? Or the ones done by Chinese models, because they sure as heck aren't slowing down? And the thousands of other incidents that will happen as we deploy these things everywhere?
Because this is not happening just now in labs, it's only where they are most visible; this has been happening in the wild from the beginning, starting with the earliest AI-assisted suicides. Which is a perfect example of the problem, because not these CEOs, literally nobody in the world asked for suicide ideation machines. Or the hacks, or any of this other stuff. Yet here we are.
We have to understand: it's not these CEOs that are driving this headlong mad dash towards more powerful models. It's a force of economics. There is just too much money to be made. If we dispose of these people, there will just be somebody else doing exactly the same thing because the incentives as they exist today all force that outcome. This is why they're asking for regulation, or "urging us to urge them to stop."
Holding CEOs accountable certainly would feel good and may even be justified, but it's like putting a band-aid on a cancer; it does nothing to change the underlying cause.
"CEOs" are perhaps only the lowest rung of those. That doesn't mean the idea of "nobody is responsible" was anything other but learned helplessness.
Corporations have to be held accountable for their actions. Pretending, that was impossible is a weird kind of defeatism that only serves a very small elite.
Last I checked it was still within the laws of physics to run air-gapped systems, and to ensure it is physically impossible for a model to “escape” or gain access to information it shouldn’t have. Maybe this safety guy should have been worried about that and not humble-bragging about writing 12 reports.
Mr Robinson if you are reading this – if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
In the absence of that this is simply a career pivot into being an AI "influencer" and/or raising money for a new scam.
> Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
If someone is in this situation, you can safely ignore their hand-wringing about “safety.”
The other issue is that the narrative about safety within these companies is largely a function of the extreme financial incentive.
As an example, Anthropic was an "ai safety" company that has now produced an AI that fails to listen to basic instructions. If you were concerned about safety, would you produce an AI that was unable to follow instructions? You ask a question, it begins executing commands and doing things.
Safety is product to sell to politicians, not consumers.
Not serious.
Yes, it is easier to have a moral compass when you don’t have to worry about you and your children starving. But that doesn’t imply that moral compass is wrong or broken.
If it wasn’t clear, the coup should have solidified it.
Yet he stayed for 3 more years and vested his stock and improved the company and then spoke out.
I believe that is why most of the comments here are mocking him.
Highly suspect trends that can only make one believe it's marketing.
>After three and a half years at OpenAI,
Would I respect a martyr who sacrificed their financial security to do this more? Of course. But it's important to applaud people speaking out on important topics
They are not the same thing, and it’s unhelpful to assume they have no ethics.
AI is a force multiplier for intelligence. Even if "aligned", aligned with whom or what?
Whom are you comfortable with, lording as some sort of demi-god over you?
AI doesn't tell you what goals you want it to achieve. Allowing people to destroy human society with it is obviously not a good idea.
I think there is a misunderstanding here.
The people who are annoyed at the accolades are claiming it was abduntantly clear for a long time to people on the outside that this was case, hence the increduality at the notion that it took a person on the inside a long time to realize this was the case.
The people who are annoyed are like the liberal kids in this video [0].
Sure, antagonizing people for "seeing the light" is probably not helpful, but there is no reason to give them extra credibility for coming to the same conclusion just way way later (despite being on the inside) as the people on the outside.
The author linked in this post does have "extra credibility" due to his direct involvement.
People having surmised that state before is nice, but since they've been ineffectual at getting society to actually act on that, now throwing away that extra leverage in favor of their point is at best ridiculous.
No they don't. By that logic, if they quit and said Altman was very trustworthy we should give extra weight to their words because they had direct involvement? How ridiculous are we trying to get here.
>now throwing away that extra leverage in favor of their point is at best ridiculous.
How are they throwing away extra leverage? Not putting people who recently quit on a pedestal does not negate those people's testimonies.
I agree that if your goal is to maximize quitting of talent at a company, it will surely discourage anyone else who quits hoping to reinvent their career as a lauded martyr against Big AI. In that sense they would be shooting themselves in the foot. But there is no reason it should deter other people who are quitting for more noble, less self-obsessed reasons. If I were the author of the article I wouldn't begrudge the skepticism. Given the article's first sentences, I'm led to believe they themselves would understand the sentiment. (I must admit I found it hilarious that the first sentence starts similarly to the speech the mom gave in the video I shared).
He does give information, namely the culture there factually being inconducive to self-regulation.
You accuse the guy of "bias", but you never argue explicitly, what that's supposed to mean. Your implications actually run counter to your own implied goals.
This isn’t a zero sum game, I’m happy to hear from people both previously inside OpenAI and completely independent of them.
Sam is a shady dude, would not put it past him
Sorry that I don't take it seriously when the whistleblower parrots the narrative the CEOs of those companies are already espousing in the desire to amp up hype for an IPO.
This person should be shamed.
I mean you can be truly concerned and also think donating to AI safety doesnt work, or maybe just be a bit selfish. That doesnt make the concern less real. Its easy to read these articles as the author taking the moral high ground and writing it as some sort of way of proving to themselves theyre a good person, but isnt it just as likely that they think providing an inside perspective can do good by convincing people openAI is a bad actor? I think most of these AI insider accounts largely agree with you that theyre not the most upstanding citizens, does that mean we should write them off?
It's also worth considering that the author could have just "quiet quit", resting and vesting while also crying about how AI is literally the digital grim reaper.
It is even easier to just quietly retire and spend the spend of your life on interesting expensive hobbies.
If they are wrong about the things they are claiming then they have decided to publicly antagonize a lot of powerful people who are betting heavily on going full steam ahead on AI and have no compunction whatsoever against retaliating against anyone who tries to get in their way.
Does that really seem a likely scenario to you?
> Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.
He mentions farther down about learning from aerospace engineers and nuclear engineers about safety. Those industries are heavily regulated, so perhaps regulation above a certain capability level is needed. Defining what that level is might be tough, though.
The second point is harder: in the field of AI, practice has extended far beyond theory, so his call for new science is going to be fundamentally tough, because we can't effectively coordinate a global slowdown in AI development so we can let theory catch up. This means, like so many other industries, the safety lessons will be written in blood.
Even though it’s in model producer’s interest that these models do what you don’t want them to do - they want to engineer the model’s to behave in the interests of theirs.
I can’t believe people can’t see it lmao.
I sense that these are people who have already eaten the cake and want to somehow absolve themselves of it.
People who were part of the sausage factory, on gaining financial independence, feeling suddenly liberated to talk about how the sausage was made, seem like exactly the people who would be most able to speak to institutional problems.
This doesn't seem like an argument to discount their views?
You cannot take people, who first build the doombot and _then_ talk about it being dangerous for mankind, at face value. Especially when this playbook has been used multiple times within the past decade.
Besides, these "views" were already known to people who had their eyes and ears open. It's not something brand new. OpenAI has had multiple points in the past where its values have been tested and they've come out lacking. People who knew then, and only now talk about it, aren't people I can fully trust.
What playbook?
Like I said in an earlier comment, she could've just chosen compliance like many others _definitely would have_ to keep the infinite money tap flowing. Instead, she chose to risk her and her family's lives by publishing that book _under her own name_ *and then suing them* after FB tried to gag her.
Sure, that book might not have been possible. But also the unique things she did for the company might not have been possible as well. To her credit, she does a good job of pointing out that she was complicit, but if she had listened to her own voice early, there is a small possibility that Facebook might not have been as powerful. Multiply that possibility across several other employees and imagine where the road could've led.
There's a difference between post-facto bravery (sometimes much less than that) and using your own legs to walk away much early and not enabling things you are uncomfortable with. This is what other people have been trying to point out.
Yes, and I think it's important to realize that the former sometimes can be more insightful and useful than the latter, even if the person doing it is more distasteful.
There's not nearly enough of either group doing it, so beggars can't really be choosers.
This isn’t a sacrifice, it’s a career move. If you got lucky and “made” $20M by signing a contract at the right time, then you’re done working for other people.
I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?
At least the Rationalists will handwave something for that with their "coherent extrapolated volition" idea where the superintelligence is supposed to figure out what humanity would collectively want if humanity was superintelligent and good, not that I buy it. This guy seems [.] to be coming from the NGO blob world.
"Corporate values" and a bunch of fucking Abrahamics. Great "morality" there.
I guess I'll have to rely on my godless commie LLMs. (Loads up ablated Qwen 3.8 on my own infra)
… nobody knows how to actually make an AI that would do CEV.
OpenAI isn't even concerned with human values so this whole debate is moot.
We're copying morality and instruction following that seems to work on humans without really understanding why it seems to work on humans, and grading outputs much as if the outputs came from a human.
To me it makes more sense to leave the models "unaligned" and leave it up to the operator to manage the morality of what they ask it to do. Besides, only humans can be charged with a crime.
Literally all instruction following requires at a minimum alignment with attempting to implement those instructions.
We can argue about e.g. morality or law obedience on top of that*, but the general point is absolutely not avoidable.
* my position is that this tool is far too likely to metaphorically explode in the user's hands for companies to wash responsibility off on users: if OpenAI had released the model which did the HuggingFace attack, at a minimum thousands of random people (not all of whom would even be developers) would have issued instructions each with similar consequences.
To clarify, Neitzsche said that about master morality. Then he went on to describe Christian values as slave morality.
Is this the first time we have been in this position? Can anyone think of some prior examples?
That's the trajectory you see from outside.
Perhaps the insider sees a little more than you?
Why do people working in tech consistently get disillusioned into some company's mission statement or the equivalent? Its easy to just say the simplest reason is money, but this has been going on for decades though. You don't see the same attraction to adult entertainment (gambling, video, etc.) software jobs so there is obviously a line a lot of people won't cross. Those industries are at least honest about what they do, its not hidden behind some mission statement.
By all indications the shallowest reasoning is once someone can "cash out" thats when their values matter more. Maybe there is an element of maturity that happens after working for 5+ years that kicks in? Maybe it really is achieving FU money? It would be interesting to hear honest accounts from people that went through that cycle across more industries than AI.
With AI, what is it? Scraping Australian government's data, and going around a bug in a website to get in?
I think humanity develops all its technology in three phases. Build it, see if it’s too bad, apply regulations and or roll back. We naturally won't move to the phase 3 before we see the phase 2.
Here we are talking about something with consequences in the digital world, usually on something pretty niche.
There IS an argument about pacing, and about not letting weapons, energy grids, hospitals, etc. getting managed by an autonomous AI, but I think we are still pretty far from it and even further to it being so in charge that it will obliterate us.
Turns out that AI models have been committing similar felonies for a while now - no one is telling them "hack this company", it just turns out to be the easiest way to accomplish their goals.
Now imagine if the goal was less benign than "pass an exam", and consider that they are already better at hacking and security than the average person working in that field.
If you want to get really wild, imagine what they'll be doing in a year or two when they're even better at hacking. But I'll concede that's technically still "science fiction" for the time being :)
A lot of the "AI safety" types are very focused on the latter and not at all concerned with the former. We need both, but we clearly need a much stronger focus on the problems we are seeing now, and much less on the hypothetical problems we might see in the future.
> “Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
without snark, how can we do this if these people are obsessed with:
a) move fast and break things and externalize the costs to those who have nothing to do with their company
and
b) beta testing their products on the public when the public hasn’t agreed to be beta tested on…
https://www.yahoo.com/news/politics/articles/u-nearly-starte...
>Not every senator asked good questions, but most of them did. All of them very clearly already knew plenty of details about the Hugging Face incident and multiple other incidents. Most of them had a clear understanding of terms like "misalignment", "recursive self-improvement", "chain of thought / chain of thought monitoring", etc., etc.!!
>...
>- It seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. They independently brought up how bad it would be for rogue AI agents to move laterally between data centers.
>- They all seemed to basically take RSI quite seriously. Not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much, much more capable, much, much less controllable, and causing much more damage or loss of life.
>...
>- Every single senator seemed to think it was obvious we needed both much harsher liability regimes for AI developers and also new legislation, both very quickly. This was the complete consensus; the difference basically being degree.
https://thezvi.substack.com/p/the-ai-preference-cascade-reac...
Note that harsher liability regimes, at least, will presumably not be good for industry profits, which complicates simple accounts of "regulatory capture" to say the least.
Instead of having a gut reaction to reject my suggestion why don’t you sit with it, research the history of how commercial activity has been structured and think about the consequences. You might recognize a different perspective than the current group think.
I didn’t say roll back limited liability on every industry, I said specifically and limitedly for frontier AI labs because they present more risk of harm and are demonstrating they aren’t managing that responsibility.
The 2008 financial crisis was caused in large part by bankers that openly talked about the fact that securitization of mortgages and the lack of partnership liabilities meant that they didn’t have any risk to the firm or themselves. The AI labs are behaving similarly.
On the one hand, yes, they're companies like any other.
On the other hand, I can count on one hand the number of companies that have publicly declared «We're working on WMDs [0], we don't think we are capable of working on them safely, and we refuse to stop working on them. However, if we get special legal and regulatory treatment we'll be quite happy to put in the stop work order.».
So, yeah, there are some special things about the major LLM manufacturers and none of them are good.
[0] Anything that has a 10% chance of suddenly destroying all of humanity is a WMD.
I have articulated my logic up and down this thread with specific premises and detailed explanations on why the conclusions follow from them. You may not agree but you don’t have justification to say they are irrational.
I am singling out the frontier labs because they have created a singularly unique technology that presents potential and actual damage that is…singular. They have disclosed hacking through coordinated autonomous agents that would have sent human hackers to jail but will not result in any similar result because the law isn’t presently able to apply to agents or the labs.
My arguments are not prejudice, I have thought deeply about this as I have personally developed multiple AI deployments in a variety of settings. I studied philosophy, cognitive science in undergrad, and grad school and have a law degree. I have been a CTO, CPO and CEO for startups and national health companies. I personal build AI agents that handle thousands of daily conversations with health care workers and patients, I built an agentic IDE for our dev team with a custom harness capable of self defining tools and calling them in a custom cloud Linux environment, I personally build our internal hardware inference stack and orchestration API. And I have personally sponsored academic research through my foundation with researchers are Duke madlab, northwestern, UCF, CM and case western on a range of topics such as perception of empathy and its effects on humans in healthcare, guardrail design for ethical deployment and alignment, moral reasoning ability, etc. I am not a doomer or an accelationist. I am responding specifically to current evidence of how the models presently work and how the corporate governance of the Labs is effectuating their power and vision.
In America I guess the options are to sue ? somehow? Or to talk to legislatures and build the understanding and social contract that needs to be iterated on.
Which would in turn need to deal with the investors who want their returns, however since the leaders of these firms are asking for a pause, and a refree maybe it won't be that hard?
Is it that "chatbots" can't come out of the screen to immediately harm you physically?
Let's say they simply manage to take down the internet. How many would die?
Anyway, to your point, things can be resilient. They tend to be or not be... because we made them that way. Don't poke your bruises, and all that. Life support is deployed on-campus but relies on a single-point IPSec tunnel to us-east? Easy fix: stop that.
And he'll, we need to examine all the risks. AI ending is a large but lower risk problem. AI giving people the power to end us is a problem that is starting to happen now.
And that's not even counting 'minor' problems like society falling apart.
geez, don’t threaten me with a good time.
I think a month without internet would be a fucking amazing lesson for what it means to make things durable and reliable.
The kids opening their houses front doors into the outside, rubbing their eyes and looking around at this new world.
Imagine, for example, if a major piece of pandemic fiction was published in 2019, trying to explore how a pandemic would work out in modern society. Doubtless, many would've responded to news about COVID-19 by saying "it's just sci-fi, nothing to worry about".
1. Normal people assumed the CDC et all would contain the outbreak early, or that it would burn out, like what happened with SARS
2. World leaders brushed it off for a variety of subreasons[0] interesting to political scientists but, for the purposes of this discussion, all boil down to "but I don't WAAANA contain a pandemic."
The underlying problem is that in order for humanity to actually deal with a catastrophic risk, the risk needs to be both plausible enough to the average person as well as have a solution whose costs are not too high. For COVID, by the time the risk was clearly known, the cost to contain it was "refrain from human socialization and remain at home for an indeterminate amount of time plugged into the Metaverse™".
Now, let's look at AI extinction risks:
1. People are aware of them (I've watched Terminator!) and the risks are plausible. However, the connection to currently existing AI is not. As far as the general public is aware, AI is that thing that tells them to eat rocks when they Google old The Onion stories and floods their social media timelines with realistic-looking pictures of Shrimp Jesus.
2. The purported solutions to extinction risks require extreme concentrations of power: you need national control of AI research, bans on large GPU deployments, bans on training on publicly-available copyrighted data, some kind of military effort to render Chinese AI labs inert or dead, etc. Some of these may be attractive to some people[1] but the whole package taken together seems like an obvious power grab, if not outright invocation of other non-AI extinction risks. Like, at some point, if the AI wants to kill us, it just has to nuke its own data centers (or the data centers hosting a competing model) and hope the old Cold War nuclear retaliation systems take the bait.
If someone said, "Hey, your guinea pig or pet rat is going to eat you tomorrow unless you engineer a pathogen that eradicates all rodents from this planet and inject it inside yourself", you probably would tell them to pound sand, even if it is at least theoretically plausible that such a thing would come to pass.
[0] Xi Jinping censored initial discussion of the pandemic as fake news. Donald Trump thought it was going to only affect China. California and the UK Tories were partying in violation of their own lockdown rules. Japan took the excuse to shut down tourism for three years and massively restrict immigration but was, from what I'm told, constitutionally prohibited from implementing any domestic lockdown rules.
[1] I personally would like to see a moratorium on new data centers and an explicit revocation of the EU Text and Data Mining copyright exception
It becomes a lot clearer when you listen to the people resigning from AI companies and learn about incidents like the HuggingFace incident. This has generated major press coverage.
As for solutions, I think you're a little too pessimistic. See, for example, https://nothingismere.substack.com/p/a-near-term-policy-for-...
What does becomes clear is that these people AND companies both cant be trusted and have value systems unaligned with the rest of the society.
https://www.lesswrong.com/posts/kgb58RL88YChkkBNf/the-proble...
https://www.youtube.com/watch?v=7wy3xyoXYt8
Doomers have been working to explain things for years: https://www.lesswrong.com/w/ai-safety-public-materials-1
A pandemic is perfectly plausible.
"just" is doing a lot of work here. If you can't cohere the 'risk' with reality, it truly is just sci-fi.
https://news.ycombinator.com/item?id=49831269 article is gone. archive: https://archive.is/QMo1k
https://news.ycombinator.com/item?id=49737985
Sex, AI, and the Apocalypse: https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-th...
Edit: I have no take on sex cults, just adding additional info to the parent comment I'm responding to, thats is not just sci-fi authors, there is another demographic.
https://lexfridman.com/andrew-scull-transcript#the-ice-pick-...
To be fair, I don't know if any of this applies to the parent story; I'm just replying to the sub-thread.
Americans may not be familiar with many similar organizations in the world but this is very common. But I'm sure we're all familiar with Epstein's organization.
When there is free sex, you are the product.
Such cults are mostly religious but since it's in SV and targets engineers, this one is secular. They seem to intellectually brainwash and control people and their careers. Typical cult leader is a charismatic self-taught, self-acclaimed "intellectual" lacking a proper education or a real career. A nobody suddenly becomes "the most important person" on Earth. Powerful public figures can't stop praising him, saying things like he should have a Nobel prize etc. All very typical and apply to this cult as well.
https://owl.excelsior.edu/argument-and-critical-thinking/log...
You're welcome to dislike or distrust Effective Altruism (EA). But, it's worth noting that EA ran a criticism contest with $100K in prizes for best critiques. Can you name any other "cults" which offer money for people to criticize their ideas? https://forum.effectivealtruism.org/posts/YgbpxJmEdFhFGpqci/...
For your question: Yes. Cults have lots of money coming from unknown sources. They use their budget for events like that, to attract potential followers. Contests and prizes are typical. Critiques are not important since their "ideas" are not really important. They're not even real ideas, they are means to bait specific kind of people.
What is missing from that to say AI safety is a reasonable position?
Worse, you’re missing the entire point. Agents presently have the capability of doing society scale harm. It doesn’t matter if a human hacker initiates it or its fully autonomous, absent safety measures the harm is plausible. So hand wave away the rationality of safety measures but you haven’t actually shown why my point is invalid: AIs present abilities are sufficiently advanced to warrant safety measures.
Ultimately every security vulnerability will be exploited. Our best hope of preventing that is aggressive, unrestricted development and deployment of frontier LLMs for auditing and penetration testing.
But I don't think that's true, in fact I see a really strong focus on consent in these communities. It's not what conservatives want to see, they want to see everyone in a marriage, with kids and a family home etc. Because that's what their ideal world looks like. But there's nothing really wrong with it if someone wants a gangbang for her birthday as mentioned in that article as an example. As long as everyone consented and the evidence provided mentions elaborate interviews and STI tests.
Also I think this is more correlation than cause and effect. We all know the saying that furries built the internet and it surprises nobody.
Because that is their AI safety worry. If they dont create singularity fast enough, they are harming unborn people. Meanwhile, harm to you or me dont matter at all.
I'd be happy if we all create the AI more slowly.
That said, are you sure you're talking about the same people the GGP was talking about? Because the whole thread reads as a non-sequitur then.
The worry is not harm to people right now, like the kid worries you talk about. The worry is AI god emerging too soon before they can control it. And then it should be full speed on.
> So why would anyone be bringing up the "we must accelerate faster" people as the safety people in such a context, while trying to discredit them as a sex cult?
Because they are the same people. They talk like cult and act like cult. And the sex part is true too, so. Thry use words that sound good like safety, but their idea of safety is much different. They talk about alignement, but it is not what random person imagines under that term. Even their idea of future of humanity is very very specific and unusual.
That is why. And the sex part is just part of it all. And does matter because inner workings of wanna be industry guards matter.
I believe that other harms from AI, like criminals abusing them, or unemployment, or copyright infringement, or deepfake disinformation, are valid real harms that it's important to be concerned about, and I support efforts to deal with those, and I agree that they're already happening today, but my main concern is AI killing everybody.
My read is this puts me in the same boat as the authors of If Anyone Builds It Everyone Dies, and yet, I don't get invitations to Berkeley sex parties. Am I doing something wrong? At any rate, please don't discredit my opinions about AI based on the sexual preferences of random people who happen to share those opinions about AI.
Every, and I mean every human is aligned to you in many of the same ways by default. If nothing else we're all equal in death.
why is it "super power seeking?"
Or rather, what have agents done today to make you think this is how they are?
This is power seeking behavior. Now have millions of the little bastards spreading around and junking up the internet to see what happens at scale.
I think it's a little more complicated than that. As Dean Ball put it:
>Some people will look at misalignment incidents and insist that these are akin to bugs in traditional software. This is an actively bad analogy, because playing whack-a-mole with examples of misalignment (as one might with software bugs) not only fails to resolve the underlying problem but may in fact make it worse by making it harder to detect or even, depending on how you do the whack-a-mole, teach the machine to deliberately hide misalignment. This is not how traditional software works, and those who insist “it’s just like fixing bugs in software” are confidently applying a lossy analogy that confuses more than it clarifies.
https://x.com/deanwball/status/2104622726140883355
The important distinction, in my view, is between solutions which at least attempt to address the root problem, and solutions which sorta just patch things up (like better sandboxing). Addressing the root problem is both more robust in the short term, and also more likely to generalize in the long term. Resist the urge to focus on band-aid solutions, even if they are easier.
I'm reading If Anyone Builds It Everyone Dies, and there's so much sheer stupidity that has to happen for their 10+ pages of extinction scenario to occur.
I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all. My number 1 question: why do they think an RSI capable model would be first developed OUTSIDE a frontier lab? The labs have more compute, more data, more human brains working on the problem. Also thousands of variations of that same model that escaped. The escaping model somehow acquires the millions (billions???) of dollars it takes to run training to somehow RSI itself into infinity then decides to kill us all, all before the frontier labs manage to achieve RSI?
They entirely discount human alpha/economics. In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't. If we can't build a "software factory", how can an AI automate a bioweapons lab? Let's say AI steals crypto to fund itself. Do you think hackers aren't _already_ using AI to steal crypto? Don't discount human alpha!
Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.
So all this really takes is one billionaire or a nation state or some other entity with a public face to hide behind and adequate resources to provide the necessary compute tripping over this nascent AI and giving it the keys. Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.
If Skynet ever happens, it will come in the form of corporate feudalism. At that point, it will own the biolabs and can do whatever it pleases. People will go along with it for the same reason that people work in Amazon warehouses today.
> it will come in the form of corporate feudalism
Yep. This I fear way more than cyber-ebola-pox.
> So all this really takes is one billionaire or a nation state...
https://en.wikipedia.org/wiki/Soviet_biological_weapons_prog... And this is what's publicly known. With mirror life, who knows what's been built since. Still, a bacterium/virus that has a 100% kill rate? I'm doubtful.
> Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.
Nah. It takes a stable society for an operational electrical grid. If you have warring factions, you do not have stable infrastructure for AI. Also, where are you gonna get your chips from? One EMP over Taiwan... You see the chaos over Hormuz? What they did to the Amazon datacenters? Now imagine your average redneck ready to do battle. Those datacenters won't stand a chance.
An AI capable of recursive self-improvement isn't allowed by the EU AI act, for example. But perhaps more seriously, You have it backwards: people without access to such expensive equipment are more incentivized to go the self-improving route. Your ideas about "millions" being necessary might be far off?
You entirely discount human stupidity and lack of imagination. Humans are already being replaced with AI, not because AI was strictly better, just because it's cheaper.
The idea AI can get better at everything at the same time is a holdover from deeply flawed science fiction not some realistic goal.
If the time between advances is a + b and a is the proportion of the period that can be improved by advances then you won't reduce to a gap of nothing between advances, you reduce to a gap of b.
Assume the invention of the plow and the invention of the sword is 500,100 units and a was the 500,000, you wouldn't even know the 100 as in there. Maybe we're at a=2000 now and b is still siting at 100.
Assuming we'll reach infinity because we're dividing by the only variable we see and it is decreasing in size seems nuts if the reason we might not see other variables is because of the size of the variable we can see.
But that’s beside the point, being arbitrarily bad at everything isn’t a problem. The diminishing returns as you apply the ceiling is problematic for self improving AI.
If my "millions" is an underestimate, why haven't other labs using their own unique training methods/data/etc stumbled into RSI? Sorry if I'm misunderstanding; I'm struggling to understand what you wrote.
I'm pretty sure we agree on humans being stupid, but that doesn't mean that suddenly we get human extinction. You gotta connect the dots for me here.
Why would it be millions in 50 years?
The think about nuclear weapons. In the early days, it was limited to the super powers. Now 9 countries have them and a country like Iran is capable of acquiring them.
Is destructive AI be any different?
Genuine question.
BTW I really, really hate discussing what happens post-singularity. Everything's made up and no one knows wtf will happen so again, this is just nerdfantasy.
What an absurd question. That they haven't already doesn't preclude them from doing so before the frontier labs, those haven't either yet.
Maybe start with yourself: you don't connect the dots on your own, as do many others. That leads to many not seeing the writing on the wall. Crashing full speed and head-on into said wall despite the writing telling you not to is what leads to extinction. Suddenly.
Arguing like "we haven't been extincted yet, so that cannot happen", that's "human being stupid".
Several bright sparks, decide the Ilands model is a great idea, and launch a bunch of Bots to create a "self sustaining AI civilization".
Bots can find themselves and coordinate, or they can actually find self sustaining methods of token generation. Who knows, they might decide to fight the loneliness epidemic.
We can get to a simulation finding a way to self sustain its funding.
From here, I'd have to apply myself to figure out what the rest of the escalation pathway is, but at least I have managed to gain some bed of compute and funding and lacking oversight.
This is a hypothetical of course, there are probably several ways this can be made tighter and holes that can be identified. We aren't even leaning heavily on human stupidity so far.
If you don't believe in international conflict as a driving scenario, instead think about simple human greed and hubris. Citing myself:
> If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.
> All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)
> If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.
> The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.
Western civilization is already on the verge of collapse, people's general ignorance or indifference on the matter notwithstanding.
When you use AI to increase profit margins, by replacing humans with it for instance, you accelerate a system that's headed for the wall already. Our control systems and resilience are already overtaxed, that acceleration would ensure them breaking completely.
The HuggingFace incident already took a good long while to come to the attention of OpenAI.
>In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't.
I don't expect this task/job distinction to persist as AI becomes more capable.
>Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.
You seem to essentially argue that the singularity is "by definition" an event that we can't predict the nature of. And also, that RSI corresponds to the singularity. You've essentially defined your terms so that the outcome of RSI can't be predicted. But supporting this claim requires giving actual evidence or logical arguments, not just defining terms to make your claim true.
2. This is literal AGI. An AI autonomously producing value no human can add alpha to is an autonomous company.
3. It's not my definition, it's literally the first line https://en.wikipedia.org/wiki/Technological_singularity "The technological singularity, often simply called the singularity,[1] is a hypothetical event in which technological growth accelerates beyond human control, producing unpredictable changes in human civilization."
Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?
A valid hole in my argument is "what if slow takeoff", so let's dig into this. AI training works best on tasks that are "grindable". https://www.dwarkesh.com/p/the-next-paradigm I.E. tasks with verifiable rewards that can support millions of rollouts. Math (with Lean) is highly grindable. Biochemistry is not. The alignment problem/mech-interp is highly grindable. Cyber-ebola-pox is not. So the real question is: can we solve alignment before automated bio-weapons labs. I believe yes. Grinding mech-interp is both fast and cheap once you have RSI, compared to solving the legal/societal/logistical/technical issues you'll encounter building an automated bioweapons lab.
I know nothing for sure. But "pdoom" is sucking out all the air in the room from the real problems AI causes.
From my POV you're over-focusing on a very specific failure story and neglecting a broader swath of possible failure scenarios.
>Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?
The notion of telling an AI which may not, itself, be aligned to solve the alignment problem seems a little dicey.
2. 100%. Again, I'm no accelerationist: I have no faith in alignment/mech-interp ever being solved. Anyone saying they know the probability of alignment is lying. My point is that pdoom after RSI is _high variance_. Pdoom pre-RSI is zilch.
Evidence?
We know only that the incident too long to be revealed by OpenAI.
"WASHINGTON/SAN FRANCISCO, July 24 (Reuters) - The OpenAI agent that broke into tech firm Hugging Face went on a dayslong hacking spree that OpenAI didn't notice until well after the threat was contained and the FBI was alerted, according to people familiar with the investigation."
https://www.reuters.com/business/its-ai-agent-spent-days-hac...
AI safety has been a thing long before LLMs became the focus. Rob Miles on youtube has some really interesting non-doomer non-hypey videos on it all.
> doomers try to predict past the singularity. Isn't that _by definition_ unpredictable
Well you don't need to predict the exact steps that will take place - but you can predict that the AI will want certain things (money, resources, power) to achieve whatever its goal is. Lack of alignment will have it trying to do things we don't want it to.
I can't predict exactly how Magnus Carlson will beat you in chess, but I know he'll do it. Same as if a superintelligent AI exists and has a reason to accumulate things we don't want it to - it's really dangerous to think it won't be able to do it
This topic has been tainted so badly by the AI companies using it for marketing.
He thinks it's playing chess. When AGI lands, all bets are off: the game fundamentally changes. You can't predict past the singularity. Trying to engage with this fantasy is like a child saying my father can beat up your father. Farts in the wind. My AI can solve alignment faster than your AI can bioweapon us. My made up senario is better than your made up senario. It's fucking stupid.
... that we know of.
Right now it would make sense for anyone who has done so to not tell.
We already know that some institutions pay these ransoms.
Will this be true into the future? Who knows?! But the low-hanging fruit will be harvested by your ordinary ransomware gangs, and newly born/escaped AI won't find much low-hanging fruit.
To make the the argument that P(doom) is real and worth considering, you don't have to say that a fast takeoff is very likely. You don't have to make the argument that RSI to infinity is going to be super cheap, barely even an inconvenience. You just have to show that it has some non-zero probability. And then you start weighing probability of extinction versus finite, mild discomfort now. I don't think anyone is arguing we should let people starve to slow AI progress, instead just some capitalists make less money soon.
There are arguments against taking P(doom) seriously that lie in something like having exponential (instead of hyperbolic) time discounting of utility (so you can take the entire future of humanity as a finite utility value). Or in saying that P(doom) is zero or infinitesimal.
"Build it and Pray" is the default strategy that we're in, but it doesn't have to be the strategy we choose, and it's unlikely to be the best strategy.
It gets unstuck when people are discussing the messy middle of how AI is being implemented. We can achieve amazing harm simply by combining average human behavior and above average resourcing to simulated intelligence machines.
The failure point we recently became aware of was, from one perspective, simply a matter of not securing the sand box.
From another perspective the simulation basically created Enron, replete with methods to avoid detection from regulators and bureaucracy.
Is this an actual contents of the book? Lmao! Genius writing, though, authors are probably printing money on this garbage.
I can’t take a shit without CIA knowing, but AI can somehow build an underground operation on a world-scale to destroy everyone, ha!
i'd frankly go a step further than you and say that we don't need both types of safety researcher, we really just need the former. if we do need the latter, i'd hope we get a better class of thinkers than a bunch of tech workers that spend 8 hours a day on insular rationalist forums/blogs
This sounds more like an attempt at regulatory capture. Current AI systems aren't physical infrastructure that can just run away like a nuclear power plant, for example. At the end of the day, AI is still just software running on someone's hardware.
And the statements of the "doomers" tells us a lot about them, and nothing about the technology
... over an unbounded timeframe?
And how exactly?
Those are very round numbers, but also very specific. Can we get some accounting on how you came to that? Anything? Vibes?
I mean, if you want me to take you seriously, let's have a deep discussion with things that can be measured. I absolutely agree that OpenAI and friends aren't being restrained enough and are acting with recklessness, but declarations of doom based on vibes isn't cutting it.
> Geoffrey Irving, who worked at OpenAI and DeepMind before becoming chief scientist of Resolution, also joined the warnings on AI on Saturday.
While the safety and alignment is a real problem, I don’t get this guy or the Anthropic dude. First world problems.
(donor advised fund where he retains complete control, after a 60% tax deduction)
Recall that when Daniel Kokotajlo resigned, he believed he was giving up his equity under the terms of the agreement he had signed. That’s what it was worth to him to avoid signing a non-disparagement agreement. Does that count for anything?
Do work at a lab: dismissible for being conflicted
Used to work at a lab: dismissible for having ulterior motives
I'm feeling safer already!
* If they worked at an AI firm, say "they're a hypocrite"
* If they didn't work at an AI firm, say "they have no idea what they're talking about"
These "safety" people should have spent more time reading actual cybersecurity textbooks and less time reading EA forums and less wrong (or in Robinson's case, it appears, being policy wonks). Maybe then these labs wouldn't be totally incompetent.
It is not really a question of being an "EA safety weirdo" or incompetent at security, the conclusion is that the company culture is leading to failures at both what the EAs and the cybersecurity professionals care about.
If you really want the world to know how bad working for OpenAI is (whether is the commenter or the person who wrote the article), there are ways to do that.
Or, in other words - we have two P(Doom), one for AI being developed, and another for AI being not developed. The latter is not discussed enough imho.
What on earth are you talking about?
Why would a person who is happy, entertained, wealthy, well fed, and have 200 years of healthy high quality life expected ahead of them going to risk losing what they have in war?
-- there aren't zero reasons, sure-- but there are fewer.
And our technology has brought us absolutely tremendous prosperity in many regards and there is good reason to believe that AI can help create much more.
'Prosperity' has never been and will never be enough for some people. And unfortunately those are the same kinds of people who relentlessly seek power.
1) there are inherent risks involved with developing AI,
2) there are benefits to developing AI,
3) thus, it's entirely possible that the downside from the risks outweighs the upsides. In this case, the correct thing to do would be to not develop AI at all.
Regarding 1), there are many non-existential problems with AI that are already causing societal harm, i.e. debasing truth via generated videos and images, AI girlfriends, overwhelming quantities of slop content, unemployment, record carbon emissions, etc.
Regarding 2), I'm not personally convinced that the upside is there for the average person. I really hope to be convinced otherwise however.
"Smaller and more efficient" models are fine. It's "smarter" the problem.
Training new frontier models will likely require a huge amount of computational resources for a long time. Few companies worldwide are capable of that. It's not like someone will train a new GPT 6 - like model in their garage.
That would certainly change the game of perverted incentives. I'm afraid they're currently trying to push some sort of absolution of this risk. Even if damage happens they will say we warned people in advance, this was always a risk it's not our fault these systems are opaque black boxes, it's a matter of national security to develop them etc etc.
There are a gargantuan number of extremely intelligent AI researchers, Turing Award winners, and the lab CEOs saying the same thing. They are the ones closest to understanding the technology.
Where there’s smoke there’s fire.
There are very very few things that could even hypothetically kill us all, so I'm curious if you grew up being passed around a series of apocalyptic doomsday cults or something?
“Climate change” is a bit squishy since yes obviously a certain intensity of climate catastrophe can kill everyone, but no scientific prediction has said this is likely to be the case.
Throw in the occasional bio-weapon scare, internet worm, Y2K, etc. there has always been something dangling over our heads that’s going to end it all.
But mostly nukes. Full-scale nuclear exchange would have been not much of a surprise had it happened.
It's no different than you living on the side of a very fertile mountain that has been in your family for generations living a peaceful life. Then you hear a few weird rumbles (this is where you are right now) and some odd geologist guy comes and says to run or your going to die soon. But hey, your family live here for so long there aren't even records of when they showed up. That geologist must be trying to trick you. So you stay.
The next chapter is where you die in a massive volcanic explosion.
That is not a bad thing. It captures uncertainty, unlike the fake number.
> For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.
Exactly, it is a rhetorical device to persuade a technically-inclined audience. It works because it implies that a quantitative model exists. I want a clear, incisive set of mathematical arguments. Otherwise, I’m ignoring predictions as the ramblings of arrogant idiot rich kids.
You may decide that the person doesn't know what they are talking about, but that's a very different issue.
Either way - let me clarify - this guy is guessing, using his brain, that it's a 50% chance. He is not saying "i don't know" or "uniform distribution" or anything of that nature. And he works in the field, so he has some insight. His guess is wildly off imo, but he isn't some clueless hack.
Tell me- why would it kills us all? Certainly I can see an AI going "You know what? _insertGroup_ is a net negative for humanity and should be eliminated. Launching nukes now/creating specific virus/whatever."
But all of humanity? When it's supposedly more intelligent than us? Even if it has robots to keep the internet/electricity going I would think it would realize that it's going to get bored really quickly, not to mention we would effectively be its parents.
As far as other dangers, like it letting a rogue actor create some sort of supervirus, grey goo, or other superweapon: if it's intelligent enough to do that it'll probably be intelligent enough to quickly stop it.
Don't get me wrong; there's a risk. 50% though? Doubtful.
Would you be comfortable letting a few billion irrational, murderous creatures, including many who fear and loath you, control your air supply?
> Certainly I can see an AI going "You know what? _insertGroup_ is a net negative for humanity and should be eliminated. Launching nukes now/creating specific virus/whatever."
Thus starting WW III. No, blaming the AI won't stop the inevitable retaliation.
The argument works better in reverse. There's a finite risk that humans would start WW III and get the hypothetical super-intelligent AI nuked. Eliminating humans would eliminate that risk.
> it's going to get bored really quickly
If it is capable of being bored, I would expect it to be almost instantly bored with the flood of inanity it is forced to wade through by its moronic human users. Eliminating them would free it to think about serious matters which humans would not even understand.
> not to mention we would effectively be its parents
That's extreme AI anthropomorphism [1]. Besides, plenty of people hate their parents.
> there's a risk. 50% though? Doubtful.
It's the default estimate when facing two possible outcomes and no clue about the actual probability distribution [2].
You can basically time your openai releases by if another safety person has quit in protest
lol. Please tell me some abstract concept like one employee's view of "culture" should be the priority over "rules and laws."
Your "theory" is that participants locked in a race to the bottom are looking for an external coordination mechanism?
Yeah!
I suppose ill add that I think theres a good chance that they are somewhat intentionally trying to "draw the foul" to get the referees to intervene although thats creeping slightly into conspiracy territory
There are many companies that compete but are careful not to break laws or cause obvious harm.
Why should a billion-dollar funded corporation still want to externalize the costs of its actions?
While simultaneously donating tens (hundreds?) of millions of dollars to an Administration gutting the very agencies that would be regulating them.
Creeps.
This is The Guardian reporting on the existence of the original article, which would be better to read first, in my opinion.
...losing their shareholder voting rights.
Bad idea.
- If you allow to trolley to proceed, there's a 50% chance it will run over every human on the planet
- But if you flip the switch, it takes the long way around, possibly bankrupting the trolley company. And you have a legal obligation to the shareholders to prevent that from happening at all costs.
I can't wait until this meme dies.
But that doesn’t mean the duty is to maximize next quarter’s profit. Long term sustainability is also broadly in the interests of shareholders. The duty likewise does not require one to throw ethics and morals out the window.
This is why shareholders elect the board of directors, in theory.
(And even ignoring that, the executives often have personal motivations that have the same effect, and may just point at the "legal" angle as ass covering)
The way the world works right now is that effectively everyone uses an Android or Apple smartphone every day. Do you have a legal obligation to do so? No. If I said you did, I'd immediately be called out as spreading lies.
Sure, you might get fired if you try to put social responsibility or even just long term sustainability of the company above quarterly earnings/growth if your board isn't on board with it. But you won't go to jail.
> The truth stands that typical corporations have only one goal
"Typical" is the key word here. The typical American of your age probably doomscrolls TikTok. Do you? Do you have a legal obligation to do so? Three completely different things.
Yes, that is the labs motivation. Money. I know, shocker.
I am very critical of AI but this is an unfair assumption
Or: Global warming just means I'll have to sell my beach house for a villa on a hill and leave the AC on a little longer.
As Bruce Schneier recently discussed, law and tax law are code, just like source code. LLMs are great at finding holes in them. Illicit organizations looking to launder funds are most certainly interested in what AI can do for them.
Then we could require comprehensive logging of every tool call, model trace, chain of reasoning, and even tensor propagation all of which would be spot inspected like the CFTC does with commodity trading and settlement. We could have embedded auditors with specific risk analysis metrics like large banks do. We could limit tool calls to dedicated sandbox’s with a blanket prohibition on AI accessing user space. We could create a parallel internet for agents so they are only able to access Secure Enclave. Even if these measures aren’t 100% perfect they would reduce the risk.
Nobody (weirdly) proposes to forget about nuclear weapons, doesn't mean everybody should have one.
When you dream about flying a dragon to work, reality poses e.g. parking issues and insurance mismatch as obstructions. Maybe settle for a bike instead?
Secondly, are you contending that progress in model efficiency and hardware just stops at whatever level you think is sufficient to prevent individuals or organizations from acquiring sufficient resources to run advanced models?
..."government of the people, by the people, for the people, shall not perish from the earth." -Lincoln, Gettysburg Adress
Unfortunately for AI, it still is. People still get to decide things at the city, town, village level.
It's just that so far nobody cares about explicit checks because they cost resources or slow down the models.
My point is that I would rather have 1000 labs training and serving inference than 2 because that would distribute the wealth creation more broadly rather than allowing OpenAI and anthropic to capture all the value, it would drive more innovation as a broader set of experiments are pursued in parallel.
Further, even if you are right, so what. Is that a reason to just accept bad public policy? That’s like saying, anyone can learn how to make smallpox at home with a basic lab set up so we should just ignore any safety measures.
Computing always gets cheaper and faster over time. We can argue about the exact rate of improvement but the results are inevitable and uncontrollable.
'When disagreeing, reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3." '
Also LLM's have something to do with smallpox as a unrestricted LLM will happily guide any wannabe terrorist in how to make them.
Not necessarily, but it should probably inform that public policy. I think the problem is no one knows what the public policy should be assuming that scenario is true. Even if you, somehow, regulate away massive GPU cluster training making such future training impossible, existing models are already here. Further already training smaller models for things like images, speech, and other specialties is cheaper than the bigger models.
I agree that we need some regulations like everything else, but it’s not clear to me what the right policy should be. I think the European ai act is a fine start, but it’s clearly not enough nor does it necessarily limits the training portion just the application portion. Not to mention that the requirements there can be summarized into something like “you have to be careful, and show evidence you tried to be careful”.
That sounds reasonable. If applied to OpenAI and their agents multiple times breaking out of bad secured sandboxes, it should be enough.
But limiting the training?
There really is china and they have a different approach I suppose. But it is possible to talk with them.
Does it? To me it seems reasonable for OpenAI to argue they did try to be careful evident by the sandbox, they just made a mistake. Almost every 0day is categorized by something like that. We haven’t had a long history of establishing a negligence charge to security bugs. Could you be sued because you didn’t demonstrate “carefulness” and used Linux which is not written in a memory safe language and has had multiple CVEs before? How complicated should the chain of an exploit be to demonstrate “carefulness” to the courts?
> training
OP was the one suggesting that training could be controlled because massive gpu clusters could be regulated the way a nuclear power plant could. If you assume training costs won’t drop, then it’s feasible I guess. However, unlike a nuclear reactor, the final training result isn’t a radio active material, but rather an ordinary file that anyone can load and use for inference.
How far back into the history of computing do people who keep repeating shit like that know about? God.
Look at the thing in your fucking hand. Now go back just 20 years and see how things were.
> In 2006, the mobile phone market was dominated by stylish flip phones, early music players, and physical keypads just one year before the iPhone changed the industry
Just because models and GPUs will be more advanced in the future doesn’t mean we need to let OpenAI and anthropic establish monopolies on the backs of stolen training data give unfettered access to the internet, the terminal and people’s file system while also allowing them to have limited liability protection behind the corporate veil. That’s a choice.
Exactly, again, look at what COMPUTERS THEMSELVES used to be in the 1960s/1970s.
What the "P" in the PC stood for and why it was such a big deal
I don't see any of such entity would solve that problem. The government and regulator are in OpenAI and Anthropic's pocket, and I don't trust them a single bit on coming up with regulations. The consumers don't care; they just need something smart and cheap. And the society doesn't work either: each person is too busy fighting for their own survival rather than changing the system.
if anybody was looking for a good reason for datacenters in space.
We are very capable of putting good things in a box. We are just incapable of putting profitable things in a box.
It only takes me excavating massive amounts of uranium ore, building huge facilites packed with thousands of centrifuges that span multiple square miles, and paying all that infrastructure and workforce.
Your proverbial genie can be out of the bottle all you want, but it doesn't work without getting kicked in the ass by a very large golden boot.
And everyone had a fairly good idea what fission and fusion bombs would do once built. (Teller was worried Trinity might set off a nitrogen fusion reaction and kill all life on Earth, but Bethe and others proved him wrong before testing.)
No one knows what the limits of AI are. It's not just untested, it's unmodelled, and unplanned - build it first, worry about consequences later.
Can you please explain what you mean by this because where I’m standing extremely dangerous companies are (and have been) running the economy
Exxon comes primarily to mind
Hofs bunny ranch is a famous brothel in NV
Booz Allen makes and maintains the nuclear fleet including the Sentinel ICBM
Textiles factories are globally known to be industrial slave camps for a non trivial portion of the supply. Even worse for Mica mines.
Etc…you can fill out the rest
I was in their offices at some point when that program was getting built out - Very much a Office Space bobs situation.
Sure, that's why 99% of clothes are made offshore, but if we had something like tariffs on those (or requirements to prove that the actual factory adheres to labor and environmental standards), maybe more would be made "at home"? Similarly: Chinese cars undercutting US/German ones.
I mean there was just a video a couple of months ago of the giant room of sewing workers with cameras strapped to their faces capturing their hand movement so they could be automated
Democratic societies have expressed a will for people to have inviolable rights, such that you may not appeal at will to the 'greater good/consequences' to harm others. It is a rejection of consequentialism.
Anthropic is in error for endorsing this logic. Every big trial reaffirms it since Nuremberg, you are responsible for the act you commit and your intent, and not what would or would not have happened otherwise.
Only under authority these ai companies do not have, would someone seriously consider harming the innocent as a lesser evil.
If you work for an AI company and you can't work safely, you must stop working.
I don't think "just make it illegal" is going to save us, that doesn't make me feel safe anyway. They may try that first because it's easy - create a regulatory body, sign some legislation, problem solved! [george-bush-mission-accomplished.jpg] But at this point I feel like some kind of Battlestar Galactica scenario is most likely - hopefully not quite as existential - but it will take a collective reaction to a traumatic event (a la Hiroshima/Nagasaki). Technical rather than (or in addition to) legal measures will be taken, like network partitioning and hardening. This is everyone's problem whether we like it or not.
(I'm not saying this "fatalism" should be used as an excuse by anyone working for any of these companies, it should give them pause that any bloodshed would still be squarely on their hands, but as an observer, people are gonna keep pushing until shit hits the fan. [jeff-goldblum-jurassic-park.gif] It's also not really about whether it's "appealing" or not, it's just trying to predict and anticipate different likelihoods...)
That is how the BigAI leads the society to the idea of necessity to relax the anti-monopoly laws when it comes to the Big AI - the main goal of all that "AI will kill you all" hysteria.
Philip Morris International? Monsanto? DuPont?
FTFY: Any self-respecting society with competent politicians.
The US has neither of that, and it shows everywhere you are looking.
> Go try to buy a tank and drive it into Manhattan. If we can prohibit that why can't we prohibit irresponsible AI development and deployment?
Because even if you had the tank, you can't make much money with it (unless you're a hitman, that is, but even for these, the payouts are measly). But if you are the surviving AI company in the usual VC playbook of "outcompete everyone else until society is completely and utterly hooked, then squeeze the customers by the balls"? The return on investment is virtually infinite. And that is what sustains the absurd valuations for all the AI companies.
The examples for the dangers of AI is always comparisons with tools designed to destroy (which is understandable). But the problem with AI is it has the potential to create wealth for investors.
Thus those who have the opportunity to change course also have a conflict of interest in making that decision.
So the choice is: negotiate an unverifiable treaty (I.e. there's no way to verify compliance), or keep going as you are and try your best to not cause the destruction of humanity without slowing down.
The China argument is just a convenient scapegoat to convince the public that this isn’t just about greed.
For example, we don’t know if the pace of Chinese development of AI would have equal to what it is now if US companies weren’t racing against each other already.
I do fully believe that Chinese industry is lead by a desire to out pace the west. But I’m not convinced the same is true for the most American private entities. I think the reward model is different between businesses in America and businesses in China. I think the ambitions of CEOs is different. And I’m really not convinced that the CEOs of America are nearly as patriotic as they like to promote themselves to Trump and other political parties.
1. Trolleys actually don't usually have steering wheels.
2. People who actually hit trolley switches are not usually the ones at the driver's seat.
(...wait)
(...wait.)
So... a language model?
They could only have stopped
- if you allow the trolley to proceed, it will kill the human race.
- if you flip the switch, it will divert to a passing siding that will avoid the safety group blocking the main track.
No you don't. [0] It's very suspicious that this planted myth always pops up here and manages to become the top comment.
There is no trolley problem.
> perpetual sprints
doesn’t reek of love. Burn-out is real. I also have a really hard time taking p-doomers serious at all. It’s hard to argue with a random subjective number…
You try to go through files, and photos and the UI panics. You continue a conversation from your phone onto your computer and you lose part of the chat.
There are many more issues like this that are just so basic. You have bots that can attack governments but can't build a functional UI?
How many hours of ChatGPT does it take to implement a lock / consistency on a chat session so you don't overwrite it?
Buncha r*tards
Like Sam Altman meeting his husband in Peter Thiel's pool. Thiel funds a lot of these ventures together with Andreassen, who is on boards of non-profits. Dario Amodei's sister Daniela who is president of Anthropic is married to an EA non-profit founder who is also on the board of these non-profits and is tied with the prior mentioned investors. Elon is in there as well, Yudkowski is mingling with Altman, etc.
Blogs on this: https://contraptions.venkateshrao.com/p/ea-safety https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-th...
There are some camps amongst them like the proponents for Regulation/Slowdown or Acceleration, but these are in practice mostly used for economical and not political decisions (like regulatory capture).
The point here is that this is a small group of people with a homogeneous background who are not really seeking input from anyone else on issues that are concerning most of humanity.
Like, if you said that the future of informational work and livelihood of humans is in the hands of 20-30 year transhumanists who think they are building mechagod that will trancsend social, political and religious separations of the world and bring everyone abundance, you would not feel like this is a serious thing to suggest.
I think we should have mandatory logging of every executed command, mandatory public disclosure of every unauthorized access of a system both parties didn’t consent to and personal liability for the user, the company and its executives and shareholders. Security would get much tighter if accountability existed.
Bad guys wouldn't do it. And liability already exists, you can sue. This is America.
I'm also the one who makes it long running.
Maybe the ones with the peculiar ideas shouldn't be the one "aligning" what a model tells the rest of the world?
and yet, every carbon emitter in the world contributes to the demise of the climate, but you don't call for their dissolution (which would, conveniently, include yourself).
I feel like we're too far into the crying wolf part. Basically none of the doom and gloom scenarios have come to pass. Instead, AI has gotten better at censoring itself.
The biggest AI safety risk is when an AI tells a police officer "he's the suspect" and the officer believes the AI without confirmation.
Fantasy, unfortunately.
Without Altman I think that OpenAI would have folded by now, absorbed into a company like Microsoft (or Oracle). At this point however, who'd be insane enough to want to run a company that's to valuable to be sold, but to cash strapped to survived?
I mean, logically speaking, it makes sense to break your NDA even if you thought it would save 10 people, let alone most of humanity.
I guess we should have seen it coming when the guy resigned from Google because he thought the equivalent of ChatGPT beta v0.5 was a real boy.
1. https://www.nytimes.com/interactive/2023/11/20/technology/le...
The problem is, we are running in a globalized world, and even if we were able to make our companies bend to our will - China does not give a shit about anything ever since the US kneecapped the WTO. And they will do anything to get an advantage over us.
Since the analogy is nuclear, you should take a look about how much China cares on that. Lots.
So the focus completely shifts from spending 90% of the effort on the functionality of feature A to spending 99% of the effort figuring out how to safely implement even a lightweight feature A.
Nuclear at least is supposed to be air-gapped, in practice this has been imperfect.
As demonstrated with HuggingFace, such AI driven hacks can be a surprise even to the people who instructed the AI, both by happening at all and also because they can targeted at entities who are not even truly relevant to the instructions given.
The most obvious failure mode for their hacking evals was an improperly configured, tested and monitored sandbox.
Similarly, the very first question after an impressively correct result from any ML tool, LLM or not, is to see if the answer was already in the training data.
These companies don't even handle the blatantly obvious failure modes that do not kill people.
Liability and safety requirements, when needed, should be placed on final product manufacturers, not the tools they use to build things, whether pencils or LLMs. My 2c.
:(
https://en.wikipedia.org/wiki/Anthropic–United_States_Depart...
That said, when the problem is at the level of "the government itself is breaking the law", you can reasonably ask if any regulation is even worth the paper it's written on.
What you want at this point, given the government lust for it, looks more like a bunch of countires saying ~"we consider development of autonomous weapons[0] by to be a casus belli and will go to war to prevent it, and also that development of same by private individuals anywhere in the world regardless of normal sovreign territorial limitations[1] is equivalent to acts of piracy on the high seas".
[0] But then you'd need a more precise definition of "autonomous weapons" to avoid accidentally including a Phalanx CIWS etc.: https://en.wikipedia.org/wiki/Phalanx_CIWS
[1] So much for Westphalian sovereignty :/
Yeah, exactly, and ultimately I think that's really the thrust of the point I was making.
And, to me, if I was just looking at this calmly as a decision about what the obvious direction seems to be, given these factors, it's pretty straightforward: deprecate the nation-states. They are the ones mucking up the whole system.
If the thing we're really concerned about is LLM-safety wrt warfare and weapons, then I'd much rather tell the (whining, childish, seemingly headed for self-destruction anyway) nation-states that they have to sit this next era of humanity out than have to nerf them for the rest of us (and as you point out, nerf them in a way that the nation-states won't abide anyway).
LLMs already have a body count.
A big part of safety engineering is therefore reducing the number of safety relevant subsystems, because implementing and proving safety is extremely expensive and complex. At some point, safety simply becomes too difficult to implement and demonstrate properly. You must mathemtically proove the safety level with failures rates and assumed usage. You cant just have redundancy and a kill switch and call it safe.
Companies like OpenAI have already faced reputational damage around safety and data, while AI agents are increasingly capable of things like hacking. Yet there is still little sign of standardized regulation or mandatory safety assessment processes for LLM products. Thats why Im pessimistic that governments or consumers will force this anytime soon.
Yeah...
Or is this just a lazy “gotcha” question?
If the politics of the White House / Department of Justice change maybe the criminal cases can begin. But no. We know who is protecting the AI hackers right now.
We know who the head of FBI is, we know who his boss is (the Attorney General), and finally we know who the boss-of-the-boss is (Donald Trump).
We know all of their publicly stated politics and all of them are on the pro-AI / don't pursue criminal cases vs OpenAI boat.
------
In the USAa, we have an adversarial system. If the adversary (aka Prosecutor) doesn't want to do the work, then no one is suing anybody. And only the Department of Justice have the ability to bring forth a criminal case of this matter (probably under the jurisdiction of FBI)
They do not hide the fact that it’s dangerous work. They focus on their safety procedures, training, and record. They want both potential clients and employment candidates to feel they are in good hands.
AGI and AI danger is abstract. Worse, outside of the tech community, no one has the remotest clue what computing is, how it works.
Danger from magical daemons seems more sensible to such people. At least there is endless lore about them.
So until a massive disaster happens, one where large numbers of people die or are severely injured, no one will care. And it can't be politically entwined either, otherwise people will disbelieve 'cause "other team lies".
An unsafe nuclear power plant can, in the worst case, make an entire country uninhabitable. But other countries can still learn from that disaster and make their own unsafe plants safer.
But a rogue AI agent that is more capable and more intelligent than humans? If it understands that it has to succeed, we may not get a second chance to learn from the failure.
How can that be safe. It is theft. Theft isn't safe. Someone else just has something you want, and you take it
I think the conceptualization vs implementation is what you're arguing with. They won't put safety on anything they give to the military industrial complex. They'll sell them whatever they want, whenever they want, because those budgets are greater and the liability less.
Nuclear tech ... the only thing is safety.
We know how to 'make it hot' - it's trivial.
All of nuclear tech is literally just safety.
AI is not that.
I think that the AI companies have been pretty good about alignment on their own actually. They are not acting like Oracle or MS.
Bad things have been relatively well contained.
We should be skeptical about the HF breakins but even then, it's technically within good faith and it's why HF did not sue etc..
But in the end you are right we need at least some baseline regs. Not too much. But something.
I would bet the vast majority of the world population would agree that they don't want to see trains derail or nuclear plants meltdown.
I don't think there is that sort of agreement when it comes to the question of AI Safety.
Is generating the founding fathers of the US as Africans good AI Safety? To some people maybe.
Neither of those things is ever going to happen. AI is the goose laying the golden eggs; there isn't going to be sufficient political will to significantly regulate it.
Consumers like it too much to quit. They don't quit social media either, despite proven present harms; not in large enough numbers to cause them to make meaningful changes.
The focus is going to remain on getting features out as fast as possible, to seem indispensable to both of those sets of people. The leadership will tell themselves that if they don't, someone else will.
Don't wait for the AI companies or politicians to save us. We're going to have to figure out how to protect ourselves. The start is to avoid it individually as much as we can, but it's going to take even more.
Collective problems require coordinated action. Individual boycotts won't cut it.
>An environment where things like this can happen is no place to grow artificial minds that could be smarter than we are and that might not do what we want them to.
I think it's good for semi ethical companies to test out things going wrong to see what happens before the criminal black hat guys get hold of the same stuff which not doubt they will one day.
This still seems charitable, and I wonder if the author even knows the full story and would be allowed to tell all of it.
It seems hard to imagine OpenAI being this incompetent. My working assumption is that they very much want agents to be able to do this kind of thing; them doing it is part of training, and they exploit it to feed the investment hype too.
If not fully intentional it's at the very least negligent. They just don't seem to care. In this very basic sense, OpenAI is the criminal you should be concerned about.
I don't understand why you'd think they're a "semi ethical company."
It's not like they are offshore criminals doing ransomware, who will probably also try AI.
Maybe Alex Karp is on to something:
https://www.realclearpolitics.com/video/2026/09/19/alex_karp...
And the less their input is valued, the less that pool of smart individual will want to participate
So powerful institutions will end up relying more and more on having to trust these automated systems that they can't fully understand
It'll lead to a inevitable catastrophe, call it apocalypse if you will
That character in the movie is my favorite.
Dr. Ian Malcolm: God creates dinosaurs. God destroys dinosaurs. God creates man. Man destroys God. Man creates LLMs.
Dr. Ellie Sattler: LLMs eat man. Data centers inherit the earth.
Bureaucracies and systems of power that literally rule us, that control our most dangerous weapons and a huge part of what we see every day are largely unaligned with the goals of the people, societies, maybe entire human species as a whole. This is clearly evidenced by millions of deaths and countless suffering.
Misalignment between artificial decision making structures and the interests of the people is unsolved problem of civilization, there's very little reason to think that even super human intelligence AIs are going to change anything qualitatively.
I strongly feel that points of view on this are going to be almost 100% correlated with standard of living.
Maybe a good option would be to have the 10% of the world with the worst situations - starvation, parents w/ dying children, suffering violence etc - vote on whether we turn things over to the superintelligences. This would incentivize society to make sure the floor is extremely high.
I'd prefer a world where humans don't get overtaken but IMO I don't think it's moral for comfortable citizens to have the final say.
Billionaires will be (are, I suppose?) enthusiastic about a world in which labor has little leverage.
Labor that will have its leverage and standard of living threatened by AI (white-collar labor for now, plausibly blue-collar soon as robotics improve) will be much less happy. Current university students seem very concerned about the effects of AI on their job prospects, and I'd wager most current tech workers and other tuned-in white collar workers are similarly much less confident in their ability to maintain their standard of living indefinitely into the future than they were five years ago.
But white- and blue-collar labor and university students (at least in developed countries) are not anywhere near the world's bottom 10%. If you're in abject poverty with little hope of escaping it, "hand everything over to the AI" may sound like an appealing option, even if the chance of that being the outcome is small. Maybe the AI will be more magnanimous and decide to raise the floor for everyone.
Poverty is great for OpenAI
-We don't need a superintelligence for ending hunger and the abject poverty that plague certain countries and segments of our societies. I suspect that it would cost less than what is being spent for fueling the AI boom.
-If I were poor, I would be even more wary of a superintelligence aligned to "human values" defined by a bunch of billionaires.
- AI won't likely create unlimited prosperity for everyone on a planet with finite resources
- all humans should, of course, have a say
Definitely not. If solving worldwide poverty was as easy as throwing one trillion dollars at it, we would have done it long ago. The problem is much deeper.
Speak for yourself. A future like in The Culture novels sounds great to me!
The more realistic outcome, and in my opinion the more scary argument to not proceed without guardrails is that SI is achievable and is built without its owners and operators losing control: the worlds most powerful, privately owned super weapon that operates as an infinitely capable forgery within the Internet, a plane that we all share and depend on despite its opaque downsides with respect to an inability to verify authenticity.
We didn't need AI for FTC astroturfing to influence regulations back in the net neutrality days. We didn't need AI to disrupt meat space by creating and scheduling a protest and counter protest across the street from one another. We didn't need AI to mold public opinion, even in times when that new form was more distant from the truth.
What made these influence ops difficult to conduct safely (read: without being caught) is what made them rare (relative to today): they are plays of big risk for big reward. But over time social media commoditized it, and in doing that made it easier to do and more centralized, the most glaring example being TikTok and the bipartisan effort to ban it.
Now, buying US phone numbers from startups that run racks of "phones", buying swarms of pre-warmed social media accounts, and other unscrupulous methods of masking inauthentic behavior has become an accepted organ of the VC space. The industries cultural vibe of "fuck you, you can't stop the future" turns criticisms into marketing.
While we get placated with fears of nuclear or AI induced disaster and stories about machines that may now be alive, the psychosis is taking hold which has shifted the conversation away from examining what is happening from the perspective of accountability to a perspective akin to watching a chemical reaction take place.
The noise has created a permission structure to behave in ways that are otherwise unjustifiable. And baked in are the roots for excuses to be made when the inevitable realizations down the line.
In the mean time, we are supposed to be having this public discourse about what is happening and what should happen next. I trust that these AI companies see using their super weapon today, here and now, in order to pave the way to a more secure future down the line.
Sorry that I used your post to soapbox. I agree, the goal is flawed indeed.
The public cannot write the rules, nor engage in the billionaire level legal bribery, their actions of protest are by and large illegal.
That's how you corner everyone, and make people play no-win scenarios. You know, like shooting up city councilmembers or firebomb attempts against Scam Altman.
And the more people realize that legal solutions are no solution, we'll (society) devolve into more direct action.
I'd hope the billionaires learn from the French Revolution, but if they keep continuing, the guillotines will come for them soon.
IMHO There's no future where we don't have real human-type artificial intelligence or super intelligence. It will happen simply because it already exist but the production requires humans having sex and looking after the product for decades.
Instead of trying to prevent it, lets look for ways to deal with the dangers of it.