But I feel the only reason Chinese models have been able to gain such popularity is BECAUSE they are open source.
I wouldn’t trust models from China enough to use them in important work unless they were open.
Restricting any powerful Chinese model [that ranks in the global top 10] is going to make them very difficult to advocate for, and very hard for global users to adopt.
Feels like this could be the beginning of the end of the democratisation of powerful AI.
And yes, I appreciate the irony that Chinese models have been the ones enabling that democratisation!
Latest and greatest China is able to make is reserved for the use of businesses and entities in the Chinese market.
Everyone else gets access to models of more limited capabilities.
It's basically the US strategy. Obvious difference being only that the Chinese allow on prem hosting of the lesser models. So plus one to them I guess.
If we want to move China or the US on from the whole, "super intelligence models for me, more limited intelligence models for everyone else" strategy, then "everyone else" will have to come up with some models competitive with the latest and greatest from the US and China. Absent that, there's just no pressure to bring the US and China to the "share the wealth" table.
Unfortunately, "everyone else" coming up with a competitive model is not gonna happen anytime soon.
It makes it harder to extract decent margins from models at or below the quality of the open models, and so makes it harder both for new entrants and for any existing providers that are depending on cheaper models for revenue. At the same time you make people comfortable with your product.
It's the IP version of dumping products below cost if you're capital rich and want to winnow down the competitive field while "buying" customers.
The models they dump may not be good by the latest Chinese or US standards, but they are far and away better than anything anyone outside of China or the US is able to come up with.
So until we start seeing EU models or South American models or African models that are better than the models being dumped onto the market, China will not be giving us anything better.
And again, it's gonna be a long long time before the rest of the world even gets to the effectiveness of some of the Chinese models being dumped. Let alone competing with the best models China or the US possess.
A model is never neutral. A model is always also a way of disseminating an ideology, and it doesn't need to be lying for that to work.
Consider the question "what caused the French revolution?"
There's many ways to answer this while being technically correct. What gets prioritized, what gets omitted, that's where the ideology lives.
(Unless someone totally bonkers spends thousands of hours and thousands of pages answering a single question)
But once we automate that, it drastically lowers the amount of viewpoints represented.
Because they help nudge to at least some infintesimally small, non-zero, degree.
This argument seems to apply to a much broader set of things than just “AI”.
But if there's an open model available then they might choose that instead of either of them. Even if the Chinese models aren't used they will bring down profit margin just by existing. They wouldn't have the control the hosted American models give but that wasn't even on the table. But denying America control is a worthwhile strategic goal in and of itself. And they don't even need to expend extra resources to do it, they needed to develop their own models for security reasons anyway.
So you've inspected the weights for backdoors?
If you're using a closed model, the model provider gets all your data, and can if they want to send back actively hostile tool calls and rely on your harness to catch them without any effort on their side to train that behaviour into their models. The only things you can do about that is to place careful restrictions
If you're using an open model against your own system or a third party provider, the origin of that model could have trained in "triggers" that'd cause the model to produce actively hostile tool calls as well, but they'd have no direct control and would need to train in that behaviour "blind" to how you'd actually use the model.
It doesn't mean you can be complacent about trusting the model, but it does change the likelyhood they'd be able to do any damage, not least because you'd be able to run evals against the model without any risk that the model or provider API could just randomly change on you after a while.
So it takes far more trust to run a closed model than to run an open model even on a hosted third party provider, and certainly far more than to run an open model on your own hardware.
Why do people keep calling these "open source" models? They're not that different from NVIDIA's binary blobs. Sure, they work, and you can run them and get results, but the source code, training data, and training process are largely hidden and proprietary. Publishing weights is not the same as publishing the source. They're open-weight models, not open-source models.
In my mind, this meets the conceptual and technical bars of an 'open source' designation. We can also imagine the opposite, where inference itself has to pass via an inscrutable binary blob that keeps the model's technology stack fully proprietary.
You're correct that we're missing the training data, the training recipe/curriculum, and any secret sauce that went into optimization of same, but I don't think that's an essential feature of an open source designation.
Instead, I think that the model weights are a kind of rendered artifact, and it should be evaluated on that basis. We don't treat permissively licensed images as something lesser, even if the photographer or illustrator supplies only the final image rather than the full camera raw file and/or Photoshop layer stack.
The difference, of course, is that the rendered artifact (weight blob) in a language model is so much more important than a typical photograph.
you're describing open weights, not open source
I don’t see how this is any different. It’s like saying Intel has to publish their transistor physical library and microcode to be “truly” open source. Like I don’t have the internal notes about the most recent microcode changes (which is akin to “training process” in the analogy).
This may be true now but probably won't be true in the future. My guess is that part of the reason for this is that the major American technology companies are acquiring talent to build out models and related capabilities. China doesn't (or at least didn't) have that same dynamic.
Rome is not what it once was, and China has plenty of universities, bright people and capital of its own.
I don’t see how anyone, looking at the past 30 years, can be so sure that the US will out-do China. But good on you for being an optimist I guess.
I don't think it's a stretch to suggest that part of the reason American startups haven't released or haven't released as many successful "open weight" models or related companies is because we have a lot of really high-end, top-tier tech firms which are hiring most of the talent and paying them a lot. It's not just Anthropic and Open AI. It's basically every tech company, plus finance companies, airlines, &c. who are researching and figuring out how to make their own companies better.
If you find that to be something wishful or you have trouble understanding, I think instead of combatting my comment you should ask yourself why you think that American tech companies aren't hovering up exceptional talent and that has had an impact on startups or other tech companies launching similar open-weight models.
You also need to understand the economics better and talk through those. Investors in the US don't generally have state backing so they may be more diligent in what they invest in.
> Rome is not what it once was, and China has plenty of universities, bright people and capital of its own.
Sometimes I think people are internalizing the troubles in Europe (I know you mean Rome as in the US being the new Romans or something) and externalizing those issues at American problems because of Donald Trump, but I would caution against this. China has plenty of universities, bright people, and capital of its own. Yes. It also has disadvantages. The US also has advantages and disadvantages.
> I don’t see how anyone, looking at the past 30 years, can be so sure that the US will out-do China. But good on you for being an optimist I guess.
We've proven we can build competitive and capable companies. I'm not being an optimist, instead I'm just calling it like it is. Our ability to continue doing what we already have demonstrated we can do isn't taken away by China (or other countries) also doing those things. There hasn't been any material change in the ability of the US to build world-class companies in recent history.
I am pretty sure Facebook has clever people and deep pockets, but it hasn’t materialized properly. I cannot say for the very latest releases, but Google too has been a bit of a laughing stock on this front - same story.
As for viewing things with an economic lens. I profess to having only a moderate grasp of economics, I do better with history and geopolitics- and I think this is another blind spot. Americans largely view the world as “business as usual “, but you profoundly underestimate the animosity, concern, suspicion and outright fear other countries have now. There is simply no way our entire infrastructure will be hooked up to an American “super intelligence” firm where the president calls every other Tuesday to extort a country. And failing to appreciate that leads to vastly more rosy ideas of the growth potential of OpenAI, Anthropic and others.
Ok - we just disagree there. It seems incredibly obvious to me.
> As for viewing things with an economic lens. I profess to having only a moderate grasp of economics, I do better with history and geopolitics- and I think this is another blind spot.
I do better with history and geopolitics as well, my point was just that no matter how "open" the so-called open models are, money is what matters and you have to be able to sell the model or associated infrastructure around it (in software terms but perhaps also hardware in the future) to keep your company going and to make money. There is no way around this.
> Americans largely view the world as “business as usual “, but you profoundly underestimate the animosity, concern, suspicion and outright fear other countries have now.
Sure, but on the other hand this exact comment has been parroted all over the Internet (which has little in common with what most people actually think) on select forums of slightly more intelligent than average people to the point where the extent to which Americans view business as usual is being underestimated we can also say that random people on the Internet trying to tell Americans they are so big mad at us is itself an overestimate of how people feel.
I'd also like to point out - which countries, exactly? And just how profoundly are they big mad at the US? If they are profoundly angry, concerned, suspicious, and fearful, and perhaps they are European, maybe we should immediately withdraw from NATO, withdraw military forces from Europe except where it benefits us directly, and we can remove sanctions on Russia because well Ukraine is in Europe and we want cheaper oil and gas here at home. After all, you can't simultaneously claim that there is all this distrust and then trust us to have armed soldiers in your countries.
There's a strong contradiction between rhetoric and action and it plays out very obviously at times like this. You see the same thing with big, grandiose statements from European leaders who want to talk tough and appear like they're "standing up" to Donald Trump, but behind closed doors it's always a different story.
> There is simply no way our entire infrastructure will be hooked up to an American “super intelligence” firm where the president calls every other Tuesday to extort a country. And failing to appreciate that leads to vastly more rosy ideas of the growth potential of OpenAI, Anthropic and others.
Yes but then you also can't be hooked up to, idk, Chinese "super intelligence" firms because they also can do that to you and have demonstrated that they will punish European countries for saying the wrong thing about their dear leader or whatever. You're also assuming that the United States will allow other countries to hook their infrastructure up to our AI infrastructure. It seems likely, but in terms of the latest, state of the art models and capabilities I'm not so sure that non-Americans will even be permitted to use those, at all.
If the promise of AI as being generally stated is true, it actually makes sense for Americans to have sole access to American AI technology so we can build and expand our businesses and maintain competitive advantages. China is also restricting models with fresh news as recently as last month on that front. Perhaps a likely scenario is that China and the United States simply come out on top and the rest of the world is left to suffer what it musts.
Another area that seems strange to me is the insistence (and honestly I think it's just a belief in how someone wants the world to be more than it likely will be) that American tech companies, run by some of the smartest people on the planet and investors, American ones who have been doing this for a long time and have access to incredibly deep capital and resources to evaluate companies are all somehow wrong about spending a trillion dollars or more (whatever the number is it's very large) on AI buildout. It just doesn't seem likely besides a typical boom/bust which is a healthy aspect of our economy. Somehow, they are all wrong and hey some Chinese companies released some open-weight models so I guess all of these people in America are totally wrong and everything will be free and open and run by China and somehow they won't build companies to make money they'll just work for free and release open-weight models and by the way Europe will never use American ones!
I'm just calling this to be all bullshit. Nobody is providing hard numbers and nobody seems to have an explanation as to why American companies can't if they want to just reverse engineer Chinese models if they were better. Let's say investors go out and lose their t-shirts on Anthropic and Open AI? Yea that sucks, so what? We'll just build more and better models. If China "gets ahead" we'll just do the same strategy to them that they seem to be doing to us.
Here's another question - why isn't Europe even in the topic of conversation about AI? Where are the companies? If the Chinese models are just cheap/easy to use or whatever why aren't we seeing a number of new AI startups? If you are going to accuse Americans of being unable to build and release open-weight models, what does that say about European companies?
This isn't a pro-American point of view, though I am pro-America (regardless of who is president). I just find the arguments put forth in this topic to be either of poor quality, or too strongly ideologically tainted.
As regards nato - again, you seem ignorant. The quiet part isn’t said out loud by anyone but the president who threatens to abandon Nato commitments, but yes, European leaders are also considering plans where the US isn’t helping.
And again, US deployments here are minuscule, approximately 2 months worth of Russian casualties. The value was always the deterrence. And the value for the US was a gateway to the middle east and further abroad along with highly favorable deals for its military industry.
The US has already been nearly useless these past 2 years. We fund the entire war and you sell the arms. The only help has been intelligence, which is hugely helpful, but costs you nothing extra.
I don’t think there is any point continuing a discussion from there.
Please refrain from writing things like this or just don't respond.
> I find it curious that, I, a European, consuming European media, living in a European country and consuming its domestic media, which you very likely don’t, am being lectured, by an American, on how we feel.
You're just one random person though. I'm in Europe often and talk to Europeans all the time. How you as a random person on the Internet feels isn't reflective of how everyone feels. There's nothing "curious" here, nor am I "lecturing" you aside from your mistaken belief that your own social circle is how everyone feels.
> The quiet part isn’t said out loud by anyone but the president who threatens to abandon Nato commitments,
Can you quote a specific article, source, or comment where the President has threatened to abandon NATO commitments?
> US deployments here are minuscule, approximately 2 months worth of Russian casualties.
The US deployments in Europe are actually quite significant. Your framing of them in terms of Russian casualties when Russia just pushes people to front lines is obviously flawed. The deployments over time have lowered because the Soviet Union and Marxist ideology that took root there was defeated by the West, but in today's world the value America can bring to a fight in Europe is logistics and air power. 600+ million Europeans need to do the infantry work and go clear Russians out door-to-door, not Americans.
> The value was always the deterrence. And the value for the US was a gateway to the middle east and further abroad along with highly favorable deals for its military industry.
Those are nice to haves but not required, and the value for Europe is US protection. The framing that typically occurs is how there are all these benefits for America and all the bases are good for us, which is wrong since Europe gains even more in benefits.
Maybe we don't need to have troops deployed all around the world anyway? Let's withdraw from South Korea and Japan. Maybe Germany can stop China from taking over Taiwan. Perhaps France and the UK can partner with Saudi Arabia to stop Iran from building nuclear weapons? Or maybe the US can withdraw from Europe and Europeans can spend another trillion Euros on their military? Sounds good to me if that's what you are advocating for.
> The US has already been nearly useless these past 2 years.
incorrect
> We fund the entire war and you sell the arms.
So? It's not turning out well for Europe right now. Look at the riots and protests. How is Europe going to sit here and pay $70+bn to Ukraine when that's more than Russia's entire military budget?
> I don’t think there is any point continuing a discussion from there.
I do as well, no need to reply from here.
So much for being one person. Again, you are actually being arrogant when presuming you know better on this very topic.
Open source model would provide you with all the data & tooling necessary for you to rebuilt the model from original or modified data.
The reason why it’s not done is because it takes a huge amount of resources to not only record the training process but to release it in a readable format.
What you’re asking for is almost like surveillance documentation for a model that’s likely put together in a way that’s not easily peaceable into clean step by step procedures.
One issue is that in the end, open models are still comparable to "freeware" from the old world. You have a black boxed binary to takes input and produces output. On typical "freeware", you could still reverse-engineer it and have have confidence to map assembly to actions it takes. But with these open models, not so much. So, we still can't put too much trust to these models, even if they are open. There can be some backdoor that is impossible currently to identify based on reading the model weight numbers. We can try to post-train models and reduce the likelihood of some "known" stuff, but we don't know what unknown there is. Interesting times ahead.
It's possible to embed malicious behavior into models, such as "secretly phone data that meets these requirements back to this address..."
This has been shown, and anyone who does cybersec should be aware that "open model running locally" isn't fool proof.
Edit: People might think I am referring to the cohere incident, I'm not, I'm referring to other labs that have purposely experimented to see if this is possible. It is.
"Second, we encourage open source, open collaboration, and resource sharing. The Chinese government is committed to making artificial intelligence a significant driving force for promoting common prosperity and safeguarding common security. To support the construction of an open source innovation ecosystem for artificial intelligence, convenient measures such as automatic licensing will be applied to technology exports involving open source AI technologies, Chinese companies building and operating their own computing centers overseas, conducting chip fabrication overseas, utilizing overseas resources for joint research and development, and participating in the formulation of international standards, basically achieving "immediate reporting and export." Technologies that have already entered the public domain through open source or published academic papers will not be included in the export management scope, creating a favorable environment for international exchange and cooperation in artificial intelligence technology."
their official stance suggests they remain in support of open-weights
What they are saying is that they won't be changing anything.
That could of course change overnight.
There are multiple aspects to this: China is cottoning on to the meaning of soft power, and besides that being a major supplier of AI related models, services and hardware further cements their place in the world economy. They may not be able to compete directly with the USA in this domain but they are able to undercut them.
Of course there are risks as well. Models produced and released into the wild for which you have not seen the training and conditioning data could have payloads that you'd rather not have. But the same thing goes for service providers in other countries as well. You can't really trust hosted inference, not to tell you the truth in so far as is was present in the training data (censorship, not inference errors) and not to leak your information.
So they're changing some things.
“Right now, your government is doing things you think only other governments do.” -Van Halen, Right Now
My (not thorougly informed) take is that they are are taking measures to have the capability of restriction but are also adding provisions so that this capability does not add friction to the current practice of enabling export of open weight models.
So to me it's like they are keeping their options open rather than leaning on one versus the other.
What they will do remains to be seen. The document at hand is what they're saying.
I.e. less important if we follow your own assessment.
Says who?
You can also download offline PDF at: https://insanai.github.io/aiexportcontrols/books/en/open-sou...
It also shows using game theory that export control leads to overall failure of both the sides.
Of course I never expected anything different from the Chinese Communist Party.
If China’s selfish behavior = more open LLM tools and greater ability to run them at home, I’ll gladly take it over the US frontier labs’ druthers, which is a restricted/closed shop we rent from.
Part of the problem is that there are too many major stories happen back-to-back nowadays, and important stories like this one never get discussed. HN only shows 30 stories on the front page, which limits discussion.
Additionally, the financial and technological equilibrium which makes OpenAI or Anthropic possible is rather precarious so they are poking on it, hoping to make it fall.
Think about the history of AWS. Amazon made a very specific point of building, specifically, a “platform”. When they developed a major internal service, they would sell it. Basically anyone who wanted to use it could pay AWS (often an outrageous amount, but it was often worth it), and AWS would provide the service, and AWS was consistently neutral and trustworthy. They didn’t really tell you what you could or couldn’t do, they didn’t even ask what you wanted it for, and they never used your private data for their own purposes. AWS pulled this off quite well despite the fact that competing products and even open source products could and still can, with effort, do most of what AWS offered.
Compare to OpenAI and Anthropic. They want to sell you their front-ends (chatgpt.com, claude.ai, their apps and harnesses), but the big money is selling enterprise contracts with per-token pricing. The latter is the platform.
But their platforms suck in comparison to their open competition. You are paying for inference on a series of models, but:
- You need to negotiate with them if you don’t want them to retain your data.
- They might not even be willing to not retain your data.
- Part of the model output gets encrypted before you see it.
- You are not allowed to give the model arbitrary input.
- They run classifiers on the token stream to try to restrict what you may or may not do.
- You cannot see the model’s internal states.
- You are extremely restricted in your ability to modify the model and you may not be allowed to modify it at all.
- Oh, and maybe the price is above market rates.
The last point is not really a big deal: AWS has demonstrated quite effectively that customers will pay many, many billions of dollars above market rates for a high quality platform.
But I think the rest are a big deal, and the open-weight models allow anyone to use a provider other than Anthropic and OpenAI and avoid all the other problems in the list. This means that, if I’m a startup building a product that uses AI, I get to choose between OpenAI/Anthropic-like terms and a far better alternative. And this makes it very hard for OpenAI or Anthropic to sell me on their platform even at equivalent pricing.
So the real question you should be asking yourself is why you thought that China supported publishing open-weight models, even though they merely didn't prohibit it.
OpenAI had been miniaturizing GPT models as far back as GPT-4o era, but Chinese free-beer models that run on 16GB consumer GPUs can handle the vast majority of LLM use cases, and now they have no options but to attempt development of actual superintelligence that can cure cancer or solve the P=NP question just to differentiate against free-beer alternatives. It's clear what China is trying to achieve.
The Chinese open-source models (which are 1-2 generations behind the frontier) cost in the low tens of millions to train. This is largely because of a lack of compute. They achieve comparable performance to frontier models via distillation.
Of course training something like a 1T LLM will probably take some time :-P but with enough people contributing it might be possible to train something small - and if the project reaches consensus, it might be adopted by some institutions with a bit more available compute power too (e.g. universities).
If i had a second PC (i don't really trust the project right now to put their auto-updating client on my main PC - maybe if it grows and there are more people behind it or there is some better governance than "trust me bro" i'd put it in a container or something) with some GPU power i might have contributed a bit too since i like the idea.
somebody wake me up when we have "SLSA, but for model provenance".
I don't (just) want "open weight + verified distribution". I want a dang audit trail.
If/when one of these models is trusted enough to make real decisions with teeth, and if/when one of those decisions screws up badly enough to hurt somebody, I want to be able to play back the tape and pinpoint the source dataset + training method responsible for the failure. At LEAST at a coarse level, on par with something like modern RPO SLAs.
P.S. this comment is applicable to pretrained models in general. I don't care if it came from China, USA, or model weights scrawled in sharpie on a random bathroom stall. I just want to know the provenance and have enough information to run a blameless "git blame" (or equivalent) if/when things go wrong.
P.P.S. IMHO, we aren't there (yet), but OTOH this proposal ain't a bad place to start.
Most people just don't understand how many compute operations occur to train a modern model, and just how infinitesimal any particular data point is. Even worse is when a model generalizes a function. You have to now prove the entire generalization.
Not even remotely possible. Not with our current understanding of how these things work.
This was a topic of discussion at Trump and Xi's AI summit just last month. We're now starting to see the export controls coming into place.
It's not in the interest of any government, or even any AI lab, to democratize dangerous intelligence/ability.
China wants to undercut the west, of course, but it's looking like good/bad model behavior are two sides of the same coin. "Show me how to protect my server from the most esoteric attacks" is simultaneously "Show me how to attack a server with the most esoteric attacks."
Hell, I expect before long hardware is going to have all kinds of lockdowns occurring.
Maybe not for long. You need another talking point.