P-0009 ● Active

“Releasing the weights of frontier AI models increases catastrophic risk.”

Community stance
No votes yet
§
Meaning & context

Open-weight models (Meta's Llama, DeepSeek, Mistral, Qwen) can be downloaded, run and modified by anyone. Their safety training can be fine-tuned away, and release cannot be undone.

The disagreement turns on marginal risk. Compared with what a bad actor could already do with the internet, closed models and their own skills, how much do open weights add? Optimists note that studies so far find little uplift and that openness speeds defensive research. Pessimists say the relevant question is about future models, and that irreversibility makes "wait and see" costly.

Releasing the weights
▶

Publishing the trained parameters so anyone can run and fine-tune the model, as opposed to offering access only through a monitored API.

Catastrophic riskContested
▶

Risk of mass-casualty events, chiefly biological or chemical weapons, or large-scale cyberattacks on critical infrastructure.

Filter by type:
↕
Arguments
6 total
↑ Supporting3
Empirical

Safety guardrails on open models can be removed cheaply. Once weights are public, any refusal behaviour can be fine-tuned away by anyone with modest compute.

Researchers have shown that safety fine-tuning of open models can be undone with a few hundred examples and minimal compute, producing models that comply with harmful requests.

Gade, Lermen, Rogers-Smith & Ladish (2023). BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13B. arXiv:2311.00117.

Limitation

Removing refusals is not the same as gaining dangerous capability. The model only knows what it was trained on.

Predictive

Release is irreversible. If a future model turns out to give serious uplift on bioweapons, an API can be patched or shut down; downloaded weights cannot be recalled.

In a 2023 exercise at MIT, participants using an uncensored open model were able to obtain information on how to acquire and assemble pandemic-capable viruses within an hour.

Gopal et al. (2023). Will releasing the weights of future large language models grant widespread access to pandemic agents? arXiv:2310.18233.

Limitation

The information obtained was largely available elsewhere, and the study had no control group. Critics argue it shows little beyond what search engines provide.

Moral

Responsibility diffuses with openness: no one can monitor misuse, detect suspicious patterns or ban bad actors, all of which API-based deployment allows.

Closed-model providers report detecting and disrupting state-linked misuse of their APIs, including influence operations and malware development, by monitoring usage.

OpenAI (2024–2025). Disrupting malicious uses of AI (threat reports); Anthropic threat intelligence reports (2025).

Limitation

Sophisticated actors can train their own models or steal closed weights, so API monitoring may stop mainly amateurs.

No arguments of this type on this side.
↓ Opposing3
Empirical

Controlled studies find little measurable uplift from current models over internet access for planning biological attacks. The marginal risk of open weights is small so far.

In a RAND red-team study, teams with LLM access did not produce significantly more viable bioweapon attack plans than teams with internet access alone.

Mouton, Lucas & Guest (2024). The Operational Risks of AI in Large-Scale Biological Attacks: Results of a Red-Team Study. RAND RR-A2977-2.

Limitation

The study used 2023 models. Capabilities have since grown, and the claim is about frontier models, including future ones.

Moral

Assessed by marginal risk, most feared harms of open models are already possible with other tools, while openness brings large benefits: research access, security auditing, competition and independence from a few companies.

Kapoor and colleagues proposed a marginal-risk framework and found that for most misuse categories evidence of marginal risk from open models was weak. The US NTIA's 2024 report recommended against restricting open weights for now while monitoring risks.

Kapoor, Bommasani et al. (2024). On the Societal Impact of Open Foundation Models. ICML 2024; NTIA (2024). Dual-Use Foundation Models with Widely Available Model Weights.

Limitation

Both recommend continued monitoring, conceding that the balance could change at higher capability levels.

Predictive

Open models lag the frontier by roughly a year. Restricting open weights mostly delays defenders and researchers, while capable actors such as states have other routes to equivalent capabilities.

Epoch AI's 2024 analysis estimated that the best open-weight models trailed the best closed models by roughly one year on benchmarks and training compute.

Epoch AI (2024). How Far Behind Are Open Models?

Limitation

A shrinking lag also means dangerous capabilities would reach open models soon after closed ones, which supports the risk side.

No arguments of this type on this side.
≈
Areas of agreement
3 points
Empirical

Safety fine-tuning on open-weight models can be removed by anyone with modest resources.

Empirical

Open release cannot be undone.

Moral

The right policy depends on capability level: almost no one wants to restrict small models, and few would release a model with proven mass-casualty uplift.

P-0009 · Last edited Oct 26, 2026 · 1 revision