Meta releasing open source AI models is either the best or worst thing for safety and I cannot figure out which
I follow the AI field closely enough to have an opinion but not closely enough to be confident in it. The debate about open source AI models genuinely confuses me and I want to think through it with people who have considered it carefully.
The case for open source: transparency, the ability for researchers to audit models for problems, democratisation of access, competition with closed systems that might otherwise have unchecked market power. These all seem like real benefits.
The case against: once a powerful model is released openly you cannot un-release it. Bad actors can use it without the safety constraints the original developer built in. The most capable open models are potentially dual-use in ways that matter.
What I cannot figure out is whether the safety risks of open release are actually higher than the risks of having a small number of companies control very powerful closed systems with no external oversight. Has anyone thought through this tradeoff carefully rather than just picking a side?