Will SB 1047 make AI less safe?
Dangerous consequences of shortcutting common law
Former OpenAI safety employee Scott Aaronson, over at his blog, has endorsed California’s SB 1047 AI legislation, based on several of the same arguments that others have put forward. One particular statement triggered me, because I’ve seen it repeatedly even though it misunderstands how SB 1047 shortcuts common law. This shortcut means SB 1047 risks making AI development less safe.
Scott says in the comments on his post:
"anyone who thinks it’s ridiculous to imagine the current generation of AIs causing serious harm, should for that very reason have no objection to a liability regime, which only starts to have non-negligible consequences in the event that there is such harm!!"
Of course, there is a existing liability regime right now, without SB 1047. If a covered model caused harm today and a model developer was sued, a court would decide whether the developer’s practices met a necessary duty of care. That duty of care would be established via analogy to other situations, and would be honed over time, proportionate to the injury caused. This common law process is grounded in actual injury and experience, rather than speculation.
SB 1047 shortcuts that knowledge-building process. That’s what makes it different than the status quo. SB 1047 requires companies to file detailed safety plans ahead of time. SB 1047 does not dictate substantive requirements for how to create safe AI, because no one knows the appropriate trade offs yet. However, under SB 1047, if anything goes wrong, courts will use the filed plans to establish whether a developer’s conduct was reasonable.
This shortcut standard of care has a very predictable outcome: everyone will file and execute extremely similar AI Safety plans. Why will they do this? Because if anything goes wrong in the future, the best court defense that you were acting reasonably will be that you did exactly what everyone else did.
What are the consequences of this effect? I see four different scenarios, depending on how events develop.
Scenario 1: No critical harms happen because the standard of care that everyone chooses now is exactly the right mitigating conduct to prevent all future critical harms, even though no one can write it down in the law now. Congrats, this is the SB 1047 winner scenario. Nailed it guys.
Scenario 2: No critical harms happen but for reasons unrelated to SB 1047. Perhaps the tech simply isn't that dangerous, or we have lots of other mitigations. In this scenario SB 1047 imposes deadweight costs with no benefit. (Although both Scenarios 1 & 2 have no critical harms, we may be able to distinguish them in retrospect by observing whether critical harms occur in jurisdictions where SB 1047 does not apply.)
Scenario 3: A critical harm happens because someone didn't comply with the procedural requirements of SB 1047. This is sort of a win for SB 1047, because although it didn't prevent the critical harm, the AG can get some extra $$$ from the violator. Yay? (This seems like a very unlikely scenario, as people will file the paperwork.)
Scenario 4: A critical harm happens even though people complied with SB 1047. THIS IS THE REALLY BIG PROBLEM SCENARIO, because it means SB 1047 made AI safety more brittle. By shortcutting the common law process, there is no evolutionary pressure to improve and every incentive to freeze AI Safety approaches. Everyone copies each other’s AI Safety approach, so if one fails, they all probably fail in exactly the same way. (They also all probably escape liability, although God knows the AG is going to want to nail somebody.)
So, where does that leave us if we adopt SB 1047?
Scenario 1: Better than status quo because we set the perfect reasonableness standard, even without experience, and despite the fact that no one can write the proper standard down in law today.
Scenario 2: Worse than status quo because unnecessarily spent money and resources.
Scenario 3: Same as status quo except, under SB 1047, extra $$$ and an AG gets a career boost.
Scenario 4: Way worse than status quo because AI Safety becomes homogenous and ossified.
You might agree that these four possibilities cover the options yet still support SB 1047, if you believe Scenarios 1 and 3 are much more likely than Scenarios 2 and 4. I personally think Scenario 1 is exceedingly unlikely, and three provides very little net positive benefit. Having watched companies flock to comply with SB 1047-like FTC Consent Orders, I think Scenario 4 is dangerously probable.
And thus yes, SB 1047’s shortcut of common law could make AI less safe.
* * *
To write better AI laws, we need to red team AI legislation. I am standing up such a red team. If you are interested in helping with this effort, please reach out to me at neil@abundance.institute. Thank you!
Agreed on all points. One other thing reading the law that has stuck out to me is that it mirrors the worst elements of drug patent regulation. Models only coming into the regs when they pass an arbitrary 100M in development costs means that companies will build on each other just under that cap, or shift money around to license “materially different” models under the cap. Pharma drug makers do the same to tweak old recipes in order to extend monopoly control. Here there’s no basis for whether the law applies when a company strings together 100 1M models into their product or creates 100 99M models that barely differ. It’s both easy to circumvent and unclear when the law will be applied at all.