This is part of a series where I'm trying to learn to think for myself. In this post, I go through Demis Hassibis's latest X article and analyse it. Then, I look at reactions of other people and finally, I read what Zvi has to say about it.
First Demis lays out the vision. Why is he even building AGI?
AGI is only a few short years away.
The magnitude of this technology's impact will be unprecedented, perhaps 10x of the Industrial Revolution at 10x the speed.
It will help us solve some of the biggest problems society faces
Putting the vision first makes sense because if someone only talks about the risks, a natural reaction is "Stop putting us in risk then!".
Then he lays out why he's worried.
Urgent action is needed to address risks that might arise as we get closer to AGI.
Cybersecurity, nuclear and bio risks may emerge.
Need robust safeguards to maintain control of increasingly agentic, recursively self-improving systems
"we need to give ourselves time"
At the moment, we are locked in an extremely intense, multilayered commercial and geopolitical race.
Need new approach to testing frontier AI model capabilities that is dynamic, adaptable, and rigorous
He then proposes a framework.
Establish a new Standards Body modelled on a federally overseen public-private partnership or self-regulatory organisation (FINRA)
The Standards Body would be responsible for developing assessment protocols and working with appropriate federal agencies and the US National Labs to conduct testing in areas relevant to national security.
This standards body will define what a "Frontier Model" is.
A model would qualify as 'Frontier-class' if it meets certain thresholds on a set of benchmarks determined by the Standards Body
Organisations with 'Frontier Models' as defined by those benchmarks would be deemed 'Frontier Labs'
Being designated a Frontier Lab would carry significant prestige and be open to any organisation by building models that meet the benchmark criteria.
Being designated as a frontier lab means having to play by a defined set of rules.
[Frontier Labs] will be encouraged to adopt best practices, such as publishing model cards with technical details, maintaining strong internal cybersecurity, vetting key personnel, and providing sufficient resourcing for safety and security research, and more.
Encouraged is a soft word. I wonder if that was intentional.
Initially, Frontier Labs would voluntarily share models with the Standards Body for review up to 30 days before release
Frontier Models would be required to pass [a review by the Standards Body] to be deployed in the US market
Model assessments should include rigorous scientific evaluations of capabilities in cybersecurity, biological threats and other high-risk domains.
The framework could apply to Frontier-class models no matter their country of origin or whether they are open or closed, but any non-frontier models, say from startups or academia, would be exempt from this process.
He says this body will allow us to coordinate a slowdown if and when required.
It is designed to keep up with the field's acceleration and adapt to the biggest risks as they are identified, and could be ratcheted up if the seriousness of the situation demands, including coordinating a slowdown in development among the Frontier Labs if deemed necessary.
And then he ends with
There is both huge excitement and uncertainty around AI, and both are warranted. But the future is not yet written, we must use this precious window before AGI arrives to shape this technology for the benefit of all humanity. What we collectively do now will determine how the next phase of civilisation unfolds. By safely stewarding AGI into the world, we can enter a new golden age of scientific discovery and progress, and usher in a bright future of incredible human flourishing.
Okay, now my thoughts.
Seemed like a good proposal to me. It is regulation that only applies to the frontier labs. It creates some means of communication between the frontier labs and a pathway to enact a slowdown/pause by the U.S. government. Fable was struck down because of an ad-hoc decision by the USG and this standards body seems like a better alternative.
How would Ilya's SSI fit into this framework though. They don't release models into the public. They wouldn't be deemed a frontier lab because they never release anything. Seems like this proposal does not address that. To be capable of releasing a frontier model, you need some minimum level of compute. This framework should also include organisations who own GPUs summing up to that level of compute and these organisations would have to adhere to the same principles.
People's reactions
A lot of people seemed to agree that it was a good proposal. Including Sam Altman, Jack Clark, Elon and others.
Some called it regulatory capture. I don't yet understand the concept of 'regulatory capture' fully but from what I can see, this proposal does not preclude anyone from becoming a frontier lab. The regulations kick in after you've created a frontier model. With great frontier models, come great responsibilities.
Connor Leahy appears to be in Stop AI camp?
Brian Armstrong makes a point about how self regulatory organisations often result in there being 2 regulatory bodies.
It's difficult to point to some massive harm that has been done
The real harms are going to come for anyone with eyes to see.
I'd say they are already very well incentivized to be responsible actors
From what I've read, xAI has not been a responsible actor. And do we really want to leave it up to chance?
Zvi: The stupid dissent, espoused here by Coinbase CEO Brian Armstrong, is 'oh AI is like software so we don't even need an SRO, existing laws and incentives are fine.'
Alright, let's read what Zvi says now.
Zvi: There is definitely a 'don't say the thing' aspect of this, where he won't name what the downside risks actually are. When Demis says 'experts disagree' he is rather avoidant about the way in which they disagree here.
Clearly this is strategic, but if you don't already know, or are looking to not realize, it is very easy to come away thinking that Demis does mean the effect on jobs, even though when he says 'safely' he very much does not (primarily) mean that.
Wait, what? I didn't see anything in the post that meant that it was about jobs. I had existential risk in my mind when I was reading this but... Zvi is saying here that others think Demis is talking about ... jobs?
Zvi: Demis keeps it short, not offering many details. To the extent that he has laid out a proposal, it seems to be a good one. It is definitely an improvement on the margin. Thus I file this post and its ask, as high praise, under 'the least you could do.'
I agree with Peter Wildeford that while better than nothing FINRA is not a great model here, with heightened risk of regulatory capture, and not a substitute for full government action. You need an SEC to your FINRA. That doesn't mean don't make the FINRA. It does mean you still need the SEC.
ChatGPT: The key idea is: FINRA is an industry-run regulator. The SEC is the government regulator that oversees both the markets and FINRA itself.
Not being familiar with American organizations, this was not clear to me. I was under the assumption that FINRA is a government agency. It's not. It's an industry-regulator. Oh, so at some point, they could act in the industries self-interest, and thus lead to regulatory capture.
But if this is just an industry-regulator, who creates it?
ChatGPT: After the 1929 stock market crash and the Great Depression, Congress passed the Securities Exchange Act of 1934, creating the SEC as the federal securities regulator.
A few years later, Congress passed the Maloney Act of 1938, which amended the Securities Exchange Act.
Instead of making the SEC regulate every broker directly, Congress allowed the creation of national securities associations—industry-run organizations that would regulate their own members, subject to SEC oversight.
Using that authority, the brokerage industry formed the National Association of Securities Dealers (NASD).
NASD officially registered with the SEC in 1939 as the first national securities association.
So we need the equivalent of a "a stock market crash" to happen for the Government to wake up? Everything would be so much simpler if risk from AGI wasn't existential and we get many tries, but we may not.
Zvi: Demis Hassabis is to be applauded for his statement, but calls on others to do things, without any consequences following, risks becoming only cheap talk. My hope is that it can become more than cheap talk, and move the conversation forward. It won't be easy, but we have an opportunity. Let's make the most of it.
The second half of his post mentions talks about how Google Deepmind's principles have been violated since their acquisition by Google. They recently signed a deal with the Dept. of War to allow them to use their models for whatever the government wants. Even autonomous weapons.
I do think Demis Hassabis is blameworthy here. He continues to insist that DeepMind's principles are unchanged and unviolated. This is not the case.
Anyway, TLDR about Demis's framework. Better than nothing. Someone needs to do something sooner rather than later.
A prediction market on this would be — "Will a FINRA like entity be up and running for AI regulation by the end of 2027"
These committees take years to get created.
WHERE is the urgency?