If you opened a newspaper last week you saw the number. A relatively senior Anthropic researcher said publicly that he puts about a 10% chance on AI truly doing away with humanity. He also implied it was a fairly common feeling inside Anthropic, and several other researchers backed him up. Over the weekend Dario Amodei published one of his essays. He didn’t double down on 10%, but he did say it’s time to talk very seriously about slowing down.
Then the politics arrived. The administration says it’s a hoax. Bernie Sanders wants a full ban. Sam Altman tossed his hat in. There’s an election in a few weeks and trillions of dollars locked up in the market, which explains why the news cycle can’t put it down.
I have a cynical read available. Only two labs really matter here, three if you squint and add Google, four if you squint harder and add Grok. If you frighten everyone into a regulatory structure only the biggest models can satisfy, only the biggest models get to compete. It casts deep shade over open weights models, particularly the ones coming out of China, which are a huge cost advantage for everyone who isn’t a frontier lab. Given how fragile those business models look under competition, the incentive is right there. I don’t think Amodei is personally built that way. The shape of it is still worth naming.
What They’re Actually Worried About
The piece of technology under all of this is recursive self-improvement. Right now models improve because researchers work the levers. Enormous amounts of training, experimentation, and human judgment about which direction to push. RSI is what you get when the model starts working those levers itself.
Two things about that give people pause. The first is method. If the model is given an incentive to improve itself and picks a direction the researchers never considered, with no malicious intent at all, it can go somewhere humans can’t follow. Once you can’t understand the model, you can’t guide it.
The second is pace, and this is the part I think leaders are skipping.
Potential Energy To Kinetic
There was a good article back in 2015 about what recursive self-improvement would actually mean, and the thing it got right was the slope. With the amount of compute sitting out there, hitting RSI is a tipping event. The rock has been sitting at the top of the mountain. Once it goes over, improvement isn’t linear. It’s the other kind of curve.
That means a model doesn’t go from Astra 6 to Astra 7. It goes to Astra 20, and then from Astra 20 to Astra 2000, in a span measured in hours or days rather than release cycles. Which is why the 2015 piece framed it as a prisoner’s dilemma. Whoever reaches that point first doesn’t get an enormous lead. They win the game and it’s over for everyone else.
That’s also why the slowdown conversation probably won’t work. Every lab knows the shape of this. They will give the national conversation lip service and keep running as fast as they can, because if somebody else gets there first they’re finished. Human nature and incentive are pointed the same direction.
Why This Lands On Your Desk
Most AI investment decisions I see are built on an assumption of gradual competitive change. Vendors improve a bit, you improve a bit, market share moves slowly, you get time to react. If the slope is real, that assumption is structurally wrong, and the question stops being which vendor is best this quarter.
There’s a nearer-term version of the same problem that has nothing to do with the singularity. It is very hard for an organization, even one that’s already AI augmented, to build against a constantly changing model. Your team develops a solution around Opus V. Then one day, under the hood, it’s actually Opus 17 and it behaves in totally different ways. From a pure commercial perspective you need some control over what those models look like, which is one reason a slower, more versioned world is not automatically bad for you.
And if someone does put the kibosh on all of it tomorrow, I’m not discouraged. I’ve been saying for four years that you could cut this off where it is and it would still change the world. We’re now four years in. You can change plenty with Astra 6.
If you want the more grounded version of this, apply the same idea one level down. Self-improvement of your own processes. Our team runs diagnostics with companies, and learning from how we run those diagnostics so we ask better questions faster is the same loop, just analog. Most organizations are barely out of the chatbot era and we already sprung agents on them. The next thing is an entity you point at your business and tell to go improve it.
Book a conversation if you’re trying to make investment decisions against this and the gradual-change assumption is baked into your model.
