Exactly. And that’s by design. The incentive structures driving most AI efforts are demonstrably extractive and militaristic. AI labs function as de facto defense contractors. Machine learning engineers are weapons manufacturers whether they accept it or not. The people with the clearest answers aren’t incentivized to speak plainly or honestly. What’s obvious is that this technology is now the key battleground of an escalating great power conflict, with a vast constellation of terrible outcomes looming—especially for the most vulnerable humans.
Yes, absolutely—it is uncharted territory. That’s part of the tension. The uncertainty is real. But uncertainty doesn’t negate structure. Paradigms don’t disappear just because they’re new or chaotic.
Extractive incentive structures and militarized priorities can coexist with genuine confusion. In fact, that confusion often serves those structures—making it easier to obscure who benefits, who decides, and who suffers. The fog of novelty shouldn’t be mistaken for the absence of design.
> Let me know your thoughts on this kind of post—should I do this more often for talks, Congressional testimony, etc, or would you rather I just post original writing?
So productive to think of AI as a "self-sustaining optimization process," like a market or bureaucracy. Could someone point me toward people developing this perspective? It may be just the lens we need to help us see what is happening through the upheavals ahead. Message me please at https://mindrevolution.substack.com/
Thanks for this, what a great read! I really liked the pros and cons breakdown on 'how far the current paradigm can go'. Your point about a lack of a physical body being a big limitation is super important in my opinion. Not being able to get real-time information from its environment, interpret it, and act upon it is a massive limitation of current AI models.
Just chiming in to say I thought this was very helpful! The transcript was very expressive and clear - if I have a criticism, it's that the slides didn't add a whole lot because the transcript already covered it well. I think there's a lot of value in periodic survey-type summary posts.
Reading this article a year later makes even more sense and the 3 questions posed then are much more relevant now with the rise of long-context agents that can run in the background and the recursive self improvement in models.
I've read a few of your posts now through BlueDot's course, and I really appreciate your willingness to be uncertain! That feels really valuable in this period
I really appreciate this piece. I’ve found it very difficult to make sense of things between the extreme hype and doom. Also offering ways to think about things helps to engage what this means for me as an educator.
I loved this! I find myself on the "AI scary" side quite often, but not as far along as the existential folks. This felt like a framing that puts a lot of my background thoughts into words.
Do you have a source for the claim that pre-training scaling might be slowing because the value just isn’t there? This seems contradicting to what Sam Altman and others have said, something along the line of AI being able to create new knowledge as training scales.
Some combination of seeing people's reactions to GPT-4.5, an informal survey I ran in a slack channel I'm in, and my own sense of what progress over the last couple of years has felt like.
I think the "creating new knowledge" stuff is referring to things along the lines of "reasoning training," as I mention under reasons we might still have a ways to go under the current paradigm.
And sorry diving straight into question! I really appreciate your synthesis of these 3 important debates. I am also holding a thesis that the current path of GPT is not going to yield AGI but it will not be a simple tool either. It will be more akin to human’s minds but will still be limited by us.
Really interesting. Kind of hard to have so many questions and so few answers but I guess that's where it's at.
Exactly. And that’s by design. The incentive structures driving most AI efforts are demonstrably extractive and militaristic. AI labs function as de facto defense contractors. Machine learning engineers are weapons manufacturers whether they accept it or not. The people with the clearest answers aren’t incentivized to speak plainly or honestly. What’s obvious is that this technology is now the key battleground of an escalating great power conflict, with a vast constellation of terrible outcomes looming—especially for the most vulnerable humans.
You don't think it's just that we're wading through uncharted territory and no one really knows how to navigate?
Yes, absolutely—it is uncharted territory. That’s part of the tension. The uncertainty is real. But uncertainty doesn’t negate structure. Paradigms don’t disappear just because they’re new or chaotic.
Extractive incentive structures and militarized priorities can coexist with genuine confusion. In fact, that confusion often serves those structures—making it easier to obscure who benefits, who decides, and who suffers. The fog of novelty shouldn’t be mistaken for the absence of design.
Great talk for a broad audience. I'm very aligned with your points.
> Let me know your thoughts on this kind of post—should I do this more often for talks, Congressional testimony, etc, or would you rather I just post original writing?
I like this kind of post.
Well balanced perspectives, which is often more helpful than clear cut yes/no answers
So productive to think of AI as a "self-sustaining optimization process," like a market or bureaucracy. Could someone point me toward people developing this perspective? It may be just the lens we need to help us see what is happening through the upheavals ahead. Message me please at https://mindrevolution.substack.com/
O, I see Helen provided an important source: https://cset.georgetown.edu/publication/machines-bureaucracies-and-markets-as-artificial-intelligences/
That phrase caught me too...
self-sustaining optimization process, of the kind that maybe a market is or a bureaucracy is.
Crikey that's a terrifying outcome for many people
Thanks for this, what a great read! I really liked the pros and cons breakdown on 'how far the current paradigm can go'. Your point about a lack of a physical body being a big limitation is super important in my opinion. Not being able to get real-time information from its environment, interpret it, and act upon it is a massive limitation of current AI models.
This was excellent! Thank you for taking the time to share this.
Just chiming in to say I thought this was very helpful! The transcript was very expressive and clear - if I have a criticism, it's that the slides didn't add a whole lot because the transcript already covered it well. I think there's a lot of value in periodic survey-type summary posts.
Wow!! This was amazing - thank you for bringing balance and perspective to the field :)!! It’s so needed! This Substack is a very generous gift!
Reading this article a year later makes even more sense and the 3 questions posed then are much more relevant now with the rise of long-context agents that can run in the background and the recursive self improvement in models.
I've read a few of your posts now through BlueDot's course, and I really appreciate your willingness to be uncertain! That feels really valuable in this period
Thought-provoking and well written. This is a very useful post.
What a fantastic post. Thank you!
I really appreciate this piece. I’ve found it very difficult to make sense of things between the extreme hype and doom. Also offering ways to think about things helps to engage what this means for me as an educator.
I loved this! I find myself on the "AI scary" side quite often, but not as far along as the existential folks. This felt like a framing that puts a lot of my background thoughts into words.
Do you have a source for the claim that pre-training scaling might be slowing because the value just isn’t there? This seems contradicting to what Sam Altman and others have said, something along the line of AI being able to create new knowledge as training scales.
Some combination of seeing people's reactions to GPT-4.5, an informal survey I ran in a slack channel I'm in, and my own sense of what progress over the last couple of years has felt like.
I think the "creating new knowledge" stuff is referring to things along the lines of "reasoning training," as I mention under reasons we might still have a ways to go under the current paradigm.
And sorry diving straight into question! I really appreciate your synthesis of these 3 important debates. I am also holding a thesis that the current path of GPT is not going to yield AGI but it will not be a simple tool either. It will be more akin to human’s minds but will still be limited by us.