AI-augmented threat modelling
'How do we protect the world against bioterrorism? How do we protect the world against mirror life? Probably we're gonna need some kind of AI monitoring system that monitors for all of these things. But then we need to build this in a way that preserves civil liberties and our constitutional rights. So I think just as anything else, it's a new security landscape with a new set of tools and a new set of vulnerabilities. [...] My worry is just that this is happening all so fast. So maybe we need to do our thinking faster about how to make these governance mechanisms work.'[1]
Dario Amodei, CEO of Anthropic
Threat modelling is a process to estimate the size of key national security risks from AI capabilities (e.g. AI-enabled pandemics,[2] cyber worms,[3] election interference, etc)
A threat model is a structured representation of who might attack a system, how they could do so, and what damage would result, originally developed in cybersecurity to identify vulnerabilities in software and networks. In the context of AI national security risk assessment, the term has been adapted to denote a quantitative scenario linking a specific actor, attack vector, and harm pathway (e.g., a lone actor using an LLM to engineer a pandemic pathogen), typically parameterised with probability estimates and damage ranges to inform policy thresholds.
Though there is no one definitive, clear-cut methodology as of yet and the threat modelling methodology is still in development iterating, and though there is there might not be a one-size-fits-all approach across different domains, a set of distinct steps that generalize across different domains (e.g. horizon scanning, building model structure, BOTECs, expert surveys, etc).
AI might soon augment or even automate these steps since, generally, AI might soon be able to do anything a human can do in front of a computer. Threat modelling, as a special case of knowledge work, might also transform soon and researchers working on threat modelling might want to prepare for this.
In 2026, new models releases have reignited the debate on knowledge work transforming (e.g. in mathematics and science[4], economics,[5] software engineering,[6] etc.).
More specifically relevant to a potential transformation of threat modelling work are forecasts suggesting that, by the end of 2026:
- LLMs will be as good at forecasting as superforecasters on ForecastBench in Dec '27 (95% CI: Jan' 26 – Nov '27),[7] the Metaculus community in Apr '26, and pro forecasters in Jun '27.[8]
- AI can now complete software tasks with a time horizon of 14 hours, 50% of the time.[9] There's a 7.6% (3-20%) chance METR[10] will assess a model's 50% time horizon at 100 hours.[11]
When LLMs can replace expert or superforecaster surveys as input to the models and complete tasks that would take an analyst 14 hours, then it might soon be possible to automate or augment much or all of the threat modelling. If and when that happens, two major crucial considerations will be threat model complexity and credibility.
Complexity: AI progress might increase the complexity of threat modelling by increasing the quantity of threat models i.e. make (many) more threat models at the current state of the art (There is a continuum from significantly speeding up a human threat modeller to complete e2e automation of threat modelling). AI progress might also increase the quality (or sophistication) of threat modelling, which might also make them more opaque.
Notes
[1] “We might live in an offense-dominant world where one person or one AI model is smart enough to do something that causes damage for everything else. In the short run, we have a limited number of players now. So we can start within the limited number of players. We need to put in place the safeguards. We need to make sure everyone does the right alignment work. We need to make sure everyone has bioclassifiers. Those are the immediate things we need to do.
I agree that that doesn’t solve the problem in the long run, particularly if the ability of AI models to make other AI models proliferates, then the whole thing can become harder to solve. I think in the long run we need some architecture of governance. We need some architecture of governance that preserves human freedom, but also allows us to govern a very large number of human systems, AI systems, hybrid human-AI companies or economic units.
So we’re gonna need to think about: how do we protect the world against bioterrorism? How do we protect the world against mirror life? Probably we’re gonna need some kind of AI monitoring system that monitors for all of these things. But then we need to build this in a way that preserves civil liberties and our constitutional rights. So I think just as anything else, it’s a new security landscape with a new set of tools and a new set of vulnerabilities.
My worry is, if we had 100 years for this to happen all very slowly, we’d get used to it. We’ve gotten used to the presence of explosives in society or the presence of various new weapons or the presence of video cameras. We would get used to it over 100 years and we’d develop governance mechanisms. We’d make our mistakes. My worry is just that this is happening all so fast. So maybe we need to do our thinking faster about how to make these governance mechanisms work.” Dario Amodei — "We are near the end of the exponential"