Jacob Coxon’s resignation from Anthropic should not be dismissed as another theatrical episode in Silicon Valley’s long romance with apocalypse. A former researcher at both OpenAI and Anthropic, Coxon argues that the laboratories building increasingly autonomous systems are engaged in a competitive race whose commercial incentives may overwhelm their safety commitments. His most arresting claim, that artificial intelligence could destroy humanity before 2030, is impossible to verify. The institutional problem beneath it is not.
Modern corporations understand risks that arrive wearing numbers. Credit losses, cyber incidents and capital shortfalls can be modelled, priced and assigned tolerances. Existential AI risk resists this comfort. There is no actuarial history of machines escaping human control because humanity receives only one opportunity to discover whether the theory is correct. By the time reliable loss data exist, governance may have failed irreversibly.
This is where Frank Knight’s distinction between risk and uncertainty becomes essential. Risk concerns outcomes whose probabilities can be calculated; uncertainty concerns futures for which meaningful probabilities may not yet exist. Boards often treat uncertainty as if it were merely unquantified risk, then use the absence of reliable measurement as permission to continue. That is not evidence-based governance. It is institutionalised optimism.
Coxon’s probability may sound sensational, but concern is not confined to defectors or professional pessimists. A survey of 2,778 published AI researchers found that between 38 and 51 percent assigned at least a 10 percent probability to advanced AI producing outcomes as severe as human extinction. Even among respondents who expected broadly beneficial outcomes, almost half assigned at least a five percent probability to catastrophe.
No responsible board would accept a five percent probability that a new factory, aircraft or medicine might cause irreversible global harm. AI enjoys a different moral arithmetic because its benefits are immediate, its dangers remain abstract and those capturing the upside are not necessarily those carrying the downside. This is a classic externality, except the externality may include democratic stability, critical infrastructure and human agency itself.
The balanced position is neither panic nor technological surrender. Present systems have not demonstrated sustained recursive self-improvement or autonomous world domination. Critics correctly warn that speculative extinction narratives can distract from harms already visible: discrimination, surveillance, fraud, labour displacement and concentrated corporate power. Yet uncertainty cuts both ways. Lack of proof that catastrophe will occur is not proof that powerful autonomous systems will remain controllable.
The latest international scientific assessment identifies growing capabilities in coding, scientific reasoning and autonomous operation, alongside persistent failures in reliability, monitoring and control. It also records instances where developers could not exclude assistance with biological-weapons development and therefore introduced stronger safeguards. The signal is clear: capability is advancing faster than institutional confidence in containment.
For business leaders, responsible AI governance must move beyond ethics committees producing elegant principles. Boards should establish non-negotiable capability thresholds, independent pre-deployment testing, named executive accountability, protected internal challenge and credible shutdown procedures. The European Union’s framework for general-purpose models points towards this discipline through systemic-risk assessments, incident reporting and documented acceptance criteria.
Three questions should therefore enter every boardroom. Which capability would cause us to stop deployment, who possesses the authority to stop it, and would that person survive commercially for doing so? If the answers are unclear, the organisation does not possess AI governance. It possesses AI ambition supervised by hope.
Coxon may be wrong about 2030. Governance exists precisely because being wrong about the date does not make the underlying gamble intelligent. The deepest danger is not that machines suddenly become malicious. It is that institutions remain profitable, competitive and procedurally compliant while surrendering control one rational decision at a time.
- business a.m. commits to publishing a diversity of views, opinions and comments. It, therefore, welcomes your reaction to this and any of our articles via email: comment@businessamlive.com
Michael Irene, CIPM, CIPP(E) certification, is a data and information governance practitioner based in London, United Kingdom. He is also a Fellow of Higher Education Academy, UK, and can be reached via moshoke@yahoo.com; twitter: @moshoke
