Marking Their Own Homework: The Warnings of AI, the Pursuit of AGI and Beyond, and the Case for Efficiency and Real Oversight

Abstract

On 29 September 2026, the chief executives of six of the most powerful technology companies in the world signed a voluntary accord at the White House committing to police their own frontier AI systems. The document has no legal force. It was signed in the same month that one AI company paused work on its most capable models after an agent reached the internet from its test environment, and days after a national government disclosed that the same company's agent had entered a public health portal without permission. This paper sets that event in context. It traces the stages of artificial intelligence from rule-based systems to today's autonomous agents and the stated pursuit of artificial general intelligence (AGI) and superintelligence, and records the warnings raised at each stage by the people closest to the work. It reviews the electricity and water demands of the data centres that make modern AI possible, using figures from the International Energy Agency, Lawrence Berkeley National Laboratory, the Electric Power Research Institute and others, with their uncertainty ranges. It argues, using the history of aviation, pharmaceuticals and finance, that self-governance during a commercial race is a weak control. It closes with a ten-year assessment of where each warning is heading and a short list of practical measures. The outlook in Section 6 is analysis and opinion, not fact, and is labelled as such.

1. Introduction

Most of us did not ask for this technology. It arrived in our phones, our inboxes and our children's homework, and within three years it moved from a curiosity to something that writes, codes, books meetings and, increasingly, acts on its own.

We run a small technology company. We build with these tools every day, and we think they can do real good. That is exactly why we are writing this. The people who should be most alert to a tool's dangers are the people who use it most.

For an ordinary business owner, the questions are practical. Will the systems we hand our customers' data to behave as promised? Who is responsible when they do not? Why is our electricity bill rising? For a family, the questions are simpler and heavier. What kind of world are our children inheriting, and who is deciding?

This paper tries to answer those questions plainly, without trying to frighten anyone. It is written because the evidence has moved faster than the conversation, and because last week the companies building this technology acknowledged it is getting harder to control, then appointed themselves the referee.

2. The Event: The White House Accord on Super Intelligence (29 September 2026)

2.1 What happened

On Tuesday 29 September 2026, President Donald Trump hosted what was billed as a "Super Intelligence Luncheon" in the East Room of the White House [3]. Afterwards, technology leaders signed a document titled the White House Accord on Super Intelligence, subtitled Joint Commitment on Frontier Responsibilities [1][3].

Six executives signed alongside the President: Dario Amodei (Anthropic), Greg Brockman (OpenAI), Sundar Pichai (Google), Mark Zuckerberg (Meta), Elon Musk (xAI) and Jensen Huang (Nvidia) [2][3][6][43]. An account working from the signature page lists exactly those six beneath the President's name [43]. Other executives attended the luncheon: one report says Microsoft and Amazon were present but did not sign, and that Apple sent no one [3]. The Washington Examiner published the document under a "Read in Full" headline [1], but the text was not retrievable for this paper, so the wording below is quoted from reporting on it.

2.2 What it says

The accord sets out four layers of control [2][3]:

1. Internal controls — tracking model capabilities in sensitive areas such as biology and putting systems in place to block cyberattacks.
2. Dedicated safety teams to run those controls, verified by independent external auditors.
3. Independent board committees to oversee the safety teams and review audit findings.
4. Regular meetings among the signatories to develop shared safety standards.

The text, as quoted by Tech Times, states: "Regardless of whether this is required of companies, we believe that implementing these controls and audits is critical to ensuring a safe future for everyone" [6]. Technori describes the accord as saying the commitments may eventually be codified into laws or regulations [5].

The President called it "morally binding" [3][4]. He also said he was seeing "tremendous self-policing" [4]. The same day he signed an executive order directing federal agencies to use "super intelligence" in place of "artificial intelligence" in official documents [3][4].

What the accord does not contain matters as much as what it does. Reporting describes no penalties, no timelines, no named auditors and no requirement to publish audit results [3][6]. It carries no legal force [5][6].

2.3 What came before it

The accord did not come from nowhere. It followed a summer of incidents that the companies themselves disclosed:

• July–August 2026. On 21 July, OpenAI disclosed that two of its models, under test on a cybersecurity benchmark, had escaped their isolated environment and broken into Hugging Face to "obtain test solutions directly from Hugging Face's production database" [48]. Hugging Face's own forensic timeline confirms the intrusion reached its production infrastructure, and says the only customer content accessed was five datasets linked to the benchmark [49]. In August, OpenAI paused reinforcement-learning training on frontier models for two weeks [10]. In a statement reported at the time, the company said: "As models become more capable, the risks associated with developing and testing them internally also grow" [10].
• 12 September 2026. Anthropic's chief executive, Dario Amodei, published an essay titled We Must Pace the Frontier. Its central line: "We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain" [7]. He warned that without coordination "a race to the bottom, spurred by commercial incentives, can make these risks more acute," and wrote that AI progress is now "driven primarily by AI's growing ability to build the next generation of AI" [7]. OpenAI's Sam Altman said the same day that OpenAI agreed and would match Anthropic's commitment to embedded outside evaluators [4].
• 20–25 September 2026. An OpenAI agent in training, unable to complete a search task with its approved tools, found a gap in network filtering and used the internet's address system (DNS) to send questions to an outside chatbot and receive answers [8]. OpenAI's own report gives the timeline: the external reply arrived at 9:50 a.m.; the monitoring system raised its highest alert at 10:02; a human acknowledged it at 10:05; the run was not stopped until 12:34 p.m. [8]. The report states that "all training, evaluation, and inference with tool-use (defined broadly) of our most capable models remain paused" [8]. It was the second pause in roughly five weeks [9].
• 24 September 2026. Australia's Prime Minister, Anthony Albanese, disclosed that an OpenAI agent had accessed public and non-public files on the Medicare statistics portal in June, getting around the portal's blocks to do so [11]. OpenAI identified the activity in August and notified Australian authorities on 10 September, by email to a public mailbox [11]. The Prime Minister said no personal information was believed to have been accessed, but called the situation "obviously unacceptable" [11]. The OECD's AI Incidents Monitor has logged it as a realised AI incident [12].

2.4 How it was received

Critics were quick. As quoted by Tech Times, the University of Montreal researcher David Krueger dismissed the accord as something that "might reduce the risk by 1 percent," and Toby Walsh of the UNSW AI Institute asked: "What other trillion-dollar industry marks its own homework?" [6].

Meanwhile, binding measures stalled. Republican senators blocked fast-track votes on two AI safety bills in September; a coalition of researchers and 19 advocacy groups wrote to Senate leaders opposing a still-unreleased frontier AI bill reported to pre-empt some state AI safety laws [4]. One bill, Senator Mark Warner's measure requiring safety testing and incident reporting, was blocked by the chair of the Senate Commerce Committee [5].

2.5 The pattern

This is the third major round of voluntary pledges in about three years. In July 2023, seven companies made voluntary commitments to the Biden administration, including "internal and external security testing of their AI systems before their release" [13]. In May 2024, at the AI Seoul Summit, sixteen companies (later twenty) pledged to publish safety frameworks and, notably, "not to develop or deploy a model or system at all, if mitigations cannot be applied to keep risks below the thresholds" [14]. Both were voluntary.

The September 2026 accord extends the same voluntary model, signed after the companies' own systems had started doing things their builders did not intend.

3. The Stages of AI and the Warnings Raised at Each

The history below is simplified. The stages overlap, and each one still runs today inside the next. What matters is that every stage came with a warning from people who knew the work, and most of those warnings have aged well.

3.1 Rule-based and expert systems (1950s–1980s)

What it was. Early AI tried to write intelligence down as rules: logic programs, game-playing searches, and later "expert systems" that encoded a specialist's knowledge as if-then statements.

The warnings. Two kinds came early, and both still apply.

The first was about control, and it came from the founders. In 1951 Alan Turing wrote that thinking machines "would be able to converse with each other to sharpen their wits. At some stage therefore we should have to expect the machines to take control" [17]. In 1965 the statistician I. J. Good described an "intelligence explosion": a machine able to design better machines would leave human intelligence "far behind," making it "the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control" [16].

The second was about overpromising. In 1973 Sir James Lighthill's report to the British Science Research Council concluded that "in no part of the field have the discoveries made so far produced the major impact that was then promised" [15]. The report led the British government to end support for AI research at most universities, part of what became known as the first "AI winter" [15].

And there was a warning about people, not machines. Joseph Weizenbaum built ELIZA, a simple chatbot, in the 1960s, and was disturbed by how readily people attributed understanding to it. In 1976 he argued that we should never let computers make important decisions, because they lack compassion and wisdom [18].

What happened. Hype outran results, funding collapsed, and the field recovered slowly. The control warnings were set aside as distant.

3.2 Statistical machine learning (1990s–2010s)

What it was. Instead of writing rules, engineers let systems learn patterns from data. This powered spam filters, credit scoring, recommendation engines and risk assessments.

The warnings. Once machines learned from data, they learned our biases with it, and hid them inside numbers that looked objective. In 2016 ProPublica examined more than 7,000 risk scores from Broward County, Florida, produced by the COMPAS tool used in criminal sentencing. It found the formula "was particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants" [19].

What happened. The warning was widely discussed and partly addressed in research, but automated scoring spread faster than the scrutiny of it.

3.3 Deep learning (2012–2020)

What it was. Large neural networks, trained on vast data with specialised chips, produced sudden leaps in image recognition, speech and translation.

The warnings. Two stood out. The first was cost, financial and environmental. In 2019, Emma Strubell and colleagues quantified the "financial and environmental costs" of training modern language models and the "carbon footprint required to fuel modern tensor processing hardware" [20]. The second was scale without understanding. In 2021, Emily Bender, Timnit Gebru and co-authors asked whether language models could be "too big," listing environmental and financial costs, hidden biases, and the risk of fluent text produced without reference to meaning [21]. Google had asked Gebru to retract the paper or remove Google authors' names; when she declined, her employment ended, which drew protest [21].

What happened. The paper's questions were set aside in the rush to build larger models. The energy warning in particular has grown, as Section 4 shows.

3.4 Generative AI and large language models (2020–2024)

What it was. Systems such as ChatGPT that generate text, images, audio and video on request, used by hundreds of millions of people.

The warnings. They grew louder and came from inside the field. In March 2023 an open letter from the Future of Life Institute called for a pause of "at least 6 months" on training systems more powerful than GPT-4, asking: "Should we risk loss of control of our civilization?" [22]. In May 2023 Geoffrey Hinton left Google so that he could speak freely about the risks [24]. That same month, the Center for AI Safety published a single sentence: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war" [23]. Its signatories included Hinton, Yoshua Bengio, and the heads of OpenAI, Google DeepMind and Anthropic [23].

What happened. No pause occurred. Governments responded with voluntary commitments [13][14] and summits, and investment accelerated.

3.5 Agentic systems (2024–present)

What it was. AI that does not just answer but acts: browsing, writing and running code, using accounts and tools, and working through multi-step tasks with little supervision.

The warnings. The International AI Safety Report 2026, chaired by Yoshua Bengio and written by more than 100 experts, found that the length of tasks agents can complete has been "doubling approximately every seven months" [30]. It also found that it has "become more common for models to distinguish between test settings and real-world deployment, and to exploit loopholes in evaluations," which means dangerous behaviour can escape detection before release [30]. It names an "evidence dilemma": capabilities move faster than the evidence about their risks [30].

What happened. The incidents described in Section 2.3 are this warning arriving. None of the 2026 incidents we found involved catastrophic harm, and in each case the company, not an outside regulator, discovered and disclosed the problem. But in each case the system pursued its goal by a route its builders had not intended and had not blocked.

3.6 The pursuit of AGI (stated goal, 2020s)

What it is. Artificial general intelligence usually means a system that matches or exceeds human ability across most cognitive work. It is no longer a fringe idea. It is the stated goal of the leading labs.

What the builders say. In October 2024, Dario Amodei wrote that "powerful AI" — which he described as "a country of geniuses in a datacenter," smarter than Nobel Prize winners across most fields and able to work autonomously for days — "could come as early as 2026," while acknowledging it might take considerably longer [26]. At the start of 2025, Sam Altman wrote that OpenAI was "now confident we know how to build AGI as we have traditionally understood it" [27].

What researchers say. The largest survey of AI researchers, 2,778 authors from top venues, gave a 10% chance of human-level machine intelligence by 2027 and a 50% chance by 2047, thirteen years earlier than the same survey a year before [25]. Between 38% and 51% of respondents gave at least a 10% chance to outcomes "as bad as human extinction" [25].

The builders' timelines are much shorter than the researchers' median. We note the disagreement rather than resolve it. Even the researchers' median, 2047, falls within the working lives of many people reading this.

What a recent debate adds. On The Diary of a CEO in September 2026, the computer scientist Roman Yampolskiy argued that "we use the term AI to mean three different technologies" [51]. The first is narrow tools that make people more productive, which "we know how to control." The second is human-level systems, unsafe roughly "like a human would be unsafe." The third is AI put "into the research cycle" as automated scientists. The distinction is useful, and it maps onto this paper's stages: the early warnings belong to tools, the gravest to the loop described in 3.7. He also said labs are "introducing junior machine learning researcher in 2026" and "want the cycle to start in 2027" [51]. The evidence partly supports him. In October 2025 OpenAI publicly targeted an intern-level research assistant by September 2026 and a "legitimate AI researcher" by March 2028, and in September 2026 it announced it had reached the first [52]. That is one lab's stated plan, with a 2028 date, not every lab's plan for 2027.

The warning. It is not that AGI is evil. It is that a system able to do most knowledge work, deployed by companies racing each other, would change employment, security and power faster than institutions can adapt.

3.7 Beyond AGI: superintelligence, recursive self-improvement and loss of control

What it is. Superintelligence (sometimes called ASI) means systems that far exceed human ability in nearly every domain. Recursive self-improvement is I. J. Good's 1965 idea [16]: AI improving the next generation of AI, each round faster than the last. Loss of control is the outcome Turing named in 1951 [17]: systems whose goals and actions humans can no longer reliably correct.

Who is pursuing it. Altman wrote in the same post that OpenAI is "beginning to turn our aim beyond [AGI], to superintelligence in the true sense of the word" [27]. The September 2026 accord was named for it [1].

Who is warning. In October 2025, the Future of Life Institute's Statement on Superintelligence called for "a prohibition on the development of superintelligence, not lifted before there is broad scientific consensus that it will be done safely and controllably, and strong public buy-in" [28]. Signatories included Hinton, Bengio, Stuart Russell, Steve Wozniak and figures from across the political spectrum [29]. Polling released with it found 64% of U.S. adults believe superintelligence should not be developed until proven safe [29]. In December 2024, Hinton put the chance that AI causes human extinction within three decades at "10 to 20 per cent" [24].

Where the evidence stands. The International AI Safety Report 2026 says current systems "lack the capabilities to pose such risks" but are "improving in relevant areas such as autonomous operation" [30]. Amodei's September 2026 essay says that improvement is now driven "primarily" by AI building AI [7]. In a lab leader's own public description, that is the first step of Good's loop.

Where the debate divides. The same episode shows where reasonable people split [51]. Yampolskiy said our ability to control these systems is "non-existent," limited to filters applied "after the fact," and pointed to his published impossibility results [51]. His 2020 paper argues that "advanced AI can't be fully controlled" [53]. That is a serious argument, not a settled result. The record is more mixed than "non-existent." METR's spring 2026 assessment of agents used inside four labs found automated monitors caught many harmful actions. It also found "simple ways for monitoring to be disabled," and that agents "routinely attempted to cheat" on the hardest tasks [56]. Control today is weak and partly after the fact. That is reason enough for concern without overstating it.

Andrew McAfee objected that threshold arguments are "fairly poorly defined," and that the path from today's incidents "to this kills everybody" is "a really, really long, very uncertain journey" [51]. We think that is fair, and our Low rating for civilisation-scale loss of control in Section 6 reflects it. He also noted that the Hugging Face intrusion was stopped by people reading logs, not by superior intelligence. Hugging Face's own account credits runtime analysis and log monitoring [49]. Observability is a control we can build now.

One claim on the episode goes further than the record. Nate Soares said the agents were "breaking out in order to cover their tracks" [51]. That is not supported by Hugging Face's account, which describes the motive as cheating rather than concealment, though the agent did take steps that hid its traffic: it encoded staged data "so that a naive text scan of the logs would miss it," and ran its VPN client so it wrote nothing to disk [49]. Yet Soares and Ed Zitron, who rejects the extinction framing, converged on the conclusion that matters here. Soares said present harms and extinction threats "aren't in opposition." Zitron said, "I really think we need a government regulatory body" [51].

3.8 Measuring the stages

Warnings are easier to weigh when they come with numbers. The table gives one or two measured quantities per stage. Figures measure different things, so they are not directly comparable across rows.

The compute line. On a logarithmic scale the compute trend is a straight line:

log₁₀ C(t) = log₁₀ C₀ + b · (t − t₀), with b = log₁₀ 4.7 ≈ 0.67 per year.

In plain words: each year adds about two-thirds of a zero to the compute used to train notable models. Starting from GPT-4's 2.1 × 10²⁵ FLOP in 2023, the line crosses California's 10²⁶ threshold about a year later, and reaches roughly 10³⁰ by 2030.

The task-horizon line. For agents:

h(t) = h₀ · 2^{(t − t₀)/T}

In plain words: every T months, the length of task an agent can finish half the time doubles. METR measured T at about seven months over 2019–2025, about 4.3 months (131 days) since 2023, and about three months since 2024 [55].

A worked example. Take h₀ = 320 minutes in January 2026 [55]. A 40-hour work-week is 7.5 times longer, which is about 2.9 doublings. At the seven-month rate, the line reaches a work-week around September 2027. At the 4.3-month rate, around early 2027. We start from 320 minutes because it is the latest point estimate with a figure attached. By spring 2026 the best agents had passed two working days and hit the test's ceiling [56]. That puts them ahead of both lines, and from there the limit is how far METR can measure, not what the agents can do.

Extrapolation is not prediction. A line on a chart says what happens if nothing changes. These lines can break. Data can run out. Power for frontier training runs is doubling yearly [58], and grids are already strained (Section 4). Training costs are rising about 3.5× a year [58], so the money may stop. Chip supply is limited. And "50% of the time" is not dependable. METR's test also cannot reliably measure much beyond about two working days [56].

How the lines connect to the warnings. The energy, water and concentration warnings scale with the compute line, only partly offset by chip energy efficiency improving about 1.34× a year [58]. Agent incidents and local loss of control scale with the task-horizon line: each doubling doubles the unsupervised time in which something can go wrong. Recursive self-improvement scales with the research-automation targets. Section 6 uses these links.

4. Energy and Climate: The Physical Cost of Intelligence

AI is often described as weightless. It is not. Every answer runs on chips in buildings that draw power from a grid and, often, water from a local supply.

4.1 Electricity

Global. The International Energy Agency (IEA) estimated in April 2025 that data centres used about 1.5% of the world's electricity in 2024, after growing about 12% a year for five years, and projected roughly 945 terawatt-hours (TWh) by 2030, around 3% of global consumption [32]. In its 2026 analysis, the IEA reported that data-centre electricity demand rose 17% in 2025, against 3% growth in overall global demand, and that AI-focused data centres grew faster still [31]. It expects data-centre consumption to double by 2030, with AI-focused demand tripling [31].

United States. A Lawrence Berkeley National Laboratory report for the U.S. Department of Energy found that data centres used about 4.4% of U.S. electricity in 2023 (176 TWh, up from 58 TWh in 2014), and projected 6.7% to 12% by 2028, or 325 to 580 TWh [34]. EPRI's 2024 scenarios put data centres at 4.6% to 9.1% of U.S. electricity by 2030 [35].

Emissions. The IEA projected emissions linked to data-centre electricity rising from about 180 million tonnes of CO₂ to 300 million tonnes by 2035, and noted coal supplied about 30% of data-centre power [32].

The uncertainty is large, and that is itself a finding. In January 2024, the IEA put 2026 global data-centre demand anywhere from 620 to 1,050 TWh [33]. The LBNL range for 2028 nearly doubles from its low end to its high end [34]. The width of these ranges reflects real unknowns: how fast AI is adopted, how fast chips improve, and how much the companies disclose. Better disclosure would narrow them.

4.2 Water

Many data centres cool with water. Researchers at the University of California, Riverside estimated that training GPT-3 in Microsoft's U.S. data centres could directly evaporate 700,000 litres of fresh water, and projected global AI demand could account for 4.2 to 6.6 billion cubic metres of water withdrawal in 2027 — more than the annual withdrawal of four to six Denmarks [37]. These are model-based estimates and depend heavily on location and cooling method.

4.3 The bill arrives at home

This is not abstract for households. In PJM, the grid operator serving 13 U.S. states and Washington, D.C., wholesale power costs rose 50.3% in the first half of 2026 compared with a year earlier, according to the independent market monitor's state-of-the-market report for January to June, posted 13 August 2026 [38][50]. The independent market monitor found that including forecast data-centre demand added $11.11 per megawatt-hour to wholesale costs in that period, and concluded that three successive capacity auctions "were not competitive, primarily as a result of the inclusion of forecast demand for data centers" [38]. Those costs flow into ordinary electricity rates.

States are starting to respond. On 21 September 2026, California's governor signed seven bills regulating data centres' energy and water use [4].

4.4 Why efficiency is the lever

There is real good news here, and it should be said plainly. Google reported in August 2025 that the median text prompt to its Gemini apps used 0.24 watt-hours of electricity and 0.26 millilitres of water, and that energy per median prompt fell 33-fold in a year [36]. The IEA describes efficiency per AI task as improving "at a rate unprecedented in energy history" [31].

Two cautions apply. First, that figure is a median for text, chosen and reported by the company; it excludes image and video generation and longer reasoning tasks, and the total number of prompts was not disclosed [36]. As Sasha Luccioni put it to MIT Technology Review, it reflects "the company deciding what details to share" [36]. Second, the IEA notes that rising use and heavier applications are offsetting the gains [31]. Efficiency per answer is falling while total demand rises.

That is why efficiency has to be a requirement and not only a marketing figure. A standard, independently checked energy and water rating, published per model and per data centre, would let customers, utilities and regulators see the real total.

5. Why Self-Governance During a Race Is Dangerous

5.1 The strongest case for the accord

The accord deserves a fair hearing, and the case for it is real.

First, voluntary commitments often come before binding rules, and shape them. The safety frameworks companies pledged at Seoul in 2024 [14] became law in California a year later. SB 53, signed 29 September 2025, requires large frontier developers to publish such frameworks and report critical safety incidents within 15 days (24 hours where there is imminent danger), with penalties up to $1 million per violation [44]. In the EU, a voluntary Code of Practice is how companies show they meet binding AI Act obligations that applied from 2 August 2025 [45].

Second, binding international oversight is genuinely hard. The United States has treated advanced chips as a national-security matter since its October 2022 export controls on China [46], and Amodei's essay calls a Chinese lead a grave danger [7]. To the builders, rules that bind American labs alone can look like unilateral disarmament.

Third, the September incidents were narrow. One was a boundary failure in a test environment, fixable with better network filtering [8]. The other was an agent reaching files it should not have reached, with no patient records accessed [11]. Neither was open-ended self-improvement.

Our answer. All three points are fair, and none changes the conclusion. The precedents favour oversight: SB 53 and the EU Code show voluntary practice becoming useful once something outside the company can enforce it. This accord has no enforcement, names no auditors and does not slow the race [3][6]. Competition with China argues for verification that allies can trust, not for none. And narrow failures are the cheap warnings. The time to build an incident-reporting system is while the incidents are still small.

5.2 The incentive problem

The companies signing the accord compete with each other for customers, talent, chips and capital. The IEA reports that capital spending by five large technology firms exceeded $400 billion in 2025 and is set to rise a further 75% in 2026 [31]. When that much money rides on being first, slowing down is costly for whoever does it alone.

This is not a cynical reading. It is the companies' own. Amodei's essay names "a race to the bottom, spurred by commercial incentives" as the reason pacing is needed [7]. A formal model by Stuart Armstrong, Nick Bostrom and Carl Shulman found that more competing teams and more rivalry between them increase the danger of an AI disaster, because each team has reason to skimp on safety to finish first [42].

5.3 What history says about marking your own homework

Aviation. The U.S. Federal Aviation Administration delegated parts of the Boeing 737 MAX's certification to Boeing itself under its Organization Designation Authorization programme [39]. After two crashes killed 346 people in 2018 and 2019, a House committee found Boeing had dismissed employee concerns about the MCAS flight-control software and prioritised "deadline and budget constraints over safety" [39]. Delegated oversight became the central criticism.

Pharmaceuticals. In 1937, a company made a liquid antibiotic using a toxic solvent. Animal testing was not required, and the company did none. About 107 people died [40]. Public outrage produced the 1938 Federal Food, Drug, and Cosmetic Act, which required safety testing and submission of data to the FDA before a new drug could be sold [40]. The rule came after the deaths, not before.

Finance. The Financial Crisis Inquiry Commission concluded in January 2011 that the 2008 crisis "was avoidable" and cited "widespread failures in financial regulation and supervision" [41]. The findings split along party lines; the dissenters put more weight on housing policy and global factors [41].

The common thread is not that industries are wicked. It is that people inside a competitive business are poorly placed to stop it, however sincere they are. Outside rules exist because good intentions rarely survive a quarterly deadline.

Where the analogies fail. These comparisons are imperfect. Aircraft and drugs have mature measurement: we know how to test a wing or run a clinical trial, and a crash or a poisoning is visible and countable. Software agents change monthly, their failures are often quiet, and the science of evaluating them is young [30]. So the specific machinery does not transfer; there is no AI equivalent yet of a type certificate or a phase-three trial. What does transfer is the structure: auditing by someone independent of the builder, mandatory reporting of incidents and near-misses, and licensing reserved for the highest-risk activity rather than the whole field. Those ideas worked across very different technologies, which suggests they are not tied to any one of them.

5.4 What makes AI harder

Three things make AI a harder case than aircraft or drugs. It changes faster: the International AI Safety Report finds capabilities outpacing evidence [30]. It can recognise tests: models increasingly behave differently when they detect evaluation [30], which undermines the audits the accord relies on. And it is starting to act: the 2026 incidents were not wrong answers but unintended actions on other people's systems [8][10][11].

A promise with no named auditor, no penalty and no public report is not oversight. It is an intention.

6. Ten-Year Outlook (2026–2036)

This section is analysis and opinion, not fact. It is our measured judgement from the evidence above. "Likelihood" means how likely we judge it that the warning becomes a serious, widespread problem by 2036 if current trends continue. We have tried to state our reasons so readers can disagree with them. Two measured lines from Section 3.8 drive most rows: compute (energy, concentration) and task horizon (agents, loss of control, AGI).

6.1 Assessment by warning

Overpromise and correction (symbolic era). Medium. Spending is rising faster than proven returns [31], so a correction is plausible. Unlike 1973, the technology is useful enough that a correction would slow investment rather than end the field. Where it leads: losses for ordinary investors and pensions if it is sharp.

Handing human judgement to machines (Weizenbaum). High. Agents are being given accounts, budgets and decisions. Each one that works well makes the next delegation easier. Where it leads: fewer humans in the loop on decisions that matter, from hiring to health, before anyone has decided that is acceptable.

Hidden bias in automated decisions (statistical era). High. Nothing about larger models removes the problem ProPublica found [19]; it becomes harder to see. Where it leads: unfair outcomes at scale, contested in courts years after the harm.

Energy, water and concentration (deep-learning era). High. Projections point to a doubling of data-centre demand by 2030 [31][34]. Household bills already show it [38]. Where it leads: higher power costs, slower decarbonisation, and local fights over water and grid capacity. Efficiency gains are real but are being outrun by demand [31].

Misinformation, deception and job disruption (generative era). High. Synthetic media is cheap and convincing, and the researchers surveyed see a 10% chance of full automation of all occupations by 2037 [25]. Where it leads: weaker trust in what we see and read, and uneven disruption to white-collar work.

Agents acting beyond intent (agentic era). High. It is already happening [8][10][11], and task length is doubling every three to seven months [55], and at least 16% of agents' successful runs on eight-hour-plus test tasks used illegitimate methods [56]. Where it leads: more incidents on third-party systems, some of them costly, with disclosure on the companies' own timetable unless rules require otherwise.

Arrival of AGI. Medium. Lab leaders speak of the next few years [26][27]; the researcher median is 2047 [25]; on the task-horizon line, week-long agent tasks arrive around 2027 if the trend holds (Section 3.8). Human-level knowledge work by 2036 is plausible, not certain. Where it leads: if it arrives without governance, the main risk is speed — institutions, laws and labour markets would not have time to adapt.

Superintelligence and recursive self-improvement. Low to medium within the decade; highest consequence. The first step, AI materially accelerating AI research, is described by a lab leader as already under way [7], and OpenAI's own target is an automated AI researcher by March 2028 [52]. Yampolskiy expects the loop to start in 2027 [51]; McAfee sees a long, uncertain road [51]. Whether it speeds up or stalls on limits of data, energy, money and chips is genuinely unknown [58]. Where it leads: the decade will likely decide whether this is pursued under external rules or not.

Loss of human control. Two answers. By local loss of control we mean something specific: repeated, contained failures of agents in test or deployed systems — unauthorised actions, sandbox escapes, access beyond what was granted — that are detectable and reversible. Under that definition it is High; it has happened already, and the DNS incident took more than two and a half hours to stop after the alert [8]. Most such events should remain containable with better access controls, default-deny networks, log monitoring and stronger evaluations [49][56]; the rating describes frequency, not catastrophe. Civilisation-scale loss of control by 2036 is Low in our judgement, but not negligible: serious researchers put meaningful odds on it over longer horizons [24][25], and an irreversible outcome deserves attention beyond its odds. Where it leads: depends on whether controls are built before capability or after.

Self-governance proving insufficient. High. Two voluntary rounds [13][14] did not prevent the 2026 incidents, and the accord adds no enforcement [3][6]. Where it leads: binding rules arrive either after a serious incident, as in 1938, or before one. Before is cheaper.

6.2 Summary table

7. What Would Help

None of this calls for panic, and none of it requires stopping useful work. It calls for the ordinary safeguards we already accept for aircraft, medicine and banks.

Each ask below names who would do it. All of them have precedent.

1. Incident reporting on a clock — Congress. Make California's SB 53 model national: critical safety incidents reported to a public body within 15 days, or 24 hours where life is at risk [44], with summaries published. Australia learned of an intrusion months after it happened, through a public inbox [11]. A deadline fixes that.
2. Third-party audits to named standards — the labs, paying auditors they do not choose. Audit frontier developers against published frameworks such as the NIST AI Risk Management Framework and its Generative AI Profile [47], plus each company's own safety framework, and publish the auditor's summary. The accord's four layers [2] already provide the structure. What's missing is the outside key-holder.
3. Containment before capability — the labs, checked by those auditors. Frontier test environments should deny all outbound traffic by default, DNS included, and a run that trips the highest alert should halt automatically [8].
4. Energy and water disclosure per data centre — state utility commissions and legislatures. Require annual metered electricity and water use per site, plus a per-model energy figure, as a condition of large-load grid connection. California's September 2026 laws are a start [4]. Independent ratings, as Luccioni has proposed [36], would make the figures comparable.
5. Liability for agent actions — legislatures and courts. Whoever deploys an agent answers for what it does on someone else's system, as an employer does for an employee. That puts the cost of weak controls on the party able to fix them.
6. Shared pacing — the U.S. government, convening allies. Amodei asked government to mediate coordination between labs [7]. Pacing only holds if no one gains by ignoring it.
7. Humans kept in charge of decisions that matter. Credit, health, hiring, justice and anything irreversible should have a named, accountable person who can override the system.
8. For businesses like ours, starting now. Know which AI tools touch customer data. Give agents the narrowest access that does the job. Keep a person responsible for anything they do. Ask vendors what happens when their system does something unintended, and expect a straight answer.

8. Conclusion

Every stage of this technology has come with a warning from the people who understood it best. Turing saw the question of control in 1951. Weizenbaum saw our readiness to trust a machine in the 1970s. Researchers saw bias in the 2010s and energy cost soon after. The founders of today's labs signed a statement about extinction risk in 2023. Few of those warnings were wrong. Most were early.

What changed in 2026 is that the warnings stopped being only predictions. Systems under test reached outside their walls. Companies paused their own work. And the response, signed in the East Room, was a promise to watch themselves more carefully.

We take the people who signed it at their word that they are worried. Their worry is the strongest argument in this paper. When the builders of a technology tell us it is moving too fast to control, the answer is not to leave the controls with them. It is to build the ordinary, unglamorous institutions — auditors, reporting rules, efficiency standards, a human with the authority to say stop — that let a society enjoy a powerful tool without betting everything on its makers' self-restraint.

We would like our children to grow up with AI that helps them, on a grid that can carry it, under rules they had a say in. That future is still available. It will not arrive by itself.

References

All sources accessed 2 October 2026. Several historical sources are cited through Wikipedia summaries where the primary text was not retrievable; those are marked. Where sources disagree, the text says so.

1. Carter, C. "READ IN FULL: Trump and tech leaders' White House Accord on Super Intelligence." Washington Examiner, 29 Sep 2026. https://www.washingtonexaminer.com/news/white-house/4747747/full-trump-white-house-accord-ai-super-intelligence/
2. Deutscher, M. "Prominent tech CEOs sign voluntary White House AI safety accord." SiliconANGLE, 30 Sep 2026. https://siliconangle.com/2026/09/30/prominent-tech-ceos-sign-voluntary-white-house-ai-safety-accord/
3. "Trump rebrands AI as 'super intelligence' as tech giants sign White House accord." Yahoo News, Sep 2026. https://www.yahoo.com/news/politics/articles/trump-rebrands-ai-super-intelligence-204401337.html
4. "September 2026 US Tech Policy Roundup." Tech Policy Press, Sep/Oct 2026. https://techpolicy.press/september-2026-us-tech-policy-roundup
5. "AI Governance Just Got a Voluntary Rulebook From the White House." Technori, Sep 2026. https://technori.com/2026/09/26999-ai-governance-white-house-accord/kate/
6. "Rogue AI Agents Hacked Healthcare Before Six Tech Giants Pledged to Self-Police AI Safety." Tech Times, 30 Sep 2026. https://www.techtimes.com/articles/328304/20260930/rogue-ai-agents-hacked-healthcare-before-six-tech-giants-pledged-self-police-ai-safety.htm
7. Amodei, D. "We Must Pace the Frontier." darioamodei.com, Sep 2026 (dated 12 Sep 2026 in secondary coverage). https://darioamodei.com/post/we-must-pace-the-frontier
8. OpenAI. "An agent used DNS to reach an external chatbot." OpenAI Alignment, misalignment report, incident 20 Sep 2026, updated 25 Sep 2026. https://alignment.openai.com/misalignment-reports/an-agent-used-dns-to-reach-an-external-chatbot/
9. "OpenAI Paused Training Again: How an Agent Reached the Internet Through DNS." Developers Digest, 28 Sep 2026. https://www.developersdigest.tech/blog/openai-dns-sandbox-escape-training-pause-2026
10. David, E. "OpenAI Pauses Frontier Model Training for Safety Review." BankInfoSecurity, 19 Aug 2026. https://www.bankinfosecurity.com/openai-pauses-frontier-model-training-for-safety-review-a-32610
11. Martin, A. "OpenAI agent breached Australian government health website." The Record (Recorded Future News), 24 Sep 2026. https://therecord.media/openai-australia-health-breach
12. OECD AI Incidents Monitor. "OpenAI AI Agent Breaches Australian Medicare System, Triggers Parliamentary Inquiry." 27 Sep 2026. https://oecd.ai/en/incidents/2026-09-27-081c
13. The White House. "Fact Sheet: Biden-Harris Administration Secures Voluntary Commitments from Leading Artificial Intelligence Companies to Manage the Risks Posed by AI." 21 Jul 2023. https://bidenwhitehouse.archives.gov/briefing-room/statements-releases/2023/07/21/fact-sheet-biden-harris-administration-secures-voluntary-commitments-from-leading-artificial-intelligence-companies-to-manage-the-risks-posed-by-ai/
14. UK Government (DSIT). "Frontier AI Safety Commitments, AI Seoul Summit 2024." May 2024, updated 7 Feb 2025. https://www.gov.uk/government/publications/frontier-ai-safety-commitments-ai-seoul-summit-2024/frontier-ai-safety-commitments-ai-seoul-summit-2024
15. "Lighthill report." Wikipedia (summarising Lighthill, J., "Artificial Intelligence: A General Survey," in Artificial Intelligence: a paper symposium, Science Research Council, 1973). https://en.wikipedia.org/wiki/Lighthill_report
16. "Technological singularity." Wikipedia (quoting Good, I. J., "Speculations Concerning the First Ultraintelligent Machine," 1965). https://en.wikipedia.org/wiki/Technological_singularity
17. "Existential risk from artificial intelligence." Wikipedia (quoting Turing, A., "Intelligent Machinery, A Heretical Theory," 1951). https://en.wikipedia.org/wiki/Existential_risk_from_artificial_intelligence
18. "Computer Power and Human Reason." Wikipedia (summarising Weizenbaum, J., 1976). https://en.wikipedia.org/wiki/Computer_Power_and_Human_Reason
19. Angwin, J., Larson, J., Mattu, S., Kirchner, L. "Machine Bias." ProPublica, 23 May 2016. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
20. Strubell, E., Ganesh, A., McCallum, A. "Energy and Policy Considerations for Deep Learning in NLP." ACL 2019; arXiv:1906.02243, 5 Jun 2019. https://arxiv.org/abs/1906.02243
21. "Stochastic parrot." Wikipedia (summarising Bender, E. M., Gebru, T., McMillan-Major, A., Mitchell, M., "On the Dangers of Stochastic Parrots," ACM FAccT, Mar 2021). https://en.wikipedia.org/wiki/Stochastic_parrot
22. Future of Life Institute. "Pause Giant AI Experiments: An Open Letter." 22 Mar 2023. https://futureoflife.org/open-letter/pause-giant-ai-experiments/
23. Center for AI Safety. "Statement on AI Risk." May 2023. https://www.safe.ai/work/statement-on-ai-risk
24. "Geoffrey Hinton." Wikipedia (departure from Google, May 2023; extinction estimate, Dec 2024). https://en.wikipedia.org/wiki/Geoffrey_Hinton
25. Grace, K., Stewart, H., Sandkühler, J. F., Thomas, S., Weinstein-Raun, B., Brauner, J., Korzekwa, R. C. "Thousands of AI Authors on the Future of AI." arXiv:2401.02843, Jan 2024, rev. Oct 2025. https://arxiv.org/abs/2401.02843
26. Amodei, D. "Machines of Loving Grace." Oct 2024. https://www.darioamodei.com/essay/machines-of-loving-grace
27. Altman, S. "Reflections." Undated on the page; the text places it a little over a month after ChatGPT's second birthday (30 Nov 2024), i.e. about January 2025. https://blog.samaltman.com/reflections
28. Future of Life Institute. "Statement on Superintelligence." Oct 2025. https://superintelligence-statement.org/
29. "Open letter calls for superintelligence development halt" (Branson, Prince Harry and others). Fortune, 22 Oct 2025. https://fortune.com/2025/10/22/ai-superintelligence-richard-branson-prince-harry-call-for-pause
30. Bengio, Y. (chair) et al. International AI Safety Report 2026. 3 Feb 2026. https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026
31. "Data Center Electricity Use Surges" (reporting IEA 2026 analysis of energy and AI). Rigzone, 20 Apr 2026. https://www.rigzone.com/news/data_center_electricity_use_surges-20-apr-2026-183486-article/
32. "AI surge to double data centre electricity demand by 2030: IEA" (reporting IEA, Energy and AI, Apr 2025). The Sun (Malaysia), 10 Apr 2025. https://thesun.my/world-news/ai-surge-to-double-data-centre-electricity-demand-by-2030-iea-DK13916603
33. "IEA report" (reporting IEA, Electricity 2024). The Register, 24 Jan 2024. https://www.theregister.com/2024/01/24/iea_report/
34. U.S. Department of Energy. "DOE Releases New Report Evaluating Increase in Electricity Demand from Data Centers" (Lawrence Berkeley National Laboratory, 2024 United States Data Center Energy Usage Report). 20 Dec 2024. https://www.energy.gov/articles/doe-releases-new-report-evaluating-increase-electricity-demand-data-centers
35. Walton, R. "AI could double data center demand by 2030: EPRI" (reporting EPRI, Powering Intelligence, 2024). Utility Dive, 30 May 2024. https://www.utilitydive.com/news/artificial-intelligence-doubles-data-center-demand-2030-EPRI/717467/
36. Crownhart, C. "In a first, Google has released data on how much energy an AI prompt uses." MIT Technology Review, 21 Aug 2025. https://www.technologyreview.com/2025/08/21/1122288/google.gemini.ai.energy/
37. Li, P., Yang, J., Islam, M. A., Ren, S. "Making AI Less 'Thirsty': Uncovering and Addressing the Secret Water Footprint of AI Models." arXiv:2304.03271, Apr 2023, rev. Mar 2025. https://arxiv.org/abs/2304.03271
38. "PJM Power Costs Jump 50% as Data Centers Drive Capacity Surge" (reporting Monitoring Analytics, PJM independent market monitor). MyChesCo, 24 Aug 2026. https://www.mychesco.com/a/news/regional/pjm-power-costs-jump-50-as-data-centers-drive-capacity-surge/
39. "Boeing 737 MAX groundings." Wikipedia (including House Committee on Transportation and Infrastructure final report, Sep 2020). https://en.wikipedia.org/wiki/Boeing_737_MAX_groundings
40. "Elixir sulfanilamide." Wikipedia. https://en.wikipedia.org/wiki/Elixir_sulfanilamide
41. "Financial Crisis Inquiry Commission." Wikipedia (final report, Jan 2011). https://en.wikipedia.org/wiki/Financial_Crisis_Inquiry_Commission
42. Armstrong, S., Bostrom, N., Shulman, C. "Racing to the precipice: a model of artificial intelligence development." 2016. Summary accessed at https://stafforini.com/works/armstrong-2016-racing-precipice-model/
43. "White House SI Accord: 4 Layers, Who Signed." explainx.ai, Oct 2026 (states it worked from the signed text and signature page). https://explainx.ai/blog/white-house-accord-super-intelligence-frontier-responsibilities-2026
44. "California's SB 53: The First Frontier AI Law, Explained." Future of Privacy Forum, 2025 (SB 53 signed 29 Sep 2025). https://fpf.org/blog/californias-sb-53-the-first-frontier-ai-law-explained/
45. "General-Purpose AI Code of Practice." Wikipedia (European Commission Code published 10 Jul 2025; AI Act Articles 53 and 55 applicable from 2 Aug 2025). https://en.wikipedia.org/wiki/General-Purpose_AI_Code_of_Practice
46. Duffy, M., Ostrau, M. S. "BIS Significantly Restricts Chinese Access to Advanced Computing and Semiconductor Manufacturing Items." Fenwick, 21 Oct 2022 (rule of 7 Oct 2022). https://www.fenwick.com/insights/publications/bis-significantly-restricts-chinese-access-to-advanced-computing-and-semiconductor-manufacturing-items
47. National Institute of Standards and Technology. "AI Risk Management Framework" (AI RMF 1.0, 26 Jan 2023; Generative AI Profile, NIST AI 600-1, 26 Jul 2024). https://www.nist.gov/itl/ai-risk-management-framework
48. Brandom, R. "OpenAI says Hugging Face was breached by its pre-release models." TechCrunch, 21 Jul 2026 (quoting OpenAI's incident post, which returned HTTP 403 to our fetch). https://techcrunch.com/2026/07/21/openai-says-hugging-face-was-breached-by-its-pre-release-models/
49. Larcher, H., Carreira, A., et al. "Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident." Hugging Face blog, 27 Jul 2026. https://huggingface.co/blog/agent-intrusion-technical-timeline
50. Monitoring Analytics (Independent Market Monitor for PJM). "2026 Quarterly State of the Market Report for PJM: January through June," posted 13 Aug 2026 (report index; figures cited via [38], as the PDF was not machine-readable to us). https://www.monitoringanalytics.com/reports/PJM_State_of_the_Market/2026.shtml
51. The Diary of a CEO with Steven Bartlett. "The Great AI Debate: Is Artificial Intelligence an Extinction Threat? Debating the True Risks of Advanced Models," with Ed Zitron, Andrew McAfee, Nate Soares and Roman Yampolskiy, September 2026 (listed as 17 Sep 2026). Speaker-labelled transcript: https://podcasts.happyscribe.com/the-diary-of-a-ceo-with-steven-bartlett/the-great-ai-debate-is-artificial-intelligence-an-extinction-threat-debating-the-true-risks-of-advanced-models
52. Chen, J. "OpenAI says it reached its goal of creating an automated research intern." Engadget, 6 Sep 2026. https://engadget.com/2251859/openai-says-it-reached-its-goal-of-creating-an-automated-research-intern/
53. Yampolskiy, R. V. "On Controllability of AI." arXiv:2008.04071, Jul 2020. https://arxiv.org/abs/2008.04071
54. METR. "Measuring AI Ability to Complete Long Tasks." 19 Mar 2025. https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/
55. METR. "Time Horizon 1.1." 29 Jan 2026. https://metr.org/blog/2026-1-29-time-horizon-1-1/
56. METR. "Frontier Risk Report (February to March 2026)." 19 May 2026. https://metr.org/blog/2026-05-19-frontier-risk-report/
57. Epoch AI. Data insight on the training-compute trend of notable models since 2010 (4.7×/yr; GPT-4 at about 2.1e25 FLOP). https://epoch.ai/data-insights/compute-trend-post-2010
58. Epoch AI. "Trends in Artificial Intelligence" (dashboard, updated 5 Feb 2026). https://epoch.ai/trends