On September 29, the President signed Executive Order 14434, “Inaugurating the Era of Super Intelligence”. It tells federal agencies that “the executive branch shall use the terms ‘Super Intelligence’ and ‘SI’ in place of ‘Artificial Intelligence’ and ‘AI’.” The part I keep rereading is the definition. For now, SI means exactly the technologies the existing legal definition of AI already covers, and the President’s science adviser has 60 days to propose a new definition that can “modify, expand upon, or otherwise supersede” the old one. So the word changed first, and the concept is still being written.
The same day, tech leaders signed a White House accord on developing the technology safely and stood behind the President at a press conference. He explained the rename there too: “now we’re calling it again super,” because the old word “is just not an appropriate word” (1:07), and “whoever wins SI is going to win.”
I’m not writing this to criticize the rename. I think the instinct behind it is right, that what we call a technology reflects what we think it’s for, and then reinforces it. As someone who works on AI safety full time, I want to follow that idea into my own field, because I think the order did in public something our field does quietly all the time: it fixed a word before fixing what the word means. My guess is that some of the next progress in safety research will come from going back to those definitions, not only from better techniques.
What did “safety” mean on that stage?
If SI is a new word for an old thing, “safety” at the press conference was the reverse: one old word for several different things. I went back through the transcript and pulled every place someone said what safety is.
| Who | What they said | Safety as… |
|---|---|---|
| Greg Brockman | “everyone coming together to agree on how to move forward safely” (4:20) | agreement |
| Jensen Huang | “There’s no conflict between innovation, technology, and safety… we’re also creating new technologies to advance the safety of AI” (4:54) | a technology |
| The President | “the smartest people in the world watching over each other” (7:53), and later “self policing and… group policing” (28:06) | mutual oversight |
| Dario Amodei | “the technology has very real risks… the mechanism how we address those risks is still under discussion… we can win safely” (17:09) | an open problem |
| Mark Zuckerberg | “confidence that the technology works in the way that we intend” (18:21) | intent matching behaviour |
| Sundar Pichai | “processes and controls like we do in other areas like financial controls” (20:40) | compliance |
| Elon Musk | “by far the most likely outcome is an age of abundance… universal high income” (26:22) | a good outcome |
Some of these overlap, and I don’t think anyone here is wrong. They come apart once you ask how you’d check them, though. Pichai’s version can be audited the way a company’s finances are, Amodei says the mechanism is still being worked out, and Musk is describing where we end up, which nobody can check in advance. When one word stretches over all of that in half an hour, I read it as a sign the field hasn’t settled what its central word means either.
智能 or 智慧?
I’ve been thinking about naming for a while because of Chinese. In mainland China, AI is 人工智能, rén gōng zhì néng. 智能 is intelligence as capability: perceiving, computing, doing a task. 智慧, zhì huì, also translates as intelligence but leans toward wisdom and judgment, knowing what is worth doing. Taiwan uses 人工智慧 for the same field.
I’m not the first to think the difference matters. In Taiwan’s Science Monthly, Chen Shuo-fu argued in 2025 that 人工智慧 is actually a mistranslation, and that AI should be 人工智力, because in psychology intelligence and wisdom are separate things and intelligence is necessary for wisdom but not enough for it. He comes down on the opposite side from Taiwan’s usage, but his reason is the same distinction I care about: the two words point at different things, and calling a system wise when it’s only capable blurs that.
So I counted how China’s two main AI policy documents use the two words.
新一代人工智能发展规划 New Generation AI Development Plan, 2017
智慧 only in: 智慧城市 ×3 · 智慧法庭 ×2 · 智慧医院 ×1
关于深入实施“人工智能+”行动的意见 Opinion on the “AI+” Action, 2025
智慧 only in: 国土空间智慧规划 ×1
In the 2017 plan, 智能 appears 633 times and 智慧 6 times. In the 2025 “AI+” opinion it’s 160 to 1. What surprised me more is where 智慧 shows up. Every time, it’s attached to a place the state deploys the technology: smart cities, smart courts, smart hospitals, smart spatial planning. The technology itself is always 智能.
Companies seem to use the words the same way. I looked at how four of China’s biggest AI companies name their business lines, in annual reports and on their own product pages, and put them next to the State Council documents. Each term is shown by the half that follows the prefix, so 城市 in the 智慧 row means 智慧城市, and I sorted the terms by what they attach to.
Sorted from machine to institution, the two words barely overlap
Machines & technology
Sectors
Places & institutions
● one source using the term, out of five: the State Council plans, iFlytek, SenseTime, Huawei, Baidu. Dashed: the one machine called 智慧. Hover a term to see its sources.
Machines and technologies get 智能: cars, driving, voice, cloud, hardware. Places and public institutions get 智慧: the city, the court, the hospital, spatial planning. I didn’t find a single place called 智能, and only one machine called 智慧, Huawei’s 智慧屏 screens. The middle is where they overlap. Sectors take either word, and medicine takes both, 智能医疗 in the 2017 plan for medicine as a technology and 智慧医疗 at iFlytek for healthcare as a service. So what gets called wise is mostly the institution the system runs, not the system. This is four companies and two documents rather than a survey, and SenseTime reorganized its segments in 2023, so I’d treat it as a pattern worth testing, not a result.
That fits how I read Chinese AI policy more broadly. If AI is an ability, the questions that come up first are industrial ones, like how much compute, which sectors and how fast. Matt Sheehan’s study of how China’s AI regulations get made finds that the overriding goal of the rules has been information control and the political and social stability that follows from it. Both are about keeping a capable technology orderly. Whether the system itself has good judgment isn’t a question the word suggests, and policy doesn’t spend much time on it. (The 2017 plan isn’t silent on risk. 安全, safety or security, appears 48 times and 伦理, ethics, 15. But much of 安全 there means national or data security.)
English has the same problem, and AI researchers have pointed it out for a long time. In 1976 Drew McDermott warned about “wishful mnemonics”, naming a piece of a program UNDERSTAND or GOAL and then believing it does that. Melanie Mitchell made the same argument in 2021: words borrowed from human intelligence shape how the public, and even AI experts, think about what these systems are. Ted Chiang has suggested that “artificial intelligence” could often be replaced by “applied statistics” (Financial Times, 2023), which would make many claims about it sound very different.
I can’t show from a word count that the word caused any of this. A country’s AI policy has many causes, and the vocabulary probably follows its priorities as much as it shapes them. My claim is that the word reflects how a technology is being deployed and then reinforces it: once 智能 is the default, some questions feel natural and others feel like a category mistake. That’s why the 60-day definition interests me. Whatever it says will decide which questions about SI feel natural in American policy for a while.
Smart, and then super
In summer 2023 I read Michael Sandel’s The Tyranny of Merit, and one detail from it came back to me during the press conference. Sandel counted how often American presidents said “smart”. Reagan and the first Bush rarely did. Bill Clinton and George W. Bush each said it at least 450 times, and Obama more than 900, often about his own policies. His argument is that over about forty years, being smart became the virtue public life ranks above the others, even though smartness isn’t the same as the practical wisdom it takes to govern well.
When a reporter asked whether it would be too late to prevent damage, part of the answer was that these are “seriously high IQ people” who are “really watching over each other” (8:03). That is trust based on intelligence, closer to 智能 than to 智慧.
“Super” has a similar double life. Someone else on stage spoke about the US remaining “the last great superpower on the planet” (16:01), so the prefix now belongs to both the country and the technology, and “whoever wins SI is going to win” joins them. I can’t tell whether “super” in SI means more intelligent than us or more powerful than everyone else, and the order doesn’t say yet. For safety research it matters which, since one points to whether we can oversee something smarter than ourselves and the other to whom it is safe for.
Are we building on the right concepts?
The executive order is unusually open about its sequence: name now, definition later. Most safety work, including mine, does the same without saying so. We take words like “agent”, “goal”, “intent” and “capability” as given and build on them. We write evals for a capability, train against an objective, look inside a model for a goal. That works if the words point at real things. If one of them doesn’t, we can be very precise and still miss.
This has happened in science before. Chemists spent decades asking how phlogiston leaves burning matter, and the question didn’t get answered so much as dropped once oxygen came along. I don’t know which of our words, if any, will go the same way. But “intelligence” seems like a fair candidate to look at, given that Chinese already has two words for it, one closer to ability and one closer to wisdom, and English keeps adding prefixes to it.
“Works in the way that we intend”
The line closest to my own work came from Zuckerberg, summarizing the accord: the goal is “confidence that the technology works in the way that we intend” (18:21). I’d accept that as a one-line definition of alignment, but most of the difficulty sits inside it.
First we have to say what we intend in a form training can use, like a reward or a spec, and models are good at meeting the letter of a signal in ways nobody meant. That’s outer alignment. Then the model has to actually learn what we said. It can pick up a goal that matches the signal in training and generalize differently later, which is inner alignment, and it’s the kind of failure audits are least likely to catch because behaviour looks fine until the situation changes. There’s also the question of who “we” is. “Humanity” isn’t one agent with one set of preferences. On that stage “we” covered the companies, their customers and the American public, who don’t always want the same thing.
The accord’s mechanisms are internal controls, risk review, external auditors and evaluators, and board review of the audit reports (18:50). I’m glad they exist. Still, an auditor checks against what someone wrote down as intended, with evaluations someone designed. A lot of our oversight ideas come from auditing companies or from control theory, where the thing being checked isn’t trying to look good to the checker. A capable model might be, and that gap is where I’d expect things to go wrong.
Can a superpower be talked out of racing?
The question I keep coming back to is how you would persuade a superpower, or people in general, that it’s worth giving up some capabilities momentum so that safety research can catch up. I don’t think asking works. If whoever wins SI wins, then slowing down alone looks like losing, and the situation reads like a Prisoner’s Dilemma where everyone races.
Katzke and Futerman argue in The Manhattan Trap that this reading is wrong. Their claim is that a race to superintelligence is a trust dilemma rather than a prisoner’s dilemma: each side would prefer mutual restraint, if it could believe the other side was restraining too. That is a Stag Hunt, which I wrote about in another post, and what gets players out of a Stag Hunt is assurance. The Cold War is the usual example. Neither side stopped building, but they still agreed on test bans and inspections the other side could check. The AI Futures Project’s AI 2040 “Plan A” (discussed here) is a version of this for SI, a verified US–China deal that pushes superintelligence out to 2040.
Maybe I’m narrowing my own worldview by seeing all of this through game theory. Since I started studying it, I tend to read strategic situations as games, and a game needs a few players with fixed payoffs. A superpower isn’t one player, though. It’s a president, agencies, companies and voters, who don’t all want the same thing. And the payoffs depend on what each side thinks it’s racing toward, which is the part nobody has defined yet. So the Stag Hunt might be the right model, or it might just be the lens I happen to have.
What this suggests to me is that the concept to examine first is “winning”. In the race framing it means getting there first. But if “super” means power, an SI that nobody can steer isn’t a win for the country that built it either, so being able to steer it is part of what winning has to mean. Amodei’s “win safely” points this way already. Seen like this, safety research is part of winning rather than a tax on it, since it’s the work that would make “win safely” something you can check instead of something you hope for.
So what is the future of AI safety research?
I think it needs both kinds of work. The work we already do on evals, interpretability, oversight and controls like the accord’s has to keep pace with deployment, and there’s more of it than people to do it. Next to that, someone has to check, slowly, whether “safety”, “intent”, “agent” and “intelligence” are the right concepts before more gets built on them. That kind of work is slow and hard to measure, which makes it hard to fund while the conversation is about winning a race.
The 60-day definition is a small, concrete version of the same problem. It could define SI only by what the systems can do, which would be the 智能 reading. It could also say something about whether we can understand and steer them, which is closer to 智慧. I don’t expect a legal definition to settle a research question. But if we’re going to call it superintelligence, I’d like the judgment part, the 慧, to be something we try to make safe too, and not just the capability.
Sources
- Executive Order 14434, “Inaugurating the Era of Super Intelligence”, September 29, 2026.
- White House AI accord press conference, USA TODAY, September 29, 2026. Timestamps above link to that moment in the video.
- Corin Katzke and Gideon Futerman, The Manhattan Trap: Why a Race to Artificial Superintelligence is Self-Defeating (2024).
- AI Futures Project, AI 2040 “Plan A”, as discussed with Daniel Kokotajlo on 80,000 Hours (2026).
- 国务院, 新一代人工智能发展规划 (2017) and 关于深入实施“人工智能+”行动的意见 (2025). Word counts are mine, from the gov.cn texts.
- 陳碩甫 (Chen Shuo-fu), AI 的中文翻譯之辯：智力與智慧的差異, 科學月刊 665 (2025).
- Company business lines: iFlytek 2024 annual report summary; SenseTime segments as reported by 财联社 and 央广网; Huawei via 中证网 and its own 智慧城市 solution page; Baidu via 格隆汇.
- Matt Sheehan, China’s AI Regulations and How They Get Made, Carnegie Endowment (2023).
- Drew McDermott, “Artificial Intelligence Meets Natural Stupidity”, SIGART Newsletter (1976); Melanie Mitchell, Why AI is Harder Than We Think (2021).
- Ted Chiang, interviewed by Madhumita Murgia, Financial Times (2023).
- Michael Sandel, The Tyranny of Merit: What’s Become of the Common Good? (2020).