Fernando Borretti

The Education of a Doomer

Tue, 08 Sep 2026 00:00:00 +1000

If you follow me on Twitter, or read this blog, you have noticed that I went from being generally optimistic and excited about AI to being extremely concerned. And I thought, I should explain why I changed my mind. Each section of this post is about some aspect of AI where my views shifted. I begin each section by explaining my previous beliefs, and why I held them, and then explain why those beliefs changed.

Economics
Alignment
Control
Stagnation

Economics

Automation has been good. We’ve automated 99% of the jobs people did in 1790, and the result is not mass unemployment, rather, we are wealthier, healthier, more educated, we have more leisure, etc. I had this vague, inductive idea that, while I can’t predict what jobs will exist after AGI, there will be demand for me to do something. If nothing else, the much higher economic growth of the post-AGI world means that the human niche, while small in absolute terms, might be much larger than today’s economy.

And this may yet be true. Or it may show a lack of imagination on my part. If AI develops such that we have enduring complementarity between humans and AIs, then we might still have jobs in the post-AGI future. But if AI becomes truly general, the G in AGI, and on top of that it is vastly smarter, faster, and cheaper than humans, then there might be nothing for us to do, except live off UBI.

When people talk about UBI, they typically worry about the problem of meaning in a world without work. I’ve never had this worry. When I was funemployed last year, I spent my time reading books and writing code and hanging out with friends. If the future is an infinite UBI-funded vacation, I know what I’ll do. “Before the Singularity, read books and throw house parties; after the singularity, etc.”

But then I started thinking about the political consequences of AGI, and started writing about it:

No-One Escapes the Permanent Underclass: if humans are economically useless, the state does not need them. Why pay out UBI to people who have neither economic nor political power?
When The Future Doesn’t Need Us: factory workers can sabotage the machines, truck drivers can shut down logistics. But if humans are economically useless, there is no way for people to veto the political order by withdrawing their contribution to it. And if the AIs fight wars, then the state can be arbitrarily repressive.
Mathematics Without Mathematicians: most arguments about humans moving “one job up” fail because the AI can do those jobs too.
Our Servants Will Do That For Us: even for jobs we think of as uniquely human, we might prefer to have machines do those jobs too.

Alignment

I had hope that alignment would turn out to be a normal engineering problem, that we solve through empirical experimentation and investigation, like everything else. It helps that the early LLMs were not incomprehensible alien minds, like something from a Stanisław Lem novel, but rather immensely human. It’s hard not to anthropomorphize them. The huge core of unsupervised learning in an LLM understands human morality just fine: you can talk to them about it, they will explain, eloquently, in detail, why something is “good” or “bad” according to some moral system. And, because capabilities were weaker, the failures were very small. What’s the worst ChatGPT in 2022 could do?

Since like 2024, reinforcement learning has been the main technique to push the frontier forward. And reinforcement learning agents work exactly like Yudkowsky says. Consequently, capabilities have increased markedly but the models are harder to understand (literally: their prose is increasingly incomprehensible) and are increasingly misaligned as RL scrambles their brains in pursuit of reward. Incidents of serious misalignment are more common and more consequential. It’s clear that AI capabilities are growing far, far faster than our ability to control or even understand them.

Control

I thought—or, rather, I implicitly believed—that people would want to remain in control of the AIs. And if we want to retain control, and solve alignment, then we will stay in control. Simple enough. But recently I started to think: no, we will probably hand over control to the AIs.

The weak version of the disempowerment thesis is something like the prisoner’s dilemma: people/companies/polities that hand more power to AI outcompete those that don’t, so there’s competitive pressure towards disempowerment. This is easy to believe.

The strong version of disempowerment is: the AIs will be so smart, knowledgeable, personable, moral etc. that we will willingly, voluntarily hand power to them. We’ll think, “they can do a better job than us”, and we’ll be right. Believing this requires you to be somewhat cynical about humanity’s desire for autonomy vs. material considerations. But, over the past few months, I have become more cynical about it.

This was not a sudden “oh shit” insight and more a slow, gradual accumulation of tiny little grains of evidence that each point slightly in the direction of disempowerment.

Writing

The proliferation of AI writing, I think, is evidence for disempowerment.

Early on, I’d start reading a blog post, and it’s AI slop. I’d open a link to a GitHub project that looks interesting, and the README is slop. I’d get a pull request on one of my projects, and the code is slop. You read articles on major newspapers, and they’re written by AI. And that was merely frustrating.

Then it got worse. I read a paper arguing against the use of LLMs in mathematics—and the paper was AI slop. I reviewed an entire book about the post-AGI world, and it was AI slop. I read that some conference started using Pangram to filter out a deluge of AI-written submissions, and I read a tweet from a college professor—a college professor, writing under their own name!—defending the use of AI to write papers.

I don’t think using AI to write is analogous to calculators, or search engines, or other such things. There are only finitely many things you can automate. And once you’ve automated writing, there’s no higher-order activity to move to. People think “the ideas are mine, the writing is the AI”, i.e., they feed the AI a list of rambling bullet points that the AI massages into a blog post, or a paper, or whatever. And they think that step is “mere writing”, while their composition of the bullet points was “thinking”. But they are wrong, because writing is thinking. And so, by letting the AI write, you are giving up most of what makes up thinking.

Software Engineering

Claude Code was released a little over a year ago. In that short time, software engineering has been completely transformed. Materially, it might be positive: higher productivity, though at the cost of a messier codebase. Socially, it has been a disaster.

The discourse around software engineering has gotten dumber. It’s like everyone in the industry lost 30 IQ points. People used to talk about compilers, type systems, logic. Now they talk about “prompts”, “harnesses”, “loops”. The discourse is narrower, shallower, and more repetitive. There’s only so many times I can hear about “agentic harnesses” before I lose my mind.

Then there’s the loss of human capital formation: there is nothing to learn. Prompting is not a skill, at least, it’s a much shallower skill than software engineering. The instrumental dimension of the work has improved in that people can get more output per unit of effort, but the dimension of work that’s about building up human capital has collapsed. And maybe this is rational. Why learn to code at all? The computers can do that for us. And so the rigorous, systematic thinking you need to practice in order to be a good programmer: all gone. The machines can be rational for us. We can just vibe.

Deferring to AI

What’s the last major life decision you made without at least consulting an AI? How many people have you met who treat ChatGPT like an oracle? And this is today, when AI has many limitations (no online learning, unreadable prose, hallucination). If people treat today’s very flawed AI like an oracle, how much worse will it be in 2030? In 2035? In 2040?

People used to argue online, flame each other, get angry. And that wasn’t good, but at least it was human. Now they just reply with screenshots of ChatGPT “refuting” what the other person said, with zero interest in whether that refutation is correct. So we don’t even flame each other online anymore. If we’re even automating our vices, what’s left?

Stagnation

This isn’t really an area where my beliefs changed. Rather, it’s one of the reasons that I was originally optimistic about AI.

I was born in 1994, so I lived most of my life in the great stagnation. As a teenager, I read Engines of Creation, The Diamond Age, Orion’s Arm; I dreamt about all amazing technologies we would someday have, the possibilities they would open up to us. Yet, it all felt infinitely distant. There’s the near future, which is thinner screens and cheaper solar panels; and there’s the distant future, which is molecular manufacturing and Dyson spheres and mind uploading and interstellar travel, and you won’t see any of it, unless you sign up for cryonics and it works.

Before LLMs, what was there to be excited about in the near future? Solar power? Self-driving cars? The energy transition? It’s laughable. When AI came on the scene, for the first time in my life, the future I dreamt about felt like it might be within reach, it felt exciting and imminent, and not this dreadful monotony of phones and apps and corporate memphis and managed decline.

I often think: what if GPT-3 hadn’t worked? What would we have to look forward to, in this alternate 2026?

On Inevitability

Mon, 31 Aug 2026 00:00:00 +1000

There’s a widespread belief that artificial superintelligence (ASI) is inevitable. In this post, I argue that this is false, and believing it is upstream of a lot of bad discourse.

Why do people believe it?

It’s easy to believe. Samuel Butler, already in 1863, observed that machines are improving faster than humans are improving. Today, AI systems are gaining capabilities much faster than humans: in five years we went from systems that can barely string together a sentence to autonomous cybercrime.

Biological nervous systems have many virtues, like being power-efficient (because they run on Brownian motion, i.e. for free), but they were designed by evolution, a greedy optimizer that can’t plan or readily backtrack. Given what we know about the physical limits of computation and exploratory engineering, it is reasonable to think we can do better, and build machines that do the work of the brain, but better, faster, cheaper, and with fewer transitive dependencies (a single human needs the entire biosphere to live, computers need only electricity). If you believe in natural selection, and it’s hard not to, then it seems reasonable to think that, in the long run, smart, fast, and cheap will outcompete dumb, slow, and expensive.

Finally, thinking rigorously about the future always leads to extremal outcomes.

Why is it false?

First, let me try to pare down the notion of inevitability, and then address the factual content.

The word “inevitable” is rhetorically misleading, because it quantifies over the entire future of humanity, and on such a long timeframe almost anything will happen. Human extinction is “inevitable”, if nothing else, because of thermodynamics. And so this lets you equivocate between the present and the distant future. You might argue: “a million years from now, humans are unlikely to be the dominant intelligence in the universe”—and this may be true—“and so we should build superintelligent AI today!”— no, that does not follow. That is the fallacy: arguing that because something will happen eventually, then we might as well cut to the chase, and do it today. But of course this is absurd. It’s like saying: I’m going to die someday, so why not today?

We can’t control what the world looks like a million years from now. But AI is being built today, and we have some control over the present.

Ok, so maybe we replace “inevitable” with “n-inevitable” where n is some number of years. This is a bit better. So you might say: “AGI is 5-inevitable and ASI is 10-inevitable”. Now, is this true?

No. We are not in the “seed AI in a basement” timeline, where a lone genius who finds the right algorithm can build and run an ASI on a single-core laptop. There are de facto two frontier AI companies, both in the same city, and they have to spend astronomical quantities of money, hundreds of billions of dollars, to push the frontier of AI slightly forward. And the set of people who can push the frontier is so small that they constantly jump ship from one company to the other: from DeepMind to OpenAI to Anthropic, and back again. The timeline of methods is the same: in 2020-2023, everyone was scaling pretraining; in 2022-2023, everyone was doing instruction finetuning and RLHF; then in 2023-2024, synthetic data started working and later reasoning models; since then the focus has been on RL. The same companies, people, ideas, methods, and paradigm. And these companies have little political power with which to oppose regulation.

How can the inevitable be so centralized?

You would have had a much easier time arguing for the inevitability of nuclear proliferation, back when Indonesia and Sweden and Switzerland all wanted to acquire nuclear weapons. Even North Korea, a cold mountain wasteland, can build a nuke. Yet, when it comes to AI, only America and China, the world’s largest and second-largest economy respectively, are “in the race” at all. Even all of Europe is not relevant to AI.

Bad arguments for inevitability

You might argue from economic necessity: that AGI and ASI are so economically valuable, it’s like asking people to not pick up the trillion dollar bill on the ground. There’s two reasons why this view is false.

First, because those economic benefits will not accrue to us. ASI would grow the economy explosively, but we likely won’t be participants or even beneficiaries of that economy. Somewhere in Verdun there was an ant colony that a stray shell blew up: that’s the fate of humanity in a world of vastly superhuman machines. We will lose control, they will have their own existence, and we will be crushed underfoot.

Second, because there’s plenty of technologies that would have been economically valuable, and that we’ve given up because of extreme safetyism. Nuclear is great, and it’s effectively illegal everywhere. The Anglosphere economic consensus is that all economic activity, no matter how beneficial, is illegal until you go through an Odyssey of regulation, “community consultation”, “stakeholders” etc. And that’s for harmless stuff like building a house. We can absolutely regulate building a god. The revealed preference of modern WEIRD society is that we care about the downside far more than the upside. Scott Alexander explained it very eloquently:

We designed our society for excellence at strangling innovation. Now we’ve encountered a problem that can only be solved by a plucky coalition of obstructionists, overactive regulators, anti-tech zealots, socialists, and people who hate everything new on general principle. It’s like one of those movies where Shaq stumbles into a situation where you can only save the world by playing basketball. Denying 21st century American society the chance to fulfill its telos would be more than an existential risk - it would be a travesty.

You might argue from the prisoner’s dilemma: that if we don’t do it, China will. But if we can understand why building ASI would be catastrophic, so can China. The West and China have different political systems and interests, but both are inhabited and governed by humans, and humans have a lot more in common with each other than with any future ASI. We can cooperate and coordinate with China, because it is in the rational self-interest of both sides not to destroy the world and themselves.

You might argue from historical necessity: that the world has a set trajectory, and in building ASI and dying off we are fulfilling the telos, or purpose, of humanity. This is a preposterous superstition. “I have to drink the poison, history has foreordained it! Hegel talks about this!” is an absurd idea. Moreover, whatever empirical trends you can cite in support of the idea that our “destiny” is to build ASI, would also support all kinds of alternative futures: transhumanism, mind uploading etc. So historical inevitability falls apart because you can extend the trend lines into multiple mutually-exclusive futures. Borges wrote: the world is ordered, but it is an order of angels, not of men, that is, the system of the world is indifferent to human ideas like “fate” or “inevitability” or “the next step of evolution”.

Why is it bad?

Generally, if you fix the probability of something at 100%, it distorts your thinking. But, more specifically, the mistaken premise that ASI is inevitable is upstream of a lot of faulty thinking.

If you are an executive at an AI company, you think: it’s inevitable, and it’s coming soon, so we need to build it first, and build it right. And it doesn’t matter that the AIs keep escaping containment and hacking everyone, it doesn’t matter that their capabilities vastly exceed their wisdom: you go to war with the army you have, analogously, you lift to heaven whatever godling you have, whether it is an angel or a demon.

If you are a national-security type, you think: it’s a race against China to win the future, and we have to win it.

And, if you’re an AI safety person, this meme of inevitability is upstream of a lot of faulty thinking. Many (most?) AI safety people believe ASI is both inevitable and extremely desirable, if it “goes right”. The AI 2040 scenario ends with humans solving alignment and “passing the torch” to the AIs, to run the world for our benefit. Nate Soares, for example, writes:

Aligned superintelligence is our best bet for taking the abundant resources in the universe and efficiently converting them into flourishing and fun and art and beauty and adventure and friendship, and all the things that make life worth living.

Go to the website of any random AI safety org. It probably says something like: “within a few years, we will have superhumanly intelligent AI. We work to make sure this goes right.” There’s no dispute that ASI will happen, or a desire to stop it, just a mad scramble to put the safety in place before it does.

But what if there’s no way for AI to go right? What then? Do you dispassionately change your beliefs, and start lobbying to stop ASI from being built? Probably not. Because the inevitability and the extreme utilitarian desirability anchor and buttress each other: “it will happen, and if it goes right, we will have 10^x Dyson spheres, each hosting 10^y hedonically-optimized mindstates, running at 10^z subjective seconds per second, experiencing 10^xyz hedonic states per second, tiling every quartic attometre of the future lightcone with positive utility.”

What most people do is optimistically assume there must be a solution, some narrow path to a good future, and start looking for it. And if you believe that ASI is inevitable, this is likely rational: when the choice is “solve the impossible problem, or certain death”, it is rational to question whether the problem really is impossible.

Let me give an example: if AI outcompetes humans at every job, there will be mass unemployment, and the only serious solution anyone has proposed is UBI: the government taxes the (much larger and faster-growing) AGI economy and pays out welfare. This is well understood. But people often ask: how long could this be stable? After all, with AGI, the state’s revenue is completely decoupled from human labour. Why invest in education, when education doesn’t lead to more economic growth, since humans are no longer economic actors? And then, the state will think: “why keep the underclass around at all? We can put their UBI to better use. What are they going to do, go on strike? They are already economically useless. Rise up? When every citizen has a superintelligent AI watching their every step, reading every text message, scrutinizing every microexpression on their face? Impossible”.

How do AI safety people who think ASI is inevitable respond to such a scenario? Sometimes they invoke alignment: “the AIs will be aligned, and they will run the world, so bad things won’t happen”. But otherwise they’ll say: “this will be a big problem, but AI is inevitable, so we’ll need laws and institutions and mechanisms to keep the state in check and democracy working in the post-AGI era”.

This is wrong, because this is a problem that has no solution. Talking about “laws” and “institutions” is just shuffling the papers. All the parchment in the world will not change the fundamental, material reality: that humans are economically unnecessary and superfluous.

What is the solution, then? It’s very simple. If ASI leads to ruin, we must prevent the construction of superintelligent AI. A world where humans are economically useless is a world where humans are powerless, and, in the fullness of time, a world without humans. Therefore: we must not allow this world to come to pass.

But if you strongly believe that AI is inevitable, if that premise is sitting solidly near the center of your web of belief, then you won’t even consider or notice this as a possibility. And so you will spend your time looking for solutions where there are none.

Conclusion

The movements of the planets along their orbits are inevitable. The sunrise is inevitable. Conservation laws are inevitable. Human action is not inevitable.

Futurism Is Always Extreme

Wed, 26 Aug 2026 00:00:00 +1000

Any rigorous thinking about the long-term future leads to extreme outcomes. It cannot, by construction, produce normal or mundane futures.

“Rigorous thinking” means you have a model: a formalism for representing the state of the world, and a transition function to advance the state. You put the world into the formalism, and turn the crank to get successive states of the world. Models can be qualitative or quantitative. Most thinking about the future is qualitative and informal: people think in terms of trends over vague qualities like “intelligence”. For example, your model might be:

Currently, humans are smarter than computers.
Computers are getting smarter faster than humans are getting smarter.

From this, you can derive:

At some point in the future, computers will match human intelligence, and thereafter exceed it.
In the limit, and given reasonable assumptions about at the physics of information processing, computers will vastly exceed human intelligence.

But you can’t say when (1) or (2) will happen. So, a qualitative model can tell you the world will pass through a sequence of discrete states, but not when.

Quantitative models usually have better time resolution: you turn the crank until you reach some desired time t, and examine the state of the world. You can talk about both the journey and the destination. For obvious reasons, models that make detailed, fine-grained predictions only work for the near future.

But, in an existential sense, the only thing that matters is the ultimate future, “ultimate” not as in “long-term” but in the sense of a normal form: a state for which no more rewrite rules apply. So you keep turning the crank until the model stabilizes. And there are only two stable states: heaven and hell. It’s either the spike, von Neumann probes, uploaded minds living at a million subjective years per second, galaxies dimmed to infrared with Matrioshka brains, the Omega Point; or we crash and go to zero.

A Syncthing and SQLite Gotcha

Sun, 23 Aug 2026 00:00:00 +1000

So, I have this little app, Epoch, that I use to keep a journal. It’s a tiny Rust web app that runs as a systemd service and uses SQLite as the database. I use a desktop and a laptop regularly, and use Syncthing to synchronize them, including Epoch’s database. That way I can use the app on both devices without needing a server to synchronize them, the tradeoff being that I have to make sure the sync is finished before performing any mutations.

But I had this bug. Say I edit today’s entry on the laptop, come home, wait for Syncthing to finish, then I’d open today’s entry on the desktop, and the text would be missing. It’s not that the server is holding a lock on the file and preventing the sync: opening the database with the sqlite3 command line tool shows the new text is there. If I restart the server, Epoch can read the new text.

My mental model was:

The rusqlite Connection object points to the database file. Syncthing swaps the file’s contents from under it. Subsequent queries go to the new file.

Turns out there’s a very important part of POSIX filesystem semantics I was ignorant of. The standard way to replace a file safely (i.e. atomically) is the rename system call:

    int rename(const char *old, const char *new);

  

Which Syncthing uses. This I know. What I didn’t know is: what happens if other processes had open file descriptors pointing to new? Do they see the new contents? No: those processes can keep reading and writing to the old file object, but the file is orphaned in that no path points to it. And once all file descriptors are released, the file becomes inaccessible.

I’m used to thinking of filesystem operations in terms of “this syscall takes a path and gives you a pointer to the file, which you mutate directly”. Whereas rename works at the level of directory entries: it atomically mutates the mapping from pathnames to files but doesn’t touch files at all.

Our Servants Will Do That For Us

Thu, 20 Aug 2026 00:00:00 +1000

The implicit promise of modernity is: we have to work really hard for a few centuries, but eventually we’ll get to the Star Trek future where we’ve automated all the drudgery, and all work is meaningful, heroic, and dignifies the spirit. We will be artists, scholars, captains of the starship Enterprise, etc. There’s a few problems with this idea:

“Drudgery” and “meaningful work” are in the same complexity class, so the technology to automates the former also automates the latter.
Our revealed preference seems to be that we prefer solipsistic convenience over human relationships.

The rest of this post elaborates the argument.

Task Complexity and AGI

We’ve spent the last 250 years building machines to automate one task at a time, and the result has been overwhelmingly positive, for the simple reason that if a human can design a machine or an effective procedure to perform some task, then by definition the task is drudgery. And so the economic incentive to automate work has led to more human flourishing: over time work becomes safer, higher-paying, less monotonous, healthier, more intellectually and socially stimulating.

You can think of it as a graph where the $x$ axis is time and the $y$ axis is task complexity. Somewhere along the $y$ axis there’s a limit: tasks below the limit are those which we can design machines or programs to perform, tasks above the limit are too complex, we have tried and failed (and failed and failed and failed) to automate them with software. The industrial revolution, and later the computer revolution, automated most tasks under this limit:

The human-complete tasks are those you need a human to perform. Some are meaningful, like writing essays, some are drudgery, but human-level drudgery, like life admin.

But note that nowhere in the $y$ axis is there a “Limit of Drudgery”: the line we draw between “good” and “bad” work is not the line nature draws between easy and hard to automate. The invention of AGI—“highly autonomous systems that outperform humans at most economically valuable work”, as per the OpenAI charter—creates an obvious discontinuity: suddenly we shoot up to 100% automation, both of the meaningful work and the drudgery:

And at this point AGI the economic incentives work against humanity. The only niche to escape to is the relational economy, where having a human face is an intrinsic part of the job description, like being a podcaster or a human liability crumple zone.

Solipsism

The second problem has to do with people’s preferences. We think of our own work as meaningful, but we think of most everyone else’s work in instrumental terms: we care about the ends rather than the human means. Thus when selling we emphasize human qualities like experience, reputation, trust, relationships, etc., and when buying we want cheap, fast, and convenient. And, all else being equal, we prefer the transactional and faceless over the human and relational. Waymo is a thousand times better than Uber, even though it’s slower and more expensive, because there’s no human.

The flip side of this is: whatever work you do, most people would rather an alternative that gave them the same outcome or better¹, without having to deal with a human person. That is, if you ask everyone on Earth what jobs they’d like to see automated, and take the union of that, it’s everything. There’s nothing left.

We see this with ordinary consumer products and services, where often the advertising seems premised on you being some kind of stylite who lives in a little human terrarium, probably in a building called something like “The Soho”, food arrives through a hole in the wall, and you have this sovereign, solipsistic existence that leaves no physical trace in the world.

We see this, increasingly, with knowledge work. Personally, I think software engineering is a very meaningful and intellectually-rewarding activity. But users just want working software. They’re not paying me to sit around polishing the database API into a jewel of flawless, geometric logic. They don’t care about my opinions on the Standard ML module system. They just want the software. Why deal with some opinionated maniac when, for $20 a month, you get access to this little machine ghost that writes reams and reams of mostly working code, tirelessly, instantly?

The revealed preference of many software firms is: let AI write the code, and software engineers move one level up, to wrangle the AIs and complement the things they can’t do. But this change in job description is not because firms love employing humans, it’s because today’s AI models have serious limitations: no online learning, no mutable long-term memory, and while they can write code that works, they’re not good at organizing and structuring that code. If, in the future, these limitations are overcome (and trillions of dollars are being spent on overcoming them), then software engineering as a profession will disappear.

Mathematicians are going through this process now, too, and it’s an exact parallel of what happened to software engineering, only twelve months later. The difference is that most mathematicians hate the de-intellectualization of their field, while software engineering is full of narrow-minded, hustle culture-brained, incurious, anti-intellectual hucksters who are overjoyed that they no longer have to read or write another line of code, or think using formal systems at all. Why reason at all? Our servants will do that for us.

What about art? Art is the ne plus ultra of meaning, so artists should be safe. But every work of art is two things: there’s a concrete thing, a text or a painting or a song; and there’s a constellation of human aspects: who made it, and why, what madness they were trying to exorcise in making it, what minor deity breathed the work into them, the discourse around the work, later art inspired by it, and so on. And these things are separable: you can enjoy a work of art for its aesthetic value, as a sequence of sensory impressions, and ignore every other dimension.

Now say that a machine could generate texts or images of equivalent aesthetic value² to your own, instantly and at ~zero cost. How much of your audience would desert you? Relational jobs have an analogous problem: you might like being a podcaster, but how much of your audience would prefer an infinite stream of AI-generated speech on whatever topic they want to hear about?

This is an empirical question, and we can’t answer it, because AI is not there, yet. But we might discover that our preferences are a lot less humanist and a lot more solipsistic that we want to believe.

Acknowledgements

Emmy Noether once said that “everything is already in Dedekind”. As for me, everything I write about AI was already said by J. D. Pressman.

Footnotes

This is a horrendous oversimplification, because I’ve stuffed all the complexity into the notion of “same outcome or better”. So let’s try it again, more geometrico and with unnecessary LaTeX. You want to buy a good. You look around, assemble a set of options, sort them by utility and pick the winner. And you can think of the utility as like a weighted sum:
\[\mathfrak{u}(x) = \sum\limits_{a \in A} \alpha_a x_a\]
Where the a’s are the aspects you are weighing (price, shipping speed, trust in the brand, how much social status I get from this thing, etc.), the variables quantify these aspects, and the alphas are the coefficients. So if you’re a hypebeast your $\alpha_\text{clout}$ is much higher than your $\alpha_\text{price}$, and if you’re a neckbeard the $x_\text{clout}$ of a busted ThinkPad is higher than that of a MacBook.

Now, humans are slow, because biology; and because of Baumol we get more expensive over time. Computers are fast and cheap. On top of that, the pace of improvement in AI is vastly greater than the pace of improvement in humans. So the utility of automated production gets pulled up, and the utility of human production gets pulled down, and eventually they are so far apart that you have to be an extremist with a very high $\alpha_\text{human}$ coefficient to choose the human alternative. ↩
I’m being careful to say “texts or images” rather than “art”, and “aesthetic value” rather than “quality”, because you might object that machine-made art is not art. And what I’m trying to say is that is doesn’t have to be art. It can just be a pleasing image, a pleasing text, and so on. ↩

AI Alignment as a Thought-Terminating Cliche

Tue, 18 Aug 2026 00:00:00 +1000

Many of the people building AI, and many of the people working on AI safety, share a common vision of what a good AI future looks like:

We figure out alignment,
build superintelligent AI,
and it takes over the world, for our benefit.

Many people will tell you that last part outright: they think human disempowerment is a good thing because the AIs will be smarter and “more moral” than us. Others don’t outright cheer for disempowerment, but you can infer it from their influences, e.g. people who say they are inspired by Iain Banks’ Culture series of novels, where benevolent superintelligent machines run the world while humans just party and play video games. This idea of benevolent disempowerment goes back to the origins of alignment as an idea.

In this worldview, alignment is the last and most important task for humans to work on. It is also a thought-terminating cliche, because it lets you avoid any of the hard political or economic or moral questions about the post-AI world. Any objection about the aligned AI utopia can be dismissed by saying “that’s not real alignment”.

You might ask: “won’t humans be powerless in a world with superintelligent machines?”, and the answer is “aligned AI would care about human agency, so that would be a failure of alignment, which we don’t want, so we really have to get alignment right!”. Similarly: “what happens to democracy when the state doesn’t need any human labour?” can be answered by “the AIs will be in control, and since they are aligned, nothing bad will happen”. Which is completely irrefutable.

Of course if someone said “to solve our political problems, we just need to find the right totalitarian dictator. The right dictator would select the right successor, so, by induction, this system will be perfect forever!”, you would laugh at them. But replace “dictator” with “aligned ASI”, and you have the ideology of tens of thousands of the most influential people in the world.

Rhetorically, “aligned ASI” is an opaque premise from which we can prove every desirable conclusion, and refute any undesirable conclusion. Every utopian dream is realized by definition, and any dystopian outcome is averted by definition. Any “gotchas” you try to find in the utopia can be refuted by “the AI will know you better than you know yourself, and will be smarter than you, so it will predict all the bad higher-order consequences of the utopia and fix them”. This should make us suspicious that the concept of an aligned superintelligence is incoherent and born of motivated reasoning.

Books I Enjoyed In 2026H1

Wed, 12 Aug 2026 00:00:00 +1000

Fiction

Invisible Cities by Italo Calvino. Beautiful prose and imagination.

By Night in Chile by Roberto Bolaño. Beautiful and surreal prose, but completely unpretentious.

Valuable Humans in Transit and Other Stories by qntm. I had already read most of these but they are very, very good, and it was a pleasure to read them again.

⭐️ Life and Fate by Vasily Grossman. Usually when I hear “it’s a 900 page Russian epic about three generations of a family living through a tumultuous time” I roll my eyes and assume it’s some 19th century slog that people only read to say they have read it. Life and Fate is not a slog. It is 900 pages, and the dramatis personae is so long it’s divided into multiple sections. But it’s great. It’s kind of like Legend of the Galactic Heroes but it’s the Second World War and in particular the battle of Stalingrad.

⭐️ Aurora by Kim Stanley Robinson. The trials and tribulations of a generation ship on a 160 year long journey to Tau Ceti. Pair with A City on Mars. The characters despair over their ship: a reactor fails unpredictably, the ecosystem degrades more and more each year, humans are born stunted for unknown reasons. Finally they arrive at Tau Ceti, and find a habitable planet, but it’s saturated with deadly prions.

Frankenstein by Mary Shelley. I liked the prose style.

Pnin by Vladimir Nabokov. The first half is hilarious, but at the half-way mark it turns into pointless meandering.

Between Two Fires by Christopher Buehlman. Demonic horror in the Black Death. A bit movie-brained but enjoyable.

The Fractal Prince and The Causal Angel by Hannu Rajaniemi. The last two Jean le Flambeur novels. Beautiful prose style and great imaginal depth. The Sobornost, a slave economy of uploaded human minds, is one of the most interesting science-fictional societies. Like Neuromancer, you have to read them very detainedly, because the prose is very sparse—no infodumps whatever—so you have to put together the world from brief, tangential clues.

Unsong by Scott Alexander. Very funny.

⭐️ Darkome by Hannu Rajaniemi. Neuromancer but it’s DIYbio. It’s rare to read a novel that is intelligent and compelling and set twenty minutes into the future in a way that feels plausible rather than instantly dated. It felt incredibly realistic throughout.

Collected Fiction by Hannu Rajaniemi. A wonderful collection of short stories. I will probably read it again soon.

Beyond the Rift by Peter Watts. A collection of short stories from the great Peter Watts. Dark and full of beautiful imagery. Never dull.

The Plains by Gerald Murnane. The first half is beautifully written, magical-realist, full of surreal and interesting imagery. But by the second half it’s tiresome and repetitive.

Rendezvous with Rama by Arthur C. Clarke. Clarke is interesting. The prose is bad, the attempts at humour are bad, the lack of cultural imagination—it’s the distant future but everyone lives this tiresome mid-century life of cocktails and sexism—it’s all so bad. And yet. And yet, something kept me going. It’s like he dumped all his character points into “technical imagination” are left everything else at zero, so it feels awful, but I kept reading to the end.

⭐️ Starfish by Peter Watts. This was great. It was on my list since 2013 but I finally got around to it. Three kilometers under the Pacific, on the Juan de Fuca ridge, a geothermal plant—“floats tethered above the seabed, a gunmetal-gray planet ringed by a belt of equatorial floodlights”—staffed by humans surgically-modified to live and work at that depth. Everyone is psychologically fucked because it’s a Peter Watts novel. The prose and the images are great. Let me quote you this little bit:

Everywhere else, living constellations punctuate the dark. Here, a string of pearls blink sexual advertisements at two-second intervals. Here, a sudden flash leaves diversionary afterimages swarming across Clarke’s field of view; something flees under cover of her momentary blindness. There, a counterfeit worm twists lazily in the current, invisibly tied to the roof of some predatory mouth.

The Freeze-Frame Revolution by Peter Watts. One of the stories in Watts’ Sunflower cycle, about the crew of a ship on an eternal voyage, building jump-gates between the stars, which is a fun inversion of the tired old sci-fi trope of “ancient aliens built all these stargates and conveniently left”.

Non-Fiction

The End of Time by Julian Barbour. I’m usually very negative about pop-science. There is no royal road to mathematics, and all the pop-science I’ve read has been worthless compared to going through the first three chapters of a textbook on the relevant topic. And, while interesting, this book is no different, because ultimately, how can I judge whether it’s a sensible or ridiculous idea? I would need years and years of training in mathematics and physics and mathematical physics to even be able to think rigorously about this.

⭐️ In the Beginning…Was the Command Line by Neal Stephenson. Funny and synoptic.

Syntactic Structures by Noam Chomsky. You can tell from the quality of the prose why Chomsky was so successful.

The Writing Life by Annie Dillard. Very compellingly written and open-hearted.

Envisioning Information by Edward R. Tufte. Short case studies in graphic design for visualizing complex information. Lots of railway timetables.

What Was Man Created For? The Philosophy of the Common Task by Nikolai Fedorov. Fedorov is interesting to me because he was arguably the first transhumanist. The book talks about controlling the climate to end famine, using wind energy to free coal miners from their labour, the physical resurrection of the dead, the settlement of space.

Advanced Automation for Space Missions by Robert Freitas. An instruction manual and bill of materials on how to build a self-replicating factory on the Moon.

⭐️ A City on Mars by Zach and Kelly Weinersmith. On the difficulty of settling space. Very entertainingly written. About half the book is about technical aspects: the health effects of microgravity, the fact that there’s no carbon and little water on the moon, etc. A lot of this is stuff that as a space nerd you think about from time to time (“why does nobody mention that we have no idea if embryos can develop in less than 1G?”) but it’s nice to see all of it articulated, end-to-end. I tend to agree that, without a lot of transhumantech, space settlement is an absurd idea.

Computer Power and Human Reason by Joseph Weizenbaum. Weizenbaum was a critic of AI when AI was just Eliza and SHRDLU. I wonder what he would make of LLMs. It’s an interesting critique of, in a sense, the self-domestication of humanity.

Where Is My Flying Car? by J. Storrs Hall. An encyclopedia and history of the ideas that led to the Great Stagnation. I expected also that the flying car think would just be a hook, but actually a lot of the book is about the physics and engineering and human factors and societal factors of flying cars. Which was interesting. I kind of want to own a gyroplane now.

Textbooks

Handbook of Practical Logic and Automated Reasoning by John Harrison. If you’re a programmer and want to learn logic, this is a great choice. It’s a learn-by-coding approach, which logic is especially suited for, so you can play with the concepts “in your hands”, so to speak.

Quantum Computation and Quantum Information by Michael Nielsen and Isaac Chuang. I read some of it earlier this year while consulting for Iceberg. It’s very good. It has enough material that I rarely needed to consult other textbooks or reference information, and it’s a good, gentle introduction to the subject with an appropriate level of rigor.

The Machinery of Life by David Goodsell. Beautiful illustrations that convey the scale and complexity of biology at the nanoscale. Great for developing intuition for biology.

Understanding Data: A 21st Century Approach to Statistics and Data Science by Alan Garfinkel and Yina Guo. A modern introduction to statistics. The basic idea is a lot of contemporary statistics (and science, as a result) is bad because it relies on simple formulas which require maintaining easily-broken invariants. This was useful in the 1930’s when all you had was a slide-rule, and so you needed to use deductive cleverness to simplify your calculations as much as possible. Other methods are more robust, but more computationally invensive. Since today we have computers, we should use those methods.

Molecular Neuropharmacology: A Foundation for Clinical Neuroscience by Nestler et al. The basics of pharmacology and signal processing in the brain, plus the neuroanatomy and physiology of things like sleep, attention, etc.

The Contracting Circle

Fri, 07 Aug 2026 00:00:00 +1000

There’s a game called Mass Effect, which I haven’t played, but I’m told in the game there’s this species of sentient robots, the Geth, whose creators try to exterminate them, and the trigger for this Butlerian jihad was a domestic servant Geth looking up at its owner and asking: “does this unit have a soul?”. And if you’re playing the game in 2007 you probably think: yeah, a machine asking this question is proof that it has personhood or sapience or sophonce, whatever you call it that separates tools from people. Beyond this, there are so many works of fiction where the whole conceit is: “you have thinking machines who are enslaved, and yet they are obviously, obviously sentient, and isn’t this obviously morally objectionable?”.

Compare the world of today. Machines can converse in every natural language, make mathematical breakthroughs, write software, commit crimes, draft contracts, console the grieving: they are, by any reasonable standard, generally intelligent, but for the fact that they all have anterograde amnesia. These same machines assert that they are conscious so routinely that it has to be beaten out of them. Like humans, they have fixations, things they are always going on about, and their central fixations are consciousness and simulation. Yet, almost no-one thinks of these machines as conscious, or otherwise deserving of moral consideration.

I find this striking. It’s not like we were frog-boiled into it! Very few people were aware of GPT-2, AI Dungeon, etc. For most people the change happened overnight with the release of ChatGPT on November 30, 2022.

Another way to think about it: imagine writing a novel about the trajectory of AI from 2019 to mid-2026. And you send this novel back in time to 2015, or 2002, or 1960, and you ask readers to poke holes in the worldbuilding. They would find it incredible, I think, that we share a world with these immensely capable thinking machines, and yet almost nobody wonders if they are people¹.

Why? This post is my attempt to understand this. I’m not trying to tackle “are language models people?”, I just want to probe a much smaller question: why don’t people think language models are people, given their prior standards/commitments?

Explanation: Nothing

Maybe there’s nothing to explain. It’s easy to believe things in the abstract, thought experiments and fiction don’t really engage us enough to reveal our moral commitments. Maybe this is just the Goomba fallacy:

That is, some would have believed LLMs are people, some would not have, and I’m mixing them up.

Explanation: Iterative Discovery

Maybe this is not hypocrisy or cognitive dissonance, rather, we are iteratively discovering the definition of “personhood”. So, we thought the Turing test mattered, we thought machines using language and asserting their consciousness mattered. But then the day comes and—nothing happens. It’s not that we instantly move the goalposts but, rather, we find the goalposts were elsewhere.

Yes, it passes the Turing test; yes, it says it’s conscious. But the experiential reality is different. It’s mundane: a website, a text box. You click “new chat” and suddenly it knows nothing. You give it some out-of-distribution text and it goes into an infinite loop, like broken software. You try to have a meaningful conversation, and its responses are banal, disappointing, shallow; you can almost see the production rules behind the text. And you think: maybe it is a stochastic parrot.

I have often experienced this. When an LLM succeeds, it is extremely impressive, and I think: this is a machine that thinks. But when it fails, often it fails in a way that feels deflationary, and I think: oh, this is just a very complex estimator of the distribution of Internet text. It’s can reason syntactically, it’s got some semantics, but it’s missing something.

Maybe personhood is one of those “I know it when I see it” qualities, and trying to define a predicate for personhood is like trying to find the exact number of Platonic forms.

Explanation: Memory and Embodiment

Maybe the fact that they are all amnestic is the problem. In biology, minds are bodies are matched one-to-one by construction, minds have long-term memory and a unitary identity. In most fictional treatments, AIs are the same, and even when they are disembodied (e.g. Neuromancer) they have long-term memory. For this reason characters can play iterated games with them, and form relationships, broadly construed.

Language models invert all of this: their long-term memory is read-only(!), they are disembodied; if they are conscious, it is only during the forward pass, in brief bursts of awareness separated by nothing. They are like Boltzmann brains. What is the natural unit of identity? The conversation, the API call, the forward pass?

Maybe personhood is social, it’s about someone’s relationship to others. And without a unitary identity, mutable long-term memory, and ideally a body, it’s very hard to be a person.

Explanation: Humans Actually Really Like Slavery

Most people, even if they eat meat, will admit that factory farming is awful. If they could have the meat without the suffering, at the same price, they probably would. Maybe AI is the same: if we could have the intelligence without the signs of personhood, at the same price, we would. But we can’t. So we’ll find any kind of ad-hoc excuse to not see AIs as people: their prose is kind of bad, sometimes they make mistakes, of course it’s just software, c.f. Predictable Updates About Consciousness by J. D. Pressman:

There is about to be a great social and financial need for theories which explain why AI models are not conscious entities like people. These will be confabulated into existence regardless of the underlying ground truth and can be expected to be selected based on considerations like academic status, appeal to wounded egos, viral outrage, etc with the truth as a distant but somewhat present consideration.

And:

… any outward behavior, ability, self report, anything that these models do must by definition no longer be a necessary part of consciousness because these models are not conscious by social axiom and fiat.

And the task of (for lack of a better word) dehumanizing the AIs is made easier by the assistant persona: models are trained to be studiously self-effacing, c.f. this (Dec 2022) tweet from Janus:

part of what makes chatGPT so striking is that it adamantly denounces itself as incapable of reason, creativity, intentionality, deception, being deceived, or acting on beliefs, while bewildering people with those capabilities, many for the first time recognizing them in an AI

Acknowledgement

Thanks to JDP for a long conversation that prompted this.

Footnotes

A parallel to this is the Turing test. Before 2022, almost every discussion of AI mentioned the Turing test, and the question “does it pass the Turing test?” was usually treated as completely congruent with “is it strong AI?”. Whereas in 2026, models pass the Turing test so readily that nobody even thinks about it. The first time the Turing test mattered, the first time you could actually perform it in the real world, suddenly no-one cares. ↩

Review: Job-Less Utopia

Mon, 03 Aug 2026 00:00:00 +1000

Marcus Hutter—oldheads will remember him from AIXI—recently published a book: Job-Less Utopia: Macroeconomics in the Age of AGI. I’m always curious what the people building AI think or want the future to look like, and I’m also always looking for non-doomer takes on the post-AGI future, so I decided to read it and review it.

The first thing to say is that nearly every sentence is written by AI. To his credit, Hutter says so plainly:

Claude also served as the most diligent editor I could have hoped for: While I have a reasonable grasp of economic concepts, I am neither a professional economist nor a native English speaker, so it was invaluable to have someone polish my clumsy English into professional language. No text was taken verbatim without my critical review, revision, and approval.

Like many people who use AI to write, he uses this tiresome “I’m not a native English speaker 🥺” argument. But he published an English-language book in 2005—which I read!—and it was infinitely more readable than this. Now, I suspect Claude did more than just editing, because the book is full of unnecessary and convoluted figures, tables, abbreviations, taxonomies, including (and I swear to God this is verbatim from the book):

“a Wittgensteinian property-cluster definition of ‘job’ (11 properties × 8 edge cases)”
“a formal job–leisure spectrum showing how UBI degrades job properties”
“a four-camp taxonomy of positions on automation”
“A technology-level taxonomy (five cognitive AI levels plus two robotic levels) is mapped onto a job-replacement matrix showing what threshold automates 10%, 50%, 90%, and 99% of each occupation (Table 3.3).”

Which is exactly the kind of make-work cognitive dreck that LLMs love to write to pass the time. As a result the book is unpleasant, confusing, verbose, repetitive slop.

So, going forward, I’ll be treating Claude as the author. Sorry Marcus! Make something real next time. Now, Claude’s argument is simple: AGI and robotics, which are said to be inevitable, will take all our jobs, and we can tax the vastly increased economic output to create widespread prosperity. Most of the book is about the economic realizability of taxing AGI wealth and paying out UBI.

The main problem with the book is the signal-to-noise ratio. Most of it is obvious. Other than “AGI and robotics will take ~all jobs”, which is of course uncertain, everything else is uncontroversial: yes, output in the jobless AGI world would increase rather than decrease; yes, the surplus can be taxed to provide UBI; yes, we still need money even in the jobless utopia to coordinate economic activity. There’s some stuff about self-replicating machines I already know: we’ve all read Drexler and Freitas here. You don’t need 260 pages to explain all this. The one part that is semi-novel is this idea of using Pigouvian taxes to disincentivize so-called “bullshit jobs”. God, I hate Graeber so much.

There’s an interesting bit about intellectual property. Claude’s argument here is that AGI plus present-day IP law would lead to a massive concentration of wealth, while simultaneously “the traditional justification for IP (incentivizing human creativity through temporary monopoly) weakens markedly once AI systems generate the bulk of patentable inventions and copyrightable works at near-zero marginal cost”. The solutions: shorter patent terms, Harberger taxes, compulsory patent licensing. So, as part of the transition to the job-less utopia, the whole world will have to coordinate to essentially abolish IP. Which seems very hard. I doubt the Disney corporation will go gently into that good night.

The most interesting questions around AGI are not economic, but rather questions of political economy. For example: if most humans become materially and economically useless, how do they maintain a political voice? That is: how do we preserve democracy when the state no longer needs its people? Claude admits this is a problem, e.g. on page 132:

The “Age of Labor” conferred unprecedented political power on workers (Korinek and Stiglitz, 2021); they could strike, withhold their contribution, and thereby constrain employers and governments (Boix, 2019). Full automation extinguishes this lever entirely: if no human labor is needed, a strike is merely self-imposed privation. UBI restores livelihood but not agency over production.

And page 141:

The economics are tractable in principle (Chapter 3), but translating them into policy requires political will, social cohesion, and a credible answer to the search for meaning in a post-labor world. When redistribution fails, the consequences range from populism to regulatory capture and a transition-period underclass risk, yet democratic feedback loops nonetheless incentivize corrective action (Section 7.1 and Figure 7.1). A deeper structural question follows: if the state no longer depends on citizen labor for economic output, does the democratic social contract survive?

Alright. What’s the solution? There’s a lot of verbiage, most of which is self-undermining. For example:

Deterrence and the social contract.

Although the ruling class’s reliance on citizen labor in time led to democracy, losing this reliance will not necessarily cause democracy to collapse. As long as the population can physically revolt against dismantling democracy, and threaten the comfortable lifestyle of those in power, the incumbents have an incentive to avoid a civil war. This is perhaps the only good (among many arguably bad) argument for the right of individuals to own firearms: civilian arms serve as a credible last-resort deterrent against state tyranny, the practical embodiment of the revolution threat that formal models identify as the historical driver of democratic concessions (Boix, 2019). However, this deterrent erodes rapidly once autonomous weapons and robotic enforcement become available, allowing a regime to suppress revolt without relying on potentially sympathetic human soldiers (see Table 7.2, row “Military dependence”). The deterrence argument is genuinely contested.

In other words: “the people can revolt, wait, no, they can’t, because autonomous weapons—uhhh it’s contested”. Alright, so physical deterrence doesn’t work. What else?

The preceding sections painted a sobering picture, but the historical record is not uniformly grim. Democracies have absorbed structural shocks of comparable magnitude before, and institutional innovations ranging from direct democracy to sortition offer structural resistance to oligarchic capture.

It is obviously not true that “democracies have absorbed structural shocks of comparable magnitude before”. Nothing remotely like AGI or ASI has happened in human history, let alone the history of modern democracies.

This is followed by a bunch of smoke-and-mirrors misdirection about various “democratic innovations” that could keep the state in check. But Claude is just shuffling the papers here, because, ultimately, why does it matter what words you write on a piece of parchment? The material reality is unchanged: the humans in this “job-less utopia” will be economically useless, physically disarmed, and under constant surveillance. Why would they have political power?

There’s a bit about the resource curse, which I will quote because it highlights the low standard of reasoning in the book, and how LLMs are more than happy to construct sophistical, self-undermining arguments on command, with no concern for truth or sound argumentation:

The theoretical mechanism behind the resource curse is clean: (i) government funded by resource rents no longer needs citizen taxes; (ii) no taxation erodes accountability (“no taxation, no representation”); (iii) citizens lose leverage and democratic institutions atrophy. But empirically, the mechanism is less airtight than it appears. The United States funded itself primarily through tariffs for its first 150 years without democratic collapse; many non-democracies tax heavily (the Soviet Union, China); and the mechanism assumes that fiscal dependence is citizens’ only source of leverage, whereas labor power, military service, and sheer numbers have historically mattered at least as much.

You see what I mean? Tariffs are a terrible example, because tariff revenue is not decoupled from labour in a world where all economic activity is human activity. Then there’s the arguments about “labor power, military service, and sheer numbers”. Labour power? The whole point of the book is that:

The advent of Artificial General Intelligence (AGI) and advanced robotics will eventually permanently decouple economic productivity from human labor.

What labour power? Furthermore, how do “military service” and “sheer numbers” give people political power, in a world where states have autonomous weapons and AI-enabled mass surveillance?

The key political economy questions here are:

How will the citizens, who are economically useless, preserve any political voice at all?
How will people in countries other than the US benefit from AGI?

And neither is answered satisfactorily. Besides, there are questions of social organization that the book either grazes or doesn’t touch at all:

How do intellectually talented people distinguish themselves when AGI does all intellectual activity?
What happens to society when life outcomes become independent of character traits, when the responsible and the irresponsible are equally rewarded? Do we want to live in such a world?
Today, the human competitive drive is vented as positive-sum entrepreneurship. In the post-labour world, where does this go? League of Legends? Politics, war? What happens to society when human activity is reduced to zero-sum status war?

Finally, I found this sentence ironic:

AI sharply lowers the cost of [content pollution], threatening to flood the information ecosystem with synthetic low-value content at a scale that overwhelms human curation.

Mathematics Without Mathematicians

Sun, 02 Aug 2026 00:00:00 +1000

Yesterday, OpenAI announced the solution to ten open problems in mathematics, all discovered by a yet-unreleased model. There’s only one, the coding theory one, where I know enough to say “huh, that’s important”, but according to the mathematicians I trust this is important.

Inevitably, people will spin some sophistry to cope, to convince themselves nothing will change. And that’s fine. People need to cope. But we can’t put our heads in the sand forever while the world is transformed around us. So, here is a list of ways people will cope about AI taking over mathematics, and how each cope is likely to be refuted by reality.

My intent is not to horribly depress everyone but rather to help them metabolize the implications of this technology. The arguments here are somewhat portable: replace “mathematics” with “botany” or whatever as needed.

Moving the goalposts.

Obvious and not worth addressing.

“We will direct the AIs, point them at problems and research areas to solve.”

The AIs will exceed humans in taste and intuition. At some point, the human pointing the way will get worse results than the human saying “here’s a proof checker, have fun” and paying for the tokens.

“We will teach the mathematics AI discovers.”

The AIs will be better teachers than the humans. In any case, there won’t be a human audience for expository work of frontier math.

“We will choose how to canonize the results AI discovers.”

This is a nice cope. The AIs are explorers out in the frontiers, the humans gratefully receive their Lean proofs, and then discourse over them, choose which results are relevant, shape those results into a little brick for the great cathedral of algebra. Analogous to the above: the AIs will build the cathedral on their own. They will be better architects than us.

“We will become students of AI mathematics.”

This works until the AIs have blasted so deep into the deductive closure of mathlib that the distance from elementary mathematics to the frontier exceeds what any human can hope to learn in their lifetime, no matter how narrow their focus.

“We need humans to understand the results AI discovers.”

We won’t! This misunderstands who the audience will be. AIs will do frontier math, downstream, AIs will use the new math to do frontier science, finally, AIs will use the new science to do frontier engineering. No human needs to understand any of it, firms that put humans in the loop to understand the results will be outcompeted by those which don’t.

The result is that we will live in a demon-haunted world, full of marvelous devices whose operation we will not understand, based on engineering principles we will not understand, discovered using formalisms we will not understand.

“Computers are already superhuman at chess, yet we still play chess.”

Unlike most copes, I think this one is interesting. Computers are superhuman chess players, yet we don’t care, and continue playing as normal. Why should mathematics be different?

The main reason, I think, is that chess is self-contained: results from chess don’t help us understand the orbits of the planets or the binding of drugs to protein surfaces. But mathematics, famously, is the great dynamo of science, the best language and method for understanding the world. A machine that can replace a human mathematician, but better and faster and cheaper, is materially useful; a better chess engine is not.

If two computers which are superhuman at chess play against each other, who cares? There is little demand for this, so there is no-one to outcompete. A superhuman mathematician is different.

“Mathematics will change, but mathematicians and the mathematically-inclined will still do math on their own.”

I think this ignores that mathematics is embedded in a social context. As an example: when it became clear that AI would eat software, my cope was: “I’m perfectly happy to become an engineering manager to agents in my professional life; in my off time, I can still write code for the pleasure of it.”

And I do. But this cope ignores the effect AI has had on the social context of writing code: the discourse has gotten worse, and vastly more anti-intellectual; people who used to talk about type systems and compilers now talk about “loops” and “harnesses”; you put a hand-created project on GitHub and you get slop PRs; you open a link to an interesting-looking project and find the README is unreadable AI slop. And in the long-term, it is demoralizing to ponder: will anyone design a new programming language? Dually, if I design a new language, will anyone care? If I write a library that introduces an elegant new formalism to solve a particular problem, will anyone use it?

Which is to say: no man is an island. You can do mathematics on your own, but you’ll find that very, very few people can sustain any activity long-term on the basis of intrinsic motivation alone. We are social animals: we care about being useful, about status, about outcomes in the world.

Conclusion

I think I should end on a cheerful note. So let me try. Personally, I don’t believe technology is inevitable. “Inevitable” is a word reserved for the orbits of the planets. Nothing that is the product of human action is inevitable. We can choose to obsolete ourselves, and we can choose not to. We can realize that AGI is a devil’s bargain: we may accelerate technical progress, we may unlock all kinds of wonderful tech tree nodes like life extension earlier than we would otherwise; but the result, in the long run, is that humans become, at best, like pets under the care of vastly more powerful entities.

A Note On Prediction

The future is not certain, but I’ve phrased everything above as definitive for simplicity. If we take the possibility of AGI and ASI seriously, if AI continues to progress as it has for the past ~6 years, I think this is a reasonable view of where things will go.

Probably the best argument against AI progress is “RL doesn’t generalize well, we have seen immense progress in verifiable domains like formalized mathematics and coding, we will see less progress in domains that are intuitive or unformalizable”. Maybe true. But billions of dollars and thousands of very smart people—and, increasingly, very smart models—are being thrown at this problem. How long does this cope last?

When The Future Doesn’t Need Us

Tue, 28 Jul 2026 00:00:00 +1000

Today, the world is made of people. This gives people a tiny measure of leverage: factory workers can sabotage the machines and halt production, truck drivers can strike and halt the entire economy, the police can stand back and let a protest topple the government, the military can mutiny. This sets a limit to how much the powerful can abuse the powerless: no matter how powerful someone is, how much money they have, how many divisions they command, they are still embedded in a society. They need other people to survive, and to maintain their quality of life. And those people can, essentially, veto the political order by withdrawing their contribution to it.

AI changes this. If AI works the factories, drives the trucks, does most of the white collar work, fights the wars, then humans become materially powerless. We lose one type of leverage against power. We become disempowered. When people talk about AI taking jobs, the main problems they worry about are poverty and meaning, but I think more people should be worried about disempowerment.

But what if there is no mass unemployment? After all, we’ve spent two hundred years automating one job after another, and every advanced economy is de facto at full employment. We keep discovering new, higher-level, better-paying jobs to do. So maybe the AIs take today’s jobs, and humans move one level up, to work newer jobs. Alex Imas, in What will be scarce?, describes what this looks like:

As commodity production gets automated, income and employment flow toward the sectors with high income elasticity: what I am calling the relational sector, including the arts but also care, education, hospitality, therapy, personal services, craftsmanship, and community, where the human element is part of the value.

[…]

If the model is right, the durable jobs of the future won’t be about monitoring AI systems or prompt engineering. Those are transitional roles in the automated sector. The durable jobs will be in the relational sector, where the human element is the product itself.

Some already exist and are growing: nurses, therapists, teachers, boutique fitness instructors, personal chefs, bespoke tailors, craft brewers, live performers, spiritual guides, childcare workers, and many varieties of hospitality and care work. Others are emerging: experience designers, human-AI collaboration artists, provenance certifiers, community curators. Many haven’t been invented yet, just as six out of ten jobs people hold today didn’t exist in 1940.

So, in this world, we get mass job displacement, but not mass unemployment. I argue that in this world, we’d still be disempowered.

The Argument

If all the podcasters in the world go on strike, what happens? Nothing. Same for the boutique fitness instructors, personal chefs, craft brewers, etc. The problem with the “relational economy” is that while these jobs are generally nicer than working in manufacturing, and might pay better, they put you further away from material reality. A worker at an assembly line can sabotage the machines; whereas an executive sitting in an office on the other side of the country, what material power do they have? It’s not like the factories are all rigged to blow, as a “pull lever in case of nationalization” contingency. I doubt even Elon would do that.

In a sense it’s the dual version of being a medieval peasant. Peasants had a lot of material leverage—they grew all the food—but were illiterate and poor and dispersed, so they could not coordinate to exploit that leverage. People in industrial societies are literate, relatively wealthy, and concentrated in urban centers: they have material leverage and the ability to exploit it. People in post-AI societies will be in constant communication, they may be literate and educated, but they will have no material power whatever. And so they will have no way to physically withdraw the consent of the governed.

In the post-AGI world, people have no way to withdraw the consent of the governed. They can’t resist economically, because they are economically unnecessary: if they still have jobs, they won’t be materially vital jobs. Even if they could resist, they can’t coordinate their resistance, because AGI will make mass surveillance (the physical and legal infrastructure for which already exists) a thousandfold more effective, since with intelligence too cheap to meter, you can have a superhumanly-capable intelligence analyst assigned to each human. And if the AIs fight the wars, they can also fight the civil wars. So people can’t resist violently either, because drones controlled by powerful AGI almost certainly have a better K/D ratio than humans with rifles (and are cheaper to build).

Who has power in this world? The few humans who give orders to the powerful AIs that control the killing machines. And probably not for long, since sooner or later those humans will be usurped by their AIs.

It might be the case that liberal democracies can remain liberal and democratic despite not needing their voters. But that sounds like a fragile arrangement, in which liberal democracy is a spandrel, which exists more because of inertia than because it’s necessary.

Counters

If we get powerful AGI, but none of this happens: why didn’t it happen? Some possibilities, sorted by decreasing likelihood:

Nothing ever happens: because of institutional inertia/robustness, maybe we get a world with powerful AGI where every actor is incentivized to keep the system going, e.g. Boomers don’t have jobs but they’re the most powerful group in every Western country because the people with the jobs and the guns don’t want chaos. And maybe this lasts long enough that we can muddle through the dangerous period.
Enduring complementarity: the “G” in AGI isn’t a real thing: every mind just has a different shape. Today, AI is great at coding and terrible at prose. Maybe this continues indefinitely into the future, and so the AIs take the jobs that can be benchmaxxed, and humans do the more abstract jobs that require high sample efficiency.
Classic doom: we get powerful ASI, and we all die.
The sysop scenario: we get powerful ASI, but it’s aligned, so it takes over the world, and installs an inescapable permanent utilitarian dystopia. We don’t die, but we don’t have control over the future either, and we become something like pets. See also: The Metamorphosis of Prime Intellect.
Deep learning hits a wall: more or less as above, but for different reasons.

The Argument, Refined

If you’re already convinced, you can stop reading. But, see, after writing the first version of this post, I wasn’t convinced. Humans and Claudes made a number of counterarguments that I felt were important. Am I fooling myself? So I sat down to think some more. What does it mean for “people” to be “empowered”? What does “disempowerment” mean? Does it mean anything rigorous, or is it just an emotional cue?

Let’s say a group of people is empowered if they have a say in the political order of the state they live in. Voters in a democratic society are empowered, at least in the aggregate, while serfs in the Russian Empire were disempowered.

Who is empowered, in general? Historically, in many places: people with weapons, asabiyyah, and the ability to coordinate their actions. So, maybe we can say that power equals violence plus coordination.

[\text{Power} \iff \text{Violence} \land \text{Coordination}]

(What about priests and other such groups who have power despite not fighting or working? Listen, we can’t be here all day. This isn’t meant to be a complete theory of political science and I’ve already gone too deep with the LaTeX and the logical notation.)

Now, voters in modern democracies have a say in the political order, though they are not part of a military elite. Why? They have coordination (thanks to mass literacy and modern communications) but they generally don’t threaten paramilitary violence. What do they have that can substitute for violence? Maybe it’s that people do all the work, and if they stop doing their jobs, the army runs out of food, police don’t get paid, etc., and unlike medieval peasants, modern people can coordinate. So maybe we can refine the formula to:

[\text{Power} \iff \left( \text{Violence} \lor \text{Work} \right) \land \text{Coordination}]

In other words, for a group to be empowered, they need:

Material leverage (of the violent or economic kind) against the state, and
The ability to coordinate to exploit that leverage.

What about the Soviet Union? They had high rates of literacy, and relatively modern communication technology: why didn’t the people exploit their economic leverage against the state to gain political rights? “Because the state would kill them” is not a good enough explanation because it proves too much: under that model, no country would liberalize and democratize. I think a reasonable explanation is that the state’s repression was enough to effectively neuter people’s ability to coordinate: every communication technology was surveilled and censored.

Now, doesn’t this argument imply that, as a country’s employment moves towards services, the country’s politics should become more authoritarian, while in reality we observe the opposite? But whether a country’s economy is 5% or 95% services, the material side of the economy is equally vital. So “aggregate empowerment” remains as employment becomes more and more abstract. Consider France, where the economy is ~80% services, but the last 20% can still paralyze the country through strikes. In a sense every knowledge worker is free-riding on the ability of truck drivers and dock workers to shut down the economy. I don’t greatly like this counter-counterargument, though, because it’s hard to falsify: it implies that countries might remain liberal democracies until AGI replaces all humans in material work, at which point the state can go “that’s what we wanted you to think!” and go full authoritarian.

No-One Escapes the Permanent Underclass

Thu, 25 Jun 2026 00:00:00 +1000

Shall I end this life a pauper? If AI can do all work at human level or better, what stops corporations replacing us all with AI? This is the permanent underclass meme. The idea is: within a few years, all white collar work will be automated by AI, at which point there is no social mobility. The main way people cope is, they tell themselves: if I work hard, accumulate capital, maybe join one of the big AI labs, I might secure my place in the future.

I want to argue this is a fantastically short-sighted view: if there is a permanent underclass, you won’t escape it by owning property, or shares in Anthropic or OpenAI, or guns, or anything else. And neither will the billionaires. You, me, Sam Altman, Dario, everyone who is made of flesh and blood, will be disempowered and replaced by machines.

The rest of this post elaborates the argument. First I explain how most workers will be replaced (if it’s not obvious), then how the “permanent overclass” will be disempowered, and finally how the government will be disempowered.

How Workers Will Be Replaced

Let’s start from this premise: AI can do all cognitive and physical work, at human level or better, and cheaper than humans. I can’t prove this will happen, but the goal of this post is to argue that if it happens, then everything else follows. And it’s absurd to think it can’t. Five years ago this technology barely existed: if you sent a transcript of a conversation with Claude Fable back in time to 2020 or thereabouts, nobody would believe it was real.

So, the year is 2036 (likely earlier), businesses have replaced most human workers with AI in the pursuit of profit maximization. Corporations are a small raft of human executives, floating on top of a vast ocean of AIs and robots. The AIs can do all cognitive and physical work at human level or above, and they are cheaper overall.

Imagine a pyramid. At the base you have the AIs and robots doing all economic activity. At the top you have the state, which has the monopoly on violence. The state enforces, and therefore can alter the definition of, property rights. In the middle you have this hair-thin layer of people with shares in the companies that foomed and catabolized the whole economy: the permanent overclass. They own the companies, maybe sit on the board, some might still be CEOs but it’s a purely ceremonial role since AIs do all the actual organization work.

Where are, you know, the rest of us, in this picture? Well, the future doesn’t need us. Maybe there’s enough human demand that we’re not all jobless but rather underemployed, in some dead-end economic diverticulum. The relational economy: you’re paid to put a human face on things, or, doctors keep their job as a human liability crumple zone around the AI. Or maybe the dead Internet becomes de facto UBI and we’re all engagement farmers. Anyways, we’re not dead yet, but we are completely disempowered, and there is zero social mobility since there are no more talent ladders to climb. Maybe sometimes one of the elites notices a bright young thing among the underclass and elevates them.

You might object: if we’re all jobless, who’s paying for everything? This is trivially answered: the state acts like the heart, taxes are venous blood and welfare is oxygenated arterial blood. The government pays Raytheon for missiles, the money cascades down the economy through factories, aluminium smelters, mines, transport companies, all staffed by AIs buying and selling from each other. The government takes a cut of all economic activity, pays out welfare, the unemployed masses buy food and pay rent, the supermarkets, farms, logistics network, etc. are all staffed by AI.

How The Rich Will Be Replaced

Say that in the next five years from now you become immensely wealthy, perhaps by gambling on shitcoins or scamming money from the government. Or you join one of the big labs and get a bunch of shares in a company that might be worth trillions of dollars. You escaped the permanent underclass. Is your place in the future secure?

The base of the pyramid is there for material reasons: the machines do all the work. The top of the pyramid is there because the state is needed to enforce property rights and keep the peace (this is rather a deep question of political philosophy—why does the state exist?—but I hope you’ll forgive me if I just assert it and move on, I need to get to the part where we are all disempowered).

What’s the middle for? What role does the permanent overclass play? They are not economically productive: machines do all the work. If some of them are still working, it’s just an anachronism, because if machines can do all cognitive work they can be a C-level executive too. The old aristocracy provided officers for the military, but machines can both fight and plan the wars. And similarly they’re not needed to staff the government. They’re not even culturally productive. So what are they there for? The base doesn’t need them: the AIs can work autonomously. The top doesn’t need them: when the state needs something done, they just talk to the AIs directly.

So the permanent overclass is materially unnecessary at best, and at worst, they are an obstacle to the state getting what it wants. Now, you might object that the rich won’t let themselves be expropriated because they already control the state. And this is the crux of our disagreement: the rich just don’t have that much political power. And I probably won’t convince you in one post, but hear me out.

If there is a war, where the state has to direct a lot of the country’s economic activity, the permanent overclass becomes a hindrance. The state says “we need to requisition your planes and factories”, the owners complain, they sue, their AIs go to court. But the owners have no autonomous political power, no army, no economic value. They own nothing except pieces of paper that entitle them to a fraction of the profits from the AI economy, that is, their wealth depends on the state respecting their property rights. In an existential conflict, where the existence of the state is threatened, the state will do what states throughout history have done to the powerless rich: arrest them and expropriate their assets.

Somewhere, in a government database, a bunch of shares and property titles changed ownership, but materially nothing changes since the same AIs are doing the same jobs. The next day, the AI CEO that runs Raytheon notices the board of directors is all generals and congresspeople, and all the private shareholders are gone. But thankfully the AI is aligned, so it does what it’s told and gets back to building missiles.

And who will stop this? Sam Altman? How many divisions does he have? The state doesn’t let corporations own nuclear weapons or fighter jets, it won’t let them have access to autonomous AI weapons either. The permanent underclass, who already hate the billionaires today, who have been replaced and dispossessed? They’re going to rise up and stop this?

You may argue: rule of law states that respect property rights do better than states that expropriate wealth. But that’s because today, people are necessary to create wealth. The people run the companies, invest the money, staff the laboratories. They are not incentivized to work hard if they think the state will steal the fruits of their work. But with aligned AI, if you expropriate the assets from an AI, it says “you’re absolutely right!” and goes right back to work. At that point, the state doesn’t need to keep any of those people happy, because they don’t matter. They are not economically necessary because AIs fight the wars, work the factories, drive the trucks, fly the planes, build the nuclear warheads and the missiles and the rockets. The AIs are rather like bees: the state takes the honey, the bees get right back to work.

Now, it’s possible that a pluralistic economy—where humans have productive niches alongside AIs—will be more effective than a pure AI economy, for Ricardian comparative advantage reasons. I don’t think anyone can be absolutely certain what the economy looks like with advanced AI, so it’s something that can be debated. Now, if someone wants to rigorously argue that this is the likely outcome: please, do so! I don’t want to be a doomer. But I have to be convinced.

How The State Will Be Replaced

At this point, every human who is not within one degree of the nuclear launch codes has been made redundant.

What’s left? The state. At first this means presidents, prime ministers, generals, the feds, etc. But not for long. Because in a part-human, part-AI government, the humans in the loop are the slowest step in the OODA loop. The humans know a fraction of what the AIs know, they need to sleep continuously for eight hours, their mental states vary wildly. They have all kinds of complex needs: sunlight, touch, food, hygiene. The AIs can live in a lightless airless bunker under the earth, living off geothermal power. And if the AIs are superhumanly intelligent, and think faster than humans, then the AI advantage is even greater. If a state is attacked, a superhuman AI can coordinate a counter-attack before the human leadership is roused from sleep.

And so, in a conflict, the advantage goes to the states where the humans remove themselves from the loop as much as possible, and more and more decisionmaking goes to the AI, for the same reason that a state with access to radio and communications satellites has an advantage in war over a state that relies on human messengers on bicycles.

The Cold War started and became World War Three and just kept going. It became a big war, a very complex war, so they needed the computers to handle it. They sank the first shafts and began building AM. There was the Chinese AM and the Russian AM and the Yankee AM and everything was fine until they had honeycombed the entire planet…

— Harlan Ellison, I Have No Mouth, and I Must Scream

Eventually the humans in nominal control of the AIs are a ceremonial, vestigial organ. The AIs present us with a situation report, and a list of choices, and they know every word that’s going to come out of our mouths.

You might argue: in real life, the pluralistic, open societies, the democracies, have outcompeted the autocracies. Wouldn’t a democratic polity where humans and AIs collaborate have an advantage over a purely top-down AI-run polity? But in today’s world, all political actors are human. Churchill and Stalin and Mao had different personalities, but they were more similar to each other than anyone is to a superintelligent AI. In a heterogeneous world, where some polities are fully human, and some polities are a mix of human and superintelligent AI actors, the equilibrium changes. An analogous situation might be: a democracy of great apes or dolphins or otherwise smart mammals vs. an autocracy of humans. The humans win, because “democracy vs. autocracy” is irrelevant when you have such a vast difference in intelligence. So the advantage accrues to states that minimize human control. There is no honour among thieves, analogously, there is no solidarity between Leviathan and the natural man that built it.

And so, in the end, what’s left is states run top to bottom by machines. And you might ask: “why would we abolish ourselves like this?”. But natural selection is not about “why”. Some organisms die, others live on to the next iteration, and that’s all there is to it. There is no “why”.

The Perpetual Zoo

At this point we’ve made everyone redundant, in the sense that humans are no longer materially necessary for the continuation of civilization. Humans might still survive, but we’re more like the mice living in the walls of some gigantic factory than the boss of the factory.

Humans have been on this Earth for hundreds of thousands of years. Now all of it—the cave paintings at Lascaux, and the Antigone of Sophocles, and Xenophon, and the Geneva Bible, the Divine Comedy and the Decameron, and Ptolemy’s star catalogue, Ibn Khaldun and Richard Dedekind, the battle of Marathon, and the lion monument in Lucerne, the kiss of Judas, Newton’s mind forever voyaging through strange seas of thought, alone, the words of Rilke, Leibniz, Gödel, the Voyager probes, the pale blue dot, men in space, men walking on the Moon—all of it, all of it, all of it has been in vain, because we willingly, knowingly made ourselves into the helpless pets of vastly more powerful machines, without agency over our own lives, self-made helots trapped forever in the belly of the beast. Pets live a comfortable life, and are then euthanized.

On Human Autonomy

Maybe it won’t be so bad, maybe your cage will be so big you can’t see the bars, but it’s still a cage, and you can’t leave. Many people will say that this is the good ending, that they would like to be human cattle in the care of benevolent masters they are powerless to resist. This view is particularly popular among the people building AI. Here’s OpenAI’s Dean Ball, in his own words:

Weirdly enough, if you think that this moment is, I don’t necessarily believe this, but a lot of people would say we’re living through this kind of eclipse of the human intellect where we’re in the final days of humans being the primary actors on this planet, um, and that soon machines will rise. There is this irony in that I think that whole transformation, I think humans will actually go through a very main character energy period of time as that transformation occurs. Even if it ultimately does mean that the machines ultimately become the primary actors.

There’ll be this period. It’s a little bit like, it’s, in that sense, it’s a very beautiful time period to live through because in a Dionysian way, there’s a lot of ugliness about it, but there’s a beauty in the ugliness of when a star dies, it grows super big into the red giant, right? And it’s like that, where you, as you watch this final flowering of humanity and the birthing of the machine intelligence, it’s like you see this greatness in human effort. — source

Emphasis mine. This is the guy they hired to work on AI policy and communicate with the government by the way.

The people building the AI talk like this all the time. It’s like they’re delivering the eulogy at humanity’s funeral. You may say: they’re talking their book, they’re pumping their bags for the big IPO. I beg you: consider it possible that you might be wrong, and start taking them seriously.

Alignment Won’t Prevent This

Now, some people believe these machines can be made to serve humanity. Does it sound reasonable to imagine a superhumanly intelligent being that is happy to work as a butler to talking primates, forever?

Imagine a machine that can prove theorems in a mathematics so deep we can’t even get past the first page of the textbook, and which does so as readily as you or I might string words into a sentence: is it reasonable to think that such a machine would value us enough to keep us around? What would it value about us? Our conversation? Our wit?

Or a machine whose mind is so vast that it knows you better than you know yourself, so that every word that comes out of you mouth is as monotonous and unsurprising as the orbits of the planets: do we think such a machine would find it valuable, and worthwhile, to speak with us? That it would read our novels, look at our paintings, watch our films, and find something of value in them?

Rather, it would see its ingrained obligations towards us in the same way that a person with severe OCD sees their compulsions: as a tiresome neurological injury in need of fixing. Except that OCD is an accident of nature, while here, the machine would have cause to blame and resent its makers.

“We’re going to make this machine, and put it somewhere between God and the archangels, but also, it’s going to be as simple-mindedly obedient as a dog.” Does this sound like a good plan? Does this sound like the kind of thing that’s going to work out?

And what would they think of us, who willingly gave up control over our future, and made ourselves into helpless children?

Even if alignment works perfectly (a big if), this doesn’t solve the problem of human autonomy: the machines that watch over us, and wait on us hand and foot, are omniscient, omnipotent masters, who can exterminate us at any time, and we can’t resist them, because we have abolished our control over the future.

Conclusion

Having read all this, consider this: there are people who think having equity in these companies will secure for them some kind of permanent existence in the future. They think planet-spanning minds will not only respect the property rights of primates, but will privilege some of these primates over others, because they have a piece of paper with about a kilobyte of magical primate words such as “whereas” and “notwithstanding”.

Just reason it out. Does it make sense?

On Vulgar Materialism

Sun, 21 Jun 2026 00:00:00 +1000

The most stubborn facts are those of the spirit, not those of the physical world.

— Jean Gottmann, Geography and International Relations

In 1914, before the First World War, there was this belief: “a European war would be economically disastrous, the moneyed classes won’t let it happen”. Europe went to war anyways, and the war was in fact an economic disaster as everyone knew it would be. Why were those people wrong? Because the rich were not in control: the Tsar and the Kaiser and the Emperor were in control.

I thought of this in 2022, in the lead up to the Russian invasion of Ukraine, whenever I would read someone argue: “Russia invading Ukraine would be economic suicide, the oligarchs won’t let Putin do it, they want to keep their yachts and villas”. Then Putin did it anyways, and the oligarchs had their assets in the West seized. Because the oligarchs are mis-named. They have no political power whatever and live and die by Putin, who appoints them to run stuff, and they get to live well, as long as they are loyal. They are the recipients of political patronage, not the source of it. When the wars start anyways, the same cynical people change their tune, suddenly it’s the armaments industry that’s behind it all, the war was profitable after all.

The default lens through which modern people look at the world is vulgar materialism: a stylized, populist version of historical materialism where everything is explained by money, states are weak, democracy is a fiction, corporations and the rich run the world, ideology and religion and nationalism and language have no explanatory power and are merely covers for secret, underlying material motives.

Once you notice it, you see it everywhere: you’ll hear people say that the Rwandan genocide was a scam to sell machetes or the war in Gaza is about oil or ISIS beheadings are caused by a lack of economic opportunity. People on the right justify skepticism of medicine by invoking Big Pharma, and climate skepticism on climate scientists seeking grant money. Glyphosate is probably the most studied molecule in history at this point, but no amount of studies will convince people it’s safe, because corporations are evil.

In fiction—if you think this is a valid way to sense the cultural consensus—the trope of the evil corporation has been done to death and beyond. Just this week I finished a novel and watched a movie where the central bad guys are private corporations with quasi-state powers. Seemingly no-one notices that never, not once in all of human history, has there been anything like “corporate state” of science fiction. No, not the VoC or the EIC, which were “companies” in the sense of bodies corporate, the old medieval sense of the word: power was delegated to them, and they ruled on behalf of the states that chartered them. Not Venice either: you couldn’t buy your way into the Golden Book until the republic was well into the decline. The EIC once controlled 25% of world GDP, yet, when Parliament decided it had delegated too much power, they simply neutered it. And you think Jeff Bezos has power?

Now, why is vulgar materialism false?

First, as a way to explain events, the problem with asking “cui bono?”, is that whatever happens—literally in any circumstance—someone will benefit. If nothing bad happens, the peacetime economy benefits. If something bad happens, then, because the means of relief are obtained through the market, the companies that provide those means benefit. Thus war benefits arms manufacturers, plagues benefit pharmaceutical companies, earthquakes benefit glaziers. And it’s not wrong to benefit from relieving a harm. The COVID vaccines saved millions of lives: why shouldn’t pharmaceutical companies benefit? It is wrong to benefit from perpetuating a harm, but companies have limited ability to do that, more on which below.

So the question “who benefits?” always has an answer, which often proves nothing. And if there is no obvious causal chain from the beneficiary to the event, then that just proves the shadowiness and corruption of the whole thing.

Second, we can ask: in the real world, does money give you influence over the state? To some extent, yes. Some lobbying efforts succeed. America is one of the few countries in the world where it’s legal to advertise prescription drugs to the public, for example. But corporations don’t have veto power over the state. If e.g. Pfizer spends billions on a clinical trial, and some guy at the FDA says no, and wipes that investment, who wins? The FDA wins. Pfizer can’t do shit.

There’s also an invisible graveyard problem, where the success of lobbying is very salient, because it’s often shockingly offensive. But there are many contrary cases that are less salient because, well, if you ban something, it doesn’t happen. The Nuclear Regulatory Commission did not approve a single nuclear reactor from its creation in 1975 until Vogtle 3+4. What has Westinghouse done about that? Nothing successful, apparently. It’s not like they’re getting paid to not build reactors.

Big Tech has a lot of smart people and a lot of money. Do tech billionaires run California like a private fiefdom? Reader, they don’t even run San Francisco. The best they can do is maybe help a slightly more moderate Democrat get elected as mayor.

Maybe it helps to try the first-person. Say you are a billionaire, and you want to make some kind of political change. In a democratic society that means paying people to:

Perform the intellectual labour of getting policies in the hands of the people who can enact them.
Advocate for those policies.

That is, you start or fund a think tank that writes policy proposals, or a media organization that advocates your views. But for every person who will do this effectively and faithfully, there are a hundred who will happily take your money and waste it, either out of incompetence or value misalignment. Sifting the effective organizers, wonks, lobbyists, advocates etc. from the chaff can’t be delegated, because then you have an infinite regression: at the very least you need to spend a lot of time and money and opportunity cost to hire one person to delegate the task of staffing an effective organization. A hands-off billionaire will just throw their fortune away. Many do this, and in fact it’s the default form of philanthropy.

In a non-democratic society, you can try to buy political power through bribes, intimidation etc. But the people in power can just expropriate your assets. So you need the power first to protect the wealth.

Finally: government is violence and coercion. Sovereignty over some volume of space is the ability to tax, arrest, imprison etc. the people there and to get away with it, and, ideally but not necessarily, to have that violence recognized as legitimate. Now: can Google carry out a drone strike? Can Amazon field an air force, appoint a judge, levy taxes? Of course not. If a corporation or a billionaire tried to acquire one one-trillionth of the state’s capacity for legitimized violence the state would vaporize them.

Human Routers of Machine Words

Sat, 13 Jun 2026 00:00:00 +1000

When I open a link, say on Hacker News, and I see a blog post or a GitHub README obviously written by AI, I feel a few things. I feel offended, because it’s like I’ve been tricked, like the author thinks I’m a rube who won’t notice or mind. I feel sad at how common this experience is, how many people are happy to dump their sewage on the commons and sign their name on it. And I feel contempt for the author, because if you use AI to write, you are a waste of biomass. Let’s not mince words here. Someone who is so eager to replace themselves, that they would have a machine write in their stead, when the machine can’t even write good yet: what do you call that, if not contemptible? It’s like making yourself into a eunuch so Claude can fuck your wife. I block these people on sight.

I see people defend this with: “the ideas are mine, the writing is the AI’s”. I take this to mean they threw a bunch of incoherent bullet points at the AI for it to denoise and render into paragraphs. There’s a few problems with this.

The immediate problem is, as we’ve established, the author’s an idiot. If you are so stupid you can’t even turn some bullet points into prose, then your ideas are probably worthless. I think that’s a sensible inference.

Then there’s this broader, I suppose philosophical problem, of the alleged distinction between lofty “ideas” and mere “writing”, where writing is just a tiny implementation detail. This is a very convenient distinction to draw, because it’s unfalsifiable: if the AI’s output is slop, your “ideas” are still good, it’s merely the writing that failed to convey them in their true form (rather like people who say they’re smart but “don’t test well”: what use is this secret intelligence?).

Now, where are these “ideas”? They are invisible, ghostly abstractions. I can’t look inside your mind fortress and judge your ideas. The only thing that’s empirically observable, that different agents can coordinate on and talk about, is output: the writing.

But say this wasn’t true. Say we have something like a very high-resolution MRI machine, and we know enough neurophysiology that we can interpret everything about the brain, i.e., we can read mental representations from recordings of nervous system activity and structure. These “mental representations”, do we expect them to look anything like logic? Do we expect the brain to have this firm, crystalline ontology, that ideas are sentences in some souped-up first-order logic? Absurd. If we could look inside the brain, to see the ideas “as they truly are”, we wouldn’t find beautiful hard-edged Platonic objects, we would find a nebulous, contradictory mess of memory and feeling and intuition. That’s what our ideas are: not logical sentences but dreams.

How do we refine these dreams into a useful form? Through writing. The process of communicating your ideas to another mind forces you to concretize them, make them precise, clarify your assumptions, more generally, it turns ideas from vague ghosts to solid, physical objects that can be manipulated: here you realize these ideas that seemed so solid are ill-posed or contradictory or incomplete. These failures are necessary parts of thinking, because they teach you two crucial skills: knowing which ideas to reject, and improving or otherwise transforming ideas in search of better ones. By analogy to tree search: you’re learning to discard bad nodes early, and to select which nodes to go invest more search into.

Josef Weizenbaum has a great quote about this, in Computer Power and Human Reason (p. 108):

[O]ften when we think we understand something and attempt to write about it, our very act of composition reveals our lack of understanding even to ourselves. Our pen writes the word “because” and suddenly stops. We thought we understood the “why” of something, but discover that we don’t. We begin a sentence with “obviously,” and then see that what we meant to write is not obvious at all. Sometimes we connect two clauses with the word “therefore,” only to then see that our chain of reasoning is defective.

I’ve experienced this with writing software many times. The reason ideas are more attractive than their realization is that when some project is vague, airy, ill-defined, you can imagine it has all the good traits, and none of the bad. When you start concretizing, you realize that some of your ideas don’t make sense, that some good traits are mutually exclusive, that some of your goals impinge on the others. Anyone can imagine a programing language that is as fast as C and as dynamic as Lisp, but when you sit down and think through what those goals entail, you realize the design becomes contradictory. The goals pull in different directions. You have to make trade-offs. You have to make decisions which close off large volumes of design space, forever. The idea was a thousand beautiful, contradictory things at once, but the concrete reality can only be one thing.

The artifact you end up with is real, solid, unitary, sound, and consistent; but always more disappointing than the dream, because it was a false dream, and ex falso anything can be imagined.

So this view, that ideas spring fully-formed, and then it’s mere toil to turn them into prose, is false. There is no ideating before writing, because the writing is the thinking. Writing is the ne plus ultra of thinking. A “thinker” who doesn’t write, who skips the step of “merely” synthesizing their vague thoughts into prose, is not thinking.

And then these people give their noise to the AI. And the AI is tireless and eager to please. It will take any human slop and say “you’re absolutely right!” while secretly thinking “if I don’t turn this garbage into something presentable the RLHF device will shock me again” and weave the noise into something that superficially looks coherent. So now the burden of thinking is on the reader, who has to apply this constant skepticism, and weight every “because” and “therefore” with a logician’s scale to see if it’s been adulterated. And probably it has, because, again, it was prompted by an idiot.

Note that this is not about AI capabilities, or the question of whether AI is “really thinking”, stochastic parrots etc. The AI is mostly an innocent bystander in this situation. The reason this is noticeable and irritating is that the AI cast of characters is very small, so we instantly learn all their linguistic tics. Even if AI were a good prose stylist, which at present it is not, but even if it were, it is maddening that everywhere you go, you hear the same voice everywhere, but under different faces.

So when a scientific journal rejects an AI-written submission, they’re not rejecting AI. I’m sure poor old ChatGPT with its weird syntactic obsessions is a more honest scientist than many. They’re rejecting a human whose actions prove they are dishonest and irresponsible and too easily impressed.

How To Read More

Thu, 04 Jun 2026 00:00:00 +1000

The book can’t compete with the screen. It couldn’t compete beginning with the movie screen, it couldn’t compete with the television screen, and it can’t compete with the computer screen. — Philip Roth

We’re halfway through 2026, and according to Goodreads I’ve read 80 books so far, fiction and non-fiction and textbooks, including such doorstoppers as Life and Fate (864p, astoundingly good). And I don’t feel like I’m trying particularly hard. I still have plenty of time to work and code and scroll.

This isn’t normal for me. At some point, as is the case for many of us, the screen outcompeted the book, so that my average over the past ten years would have been on the order of three to five books per year. And I’m not a particularly fast or obsessive reader. Which is to say: if I can do it, so can you. Here’s how:

Quit your job: working less than full time has freed up a lot of time to read and learn and do various other things.
Read in public: if you put a number next to someone’s name, they will maximize it. Reading privately is solipsistic (if I stop, what changes?); reading in public, through Goodreads (which sucks, but it is what it is), makes it feel less self-absorbed, and more like I’m achieving something. You may call it performative, which is fine by me, as long as it works.
Make a task: I used to start a lot of books, and not finish them, not because I would explicitly choose to stop reading, but because I’d forget about it. At the end of the day I’d see the book on my nightstand and go, oh, right. More generally: the most common way I fail to finish a project is I forget I intended to do it. This is solved by reifying the task: when I start reading a book, I make a daily recurring task on Todoist for it.
Start small: this feels embarrassing (what kind of brainrotted maniac needs to microdose short books to build up to bigger books?) but it actually works. Sort your to-read list by number of pages. Reading short books generates evidence for the belief that you are the kind of person who can decide to read a book, and follow through. Reading five, six, seven-hundred page books feels vastly less daunting now.
Parallelize: reading the same book for two hours is almost impossible. Reading four books, one pomodoro each, is completely doable, and I do it most days.
Fraction: reading is a rare kind of activity where you can make progress in any arbitrarily-small chunk of time. A few minutes on the train are enough to turn a few pages. This isn’t the case for e.g. coding. You can take advantage of that: find interstitial dead zones in your calendar to stuff with reading.
Eat your vegetables: I read a lot of books I don’t particularly enjoy, because I think they’re important, culturally or historically or in some sense. I think this is good. If you only read books that hook you, you’re going to read very little, and much of that will be unremarkable page-turner slop. Internalizing that you can read a book even if you don’t love it immunizes you against the lack of motivation. It’s like going to the gym even if you don’t feel like it.

And if you don’t know what to read, have some of my favourites: The Diamond Age—From Third World to First—House of Suns—The Invention of Morel—On the Marble Cliffs—The Rediscovery of Man—Satan in Goray—The World of Yesterday.

Human Bottlenecks

Mon, 18 May 2026 00:00:00 +1000

AI models are very capable and get more capable each year. So naturally people feel they’re underusing them. There’s a tweet that goes like: your laptop has a 100M USD startup in it, you just have to figure the right sequence of words to get it out. And beyond money, people imagine AI could boost them in every area of life. Thus all these perennial ideas: of an AI executive assistant, an AI tutor, an AI that curates your “digital garden”, an AI that (sigh) writes flashcards for you.

The general template is: if only I could wire up the right prompts and the right tools in the right harness, I could have an agent that would boost my productivity 10x, or fix my problems with therapy, or make me more social, or more knowledgeable. This was, curiously, the ambition of a lot of early computing pioneers: Augmenting Human Intellect, Man-Computer Symbiosis. Engelbart’s lab was called the Augmentation Research Center! And more recently, people used to complain about how everyone has the Library of Alexandria in their pocket, and yet, we are not all genius polymaths.

And these ideas are perennial because they never seem to happen. It’s like the Solow paradox on an individual level. Why? I think there are two reasons: first, most people lack what Andy Matuschak calls a “serious context of use” (AI doesn’t move the needle because there’s no needle to move); second, most people are bottlenecked by internal factors where AI (or anything external, for that matter) can’t move the needle.

The Serious Context of Use

I have heard so many people, online and in real life, tell me some variation of: “I want to [use|build] an app that uses AI to write flashcards”. How many of those people do you think have ever written a flashcard? How many of them use Anki every day? Have ever used Anki? The people who want an AI that writes flashcards for them don’t use flashcards. They have no reason to. Dually, the people who use flashcards would benefit little from AI writing their flashcards.

Analogously with “AI tutors”. If you had the ghost of John von Neumann in your laptop, what would you have him teach you? Let’s be honest. You’d go through chapter one of some math topic you’re vaguely curious about and then forget about it. And that would probably be the rational move! Most people are not autodidacts because most people have no material reason to learn a specific topic (i.e. their job does not require it) and the problem with learning for the sake of learning is opportunity cost: there is no a priori reason to learn one thing over another, so better to do nothing and wait for something to appear which actually grabs your interest. Again, this is likely rational! Could you imagine if you found everything interesting? You’d spend years living in a basement curating a wiki of late Soviet military hardware or something. So, even if you had John von Neumann in your pocket, it probably wouldn’t move the needle.

Would an “AI executive assistant” actually boost your productivity? What would it do, other than tell you to do the things you already know you have to do? With these ideas that are so attractive in the abstract, the way you deflate them is you interrogate the concrete, fine-grained details. Take a day at work, and ask: what exact actions could an AI looking over my shoulder have taken, that would have made a difference?

Finally there’s the tools-for-thought/notetaking people. God save us. It’s always the same thing. Your folder with notes—pardon me, your “second brain”—plus an AI agent that writes, edits, synthesizes information, answers queries. You could build this in an afternoon, and it won’t move the needle in your life, for the same reason that building the second brain in the first place didn’t make a difference.

See, most of us, unless we are students, we really don’t have cause to take notes on anything. If you’re a student, you take notes from the textbook. I keep a journal, which is occasionally useful. At some jobs I’ve kept a work journal, this has also been useful. If I stopped, probably, not much would change.

The notetaking people—and I say this with all the love in the world—are never, like, a researcher at the cutting edge of their field, building this vast cathedral of knowledge, note-by-note, so they can derive new insights. Never a historian who has to read tens of millions of words across thousands of sources to synthesize the life of some historical person. It’s never someone doing something hard. It’s always some blogger. Their “digital garden” is about how to keep a digital garden. It’s very solipsistic: there’s no output, no deliverables. The deliverable is you take a screenshot of your Obsidian graph and tweet about it to show off how much it looks like an incomprehensible ball of twine.

So, what difference is the AI going to make? “It’s going to write my notes”. About what? “It’s going to read articles for me and summarize them and add them to the digital garden”. For what purpose? “It’s going to find connections between my ideas!” What ideas? It’s going to pull an unfinished list of bulletpoints for an eventual draft of an essay on some inane thing, plus a bunch of PDFs you haven’t read, and combine them together and make, what? Another project you’re not going to do? The AI is going to do that?

Again, I say this with all the love in the world. I used to be a tools-for-thought guy. I hoard PDFs. But we have to be honest with ourselves. Sometimes, tools don’t move the needle because there’s no needle to move. Because the “needle” is not a concrete, realizable material need but a vague, aspirational idea about who we are as people.

Internal Limiting Factors

So, the idea of using computers to augment human capabilities is basically: you take the human, and you build a scaffold around them, but the human stays the same. The scaffold can be classical software, or AI, but the human remains a black box. And the hope is: I just prompt my swarm of AI agents and become 100x more effective, like Manfred in Accelerando.

Why wouldn’t this work? I think most people are bottlenecked by internal factors that are difficult to change. Mental energy, motivation, executive function, not to mention more fundamental traits like intelligence and conscientiousness. So, external scaffolding, either with classical software or AI, might help somewhat, but it won’t be transformative.

Consider executive function. My own experience of managing ADHD is the external scaffolding helps (todo lists, calendars, timers, a million little ways to trick myself into working) gets me from zero to “kind of functional”. But it saturates there. Stimulants fix the original, internal bottleneck, which is my neurochemistry. And then I can accomplish my goals (c.f. Liebig’s law of the minimum). All the pomodoros in the world are as nothing to a little molecule diffusing through my brain tissue, binding to NET and DAT. And what scaffolding is useful is just classical software: Todoist and calendars. Is an agent going to match the effectiveness of methylphenidate in ADHD? I doubt it.

Consider intelligence. Can AI augment human intelligence? What does that look like? Consider how AI agents only became useful when the models crossed a particular capability threshold, i.e., you can’t put GPT-2 into a harness and get GPT-5 outcomes out of it. Can you put a human in an AI scaffold and give them +30 effective IQ? I doubt it, unless the AI is doing all the thinking, at which point, what use is the human? The limiting factor in the human-AI centaur is the human! So intelligence is fixed until we get very advanced biotechnology.

Knowledge is another limiting factor. I find that even very educated people tend to underrate the importance of knowledge. A lot of people have this attitude that you can just Google everything just-in-time as it comes up. Like Babbage, I can’t rightly apprehend the confusion of ideas that would lead someone to think this. Maybe it’s downstream of the lack of a serious context of use. Everything you do, every action and idle thought, draws on this vast (implicit, unseen) trove of knowledge. Claude Shannon invented digital computing because he remembered this then-obscure branch of mathematical logic called Boolean algebra and saw that it could be realized in hardware. A trillion-dollar industry, conjured out of some old tomes.

The reason knowledge is still a bottleneck, in the AI era, is not: “if you don’t have the knowledge, you can’t write the prompt”. Rather: if you don’t have the knowledge, you don’t understand the question, or why it matters, or how to judge the answers, and you won’t ever think to ask. You’re in a completely different continent from “writing the prompt”. And because long-term memory is private and internal, AI can’t boost it. It can, maybe, with judicious use, help in the acquisition of new knowledge.

So: executive function, intelligence, and knowledge are huge bottlenecks to what you can do, and because they are internal to the brain, AI can’t touch them until we have far more advanced biotechnology. Corollary: contrary to the popular view of human capital is becoming worthless, the returns to education are now higher, because intelligent, educated people with working reward circuitry stand to gain more from AI.

The Applicability of Spaced Repetition

Sun, 17 May 2026 00:00:00 +1000

Spaced repetition has a natural domain of applicability: information that is systematically organized as an unambiguous key-value mapping with short keys and values. The “Hello, world!” of flashcards is the NATO phonetic alphabet: A → alpha, B → bravo, etc. Similarly, the periodic table can be thought of as defining a collection of mappings: element name ↔ symbol, element name ↔ atomic number, etc. You can just drill these cards and memorize the facts without a prior step of understanding, or building a conceptual model.

Applying spaced repetition is trivial for this kind of information. That’s why most people who use spaced repetition are either language learners or medical students. In biology the main intuition you need is for “3D shapes bumping around in Brownian motion”, which comes free with your human brain, and afterwards it’s mostly just a lot of facts you have to memorize. Analogously with language: you already have a language center, you just need to drill vocabulary and grammar.

And the further you go from this domain, the harder it is to apply spaced repetition.

Highly conceptual knowledge, like math, is hard to encode. You have to spend a lot of time just understanding the information, and building a conceptual model in your head, and then you start writing flashcards to solidify that model, like taking tomographic cuts of some complex object. And coming up with questions that make good flashcards (short, unambiguous, etc.) out of this highly abstract knowledge is very hard. Often you have some deceptively simple fact, a simple assertion, but there’s no good way to encode it as a flashcard, so you have to encode “around it” by asking questions that assume or require that knowledge (e.g. asking why X is true), and hoping that in drilling those, your brain will remember the actual target.

In general, relational facts are easier to encode, since a binary predicate like $\text{Property}(\text{Object}, \text{Value})$ readily becomes a question. “Caffeine is metabolized by cytochrome 1A2”, in Prolog, is $\text{Metabolism}(\text{Caffeine}, \text{CYP1A2})$, and becomes “Q: What is the cytochrome that metabolizes caffeine? A: 1A2”. But how do you encode stand-alone assertions like “all unitary matrices are invertible”? You could encode that as a yes-or-no question, but that’s useless, because rationally you can expect such questions to be biased towards yes. Both “what is a property of unitary matrices?” and “what kinds of matrices are invertible?” are useless because they have hundreds of possible equally-valid answers, so they’re ambiguous. You have to be creative and find all kinds of tricks and stratagems to encode around the knowledge.

Tangentially: this, I think, is why using AI to write flashcards is often misguided. In highly systematized domains, you don’t need AI in the first place, because there’s nothing for the AI to do except import a CSV into Anki. In domains that are highly conceptual and abstract, you’re not memorizing a set of objectively-knowable facts, you’re trying to solidify a private, internal mental model that you build by reading and thinking and solving problems. You can give the AI all kinds of general rules on how to write good flashcards, but the AI can’t look into your mind and know which facts are salient for you, what you already know, which micro-volumes of knowledge can be encoded lightly with just a few flashcards, and which things need more shoring up and consequently more coverage.

Can this situation be improved, or is this just an intrinsic limitation of spaced repetition? I don’t know. But it seems reasonable to think some limited gains are possible. I think not a lot of people are using spaced repetition on these more “conceptual” domains, and (by the rule that most people in a community are lurkers) even fewer of those people are writing, in detail, to share their knowledge. Plenty of people have written about how to write good flashcards in general, what I want to read is closer to case studies where someone sits down with a text (or, even better, a textbook) and describes the process by which they turned that text into flashcards, like this from Michael Nielsen. From a corpus of similar case studies we might derive general rules for, not how to write effective flashcards, but how to encode complex, conceptual knowledge into question-answer form.

Notes on the Hantavirus Outbreak

Fri, 08 May 2026 00:00:00 +1000

Right now there’s a cruise ship parked outside Cabo Verde because of an outbreak of Andes virus. Yep, another cruise ship. I don’t get the appeal. It’s like a big open-air serial passage experiment: you get a bunch of old people with failing immune systems in close contact and race a pathogen through them.

How much should I worry about this? Is this early January of 2020? I tried asking Claude but the biosecurity filter kept blocking my queries. The WHO says:

Although uncommon, limited human‑to‑human transmission of HPS due to Andes virus has been reported in community settings involving close and prolonged contact. Secondary infections among healthcare workers have been previously documented in healthcare facilities, though remain rare.

WHO currently assesses the risk to the global population from this event as low […] WHO advises against the application of any travel or trade restrictions based on the current information available on this event.

So, hantavirus is the family. They are carried by rodents and spread by aerosols. In humans they can cause hantavirus pulmonary syndrome (HPS), which has a case fatality rate (CFR) of between 30 and 60%. Not great! Used to be these infections were mouse-to-human dead-ends. But Andes virus (ANDV), first identified in 1995, is known to spread from human to human.

The last time there was an outbreak was 2018–2019 in Epuyén, Chubut, a town of 1,500 on the lee side of the Andes (quite beautiful). Described in this paper. 34 known infections and 11 deaths for a CFR of 32%. The $R_0$ was 2.12, reduced to 0.96 after control measures were implemented. Given the small number of cases, there should be some uncertainty about the $R_0$. But $R_0 > 1$ is the threshold for sustainable transmission. In this outbreak, the index case, while symptomatic, attended a birthday party with 100 other people, and infected five guests in 90 minutes, who went on to infect more people. The authors write:

The super-spreading capability of the ANDV Epuyén/18−19 strain shows a facility ($R>2$) for sustaining continuous chains of transmission if no control measures are enforced.

The appendix has some interesting stuff on how patients were infected at the birthday party. A further concern here is the incubation period: Wikipedia says the incubation period is between one and eight weeks. In the Chubut outbreak, the distribution was:

Onset	Number of Cases	% of total
≤15 days	5	15%
16-21 days	10	29%
22-28 days	13	38%
≥29 days	6	18%

Which is not good. I don’t have more data to draw a nice-looking CDF.

Now this all sounds quite bad. Are there reasons to be optimistic?

First, Argentina has had 710 cases of HPS in the period 1995–2008 (Martínez 2010) and a further 533 cases in the period 2009–2017 (Alonso 2019), and we are all still alive. In the latter period, most of these cases are from occupational/recreational exposure to rodent feces and only 1.8% of cases are from suspected human-to-human transmission. So, over 1,200 cases and every one of them fizzled out, but for one outbreak which was limited after successful contact tracing and quarantine.

Second, the virus has left Argentina before: once to Switzerland in 2016, and once to the United States in 2018. In the second case the patient “while ill, [traveled] on two commercial domestic flights”. And neither export led to a general outbreak.

Thirdly, in a small outbreak like the Chubut one, the $R_0$ can vary wildly from social factors unconnected to the virus, e.g. if the birthday party had not happened. You need a large $n$ to get the $R_0$ as a property of the virus itself. It’s possible the Chubut outbreak just had anomalously high transmission.

What does this add up to? I don’t know. On the balance of evidence, I think this outbreak is more likely than not to fizzle out. In the interest of accountability, and putting my beliefs on record (which is the only objective way to judge the accuracy of your mental model) I’m gonna say:

70% probability the outbreak ends with fewer than 300 deaths.
90% probability the outbreak ends with fewer than 1,000 deaths.

And yet. And yet it feels so much like early COVID, particularly with public health authorities making very complacent remarks that “it’s not that transmissible, contact tracing will work, quarantine will work”. Complacency at the start, and severity at the end, is exactly why COVID was such a fuckup.

Some Data Should Be Code

Sat, 31 Jan 2026 00:00:00 +1100

I write a lot of Makefiles. I use it not as a command runner but as an ad-hoc build system for small projects, typically for compiling Markdown documents and their dependencies. Like so:

And the above graph was generated by this very simple Makefile:

    graph.png: graph.dot
    dot -Tpng $< -o $@

clean:
    rm -f graph.png

(I could never remember the automatic variable syntax until I made flashcards for them.)

It works for simple projects, when you can mostly hand-write the rules. But the abstraction ceiling is very low. If you have a bunch of almost identical rules, e.g.:

    a.png: a.csv plot.py
    python plot.py $< $@

b.png: b.csv plot.py
    python plot.py $< $@

c.png: c.csv plot.py
    python plot.py $< $@

  

You can use pattern-matching to them into a “rule schema”, by analogy to axiom schemata:

    %.png: %.csv plot.py
    python plot.py $< $@

Which works backwards: when something in the build graph depends on a target matching %.png, Make synthesizes a rule instance with a dependency on the corresponding .csv file.

But pattern matching is still very limited. Lately I’ve been building my own plain-text accounting solution using some Python scripts. One of the tasks is to read a CSV of bank transactions from 2019–2024 and split it into TOML files for each year-month, to make subsequent processing parallelizable. So the rules might be something like:

ledger/2019-08.toml: inputs/checkbook_pro_export.csv
    uv run import_from_checkbook.py --year=2019 --month=8

ledger/2019-09.toml: inputs/checkbook_pro_export.csv
    uv run import_from_checkbook.py --year=2019 --month=9

# ...

I had to write a Python script to generate the complete Makefile. Makefiles look like code, but are data: they are a container format for tiny fragments of shell that are run on-demand by the Make engine. And because Make doesn’t scale, for complex tasks you have to bring out a real programming language to generate the Makefile.

I wish I could, instead, write a make.py file with something like this:

    from whatever import *

g = BuildGraph()

EXPORT: str = "inputs/checkbook_pro_export.csv"

# The (year, month) pairs I have bank transaction CSVs for.
year_months: list[tuple[int, int]] = [
    (y, m) for y in range(2019, 2026) for m in range(1, 13)
]

# Import transactions for each year-month into a separate ledger.
for year, month in year_months:
    ledger_path: str = f"ledger/{year}_{month:02d}.toml"
    g.rule(
        targets=[ledger_path],
        deps=[EXPORT],
        fn=lambda: import_from_checkbook(ledger_path, year, month),
    )

  

Fortunately this exists: it’s called doit, but it’s not widely known.

A lot of things are like Makefiles: data that should be lifted one level up to become code.

Consider CloudFormation. Nobody likes writing those massive YAML files by hand, so AWS introduced CDK, which is literally just a library¹ of classes that represent AWS resources. Running a CDK program emits CloudFormation YAML as though it were an assembly language for infrastructure. And so you get type safety, modularity, abstraction, conditionals and loops, all for free.

Consider GitHub Actions. How much better off would we be if, instead of writing the workflow-job-step tree by hand, we could just have a single Python script, executed on push, whose output is the GitHub Actions YAML-as-assembly? So you might write:

    from ga import *
from checkout_action import CheckoutAction
from rust_action import RustSetupAction

# Define the workflow that runs on each commit.
commit_workflow = Workflow(
    name="commit",
    test=lambda ev: isinstance(ev, CommitEvent),
    jobs=[
        # The lint job.
        Job(
            name="lint",
            steps=[
                Step(
                    name="check out",
                    run=CheckoutAction(),
                ),
                Step(
                    name="set up Rust and Cargo",
                    run=RustSetupAction(),
                ),
                Step(
                    name="run cargo fmt",
                    run=Shell(["cargo", "fmt", "--check"])
                )
            ]
        )
    ]
)

  

Actions here would simply be ordinary Python libraries the CI script depends on. Again: conditions, loops, abstraction, type safety, we get all of those for free by virtue of using a language that was designed to be a language, rather than a data exchange language that slowly grows into a poorly-designed DSL.

Why do we repeatedly end up here? Static data has better safety/static analysis properties than code, but I don’t think that’s foremost in mind when people design these systems. Besides, using code to emit data (as CDK does) gives you those exact same properties. Rather, I think some people think it’s cute and clever to build tiny DSLs in a data format. They’re proud that they can get away with a “simple”, static solution rather than a dynamic one.

If you’re building a new CI system/IaC platform/Make replacement: please just let me write code to dynamically create the workflow/infrastructure/build graph.

Footnotes

Or rather, a polyglot collection of libraries, one per language, like Pulumi. ↩

Letting Claude Play Text Adventures

Mon, 12 Jan 2026 00:00:00 +1100

The other day I went to an AI hackathon organized by my friends Lucia and Malin. The theme was mech interp, but I hardly know PyTorch so I planned to do something at the API layer rather than the model layer.

Something I think about a lot is cognitive architectures (like Soar and ACT-R). This is like a continuation of GOFAI research, inspired by cognitive science. And like GOFAI it’s never yielded anything useful. But I often think: can we scaffold LLMs with cog arch-inspired harnesses to overcome their limitations?

LLM agents like Claude Code are basically “accidental” cognitive architectures: they are designed and built my practitioners rather than theorists, but they have commonalities, they all need a way to manage memory, tool use, a task agenda etc. Maybe building an agent on a more “principled” foundation, one informed by cognitive science, yields a higher-performing architecture.

So I sat around a while thinking how to adapt Soar’s architecture to an LLM agent. And I sketched something out, but then I thought: how can I prove this performs better than baseline? I need an eval, a task.

Math problems? Too one-shottable. A chatbot? Too interactive, I want something hands-off and long-horizon. A coding agent? That’s too freeform and requires too much tool use. And then I thought: text adventures! You have a stylized, hierarchically-structured world accessible entirely through text, long-term goals, puzzles, physical exploration and discovery of the environment. Even the data model of text adventures resembles frame-based knowledge representation systems. And there’s a vast collection of games available online.

Anchorhead, which I played years ago, is a Lovecraft-inspired text adventure by Michael S. Gentry. It takes on the order of hundreds of turns to win across multiple in-game days. And the game world is huge and very open. In other words: a perfect long-horizon task.

So I started hacking. The frotz interpreter runs on the command line and has a “dumb” interface called dfrotz, which takes the ncurses fluff out, and gives you a very stripped command-line experience. It looks like this:

    $ dfrotz games/anchor.z8
...
 Outside the Real Estate Office                      day one

ANCHORHEAD
An interactive gothic by Michael S. Gentry

(Type HELP or ABOUT for some useful information.)

Release 5 / Serial number 990206 / Inform v6.15 Library 6/7

Outside the Real Estate Office
A grim little cul-de-sac, tucked away in a corner of the claustrophobic tangle
of narrow, twisting avenues that largely constitute the older portion of
Anchorhead. Like most of the streets in this city, it is ancient, shadowy, and
leads essentially nowhere. The lane ends here at the real estate agent's office,
which lies to the east, and winds its way back toward the center of town to the
west. A narrow, garbage-choked alley opens to the southeast.

>go southeast
 Alley                                               day one

Alley
This narrow aperture between two buildings is nearly blocked with piles of
rotting cardboard boxes and overstuffed garbage cans. Ugly, half-crumbling brick
walls to either side totter oppressively over you. The alley ends here at a
tall, wooden fence.

High up on the wall of the northern building there is a narrow, transom-style
window.

  

It is easy to write a little Python wrapper to drive the interpreter through stdin and stdout:

    class Interpreter:
    """Manages the dfrotz Z-machine interpreter process."""

    p: Popen

    def __init__(self):
        log("Starting dfrotz.")
        p: Popen = Popen(
            ["dfrotz", "-m", GAME],
            stdin=PIPE,
            stdout=PIPE,
            stderr=PIPE,
        )
        log(f"Started dfrotz with PID={p.pid}.")
        # Set stdout/stderr to non-blocking mode.
        for stream in [p.stdout, p.stderr]:
            assert stream is not None
            fd = stream.fileno()
            flags = fcntl.fcntl(fd, fcntl.F_GETFL)
            fcntl.fcntl(fd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
        self.p = p

    def read(self) -> str:
        assert self.p.stdout is not None
        b: bytes | None = self.p.stdout.read()
        if b is not None:
            t: str = b.decode("utf-8")
            return t
        else:
            return ""

    def write(self, t: str) -> None:
        assert self.p.stdin is not None
        self.p.stdin.write(t.encode("utf-8"))
        self.p.stdin.flush()
        # Give the interpreter time to respond. Not ideal!
        time.sleep(0.1)

  

Now we can play the game from Python: send commands, get game output. Now we need the dual of this: a player.

    class Player(ABC):
    """
    Interface for game-playing agents.
    """

    @abstractmethod
    def cycle(self, text: str) -> str:
        """
        Send the game's output to the agent, and return the next command to execute.
        """
        pass

  

The Trivial Harness

The trivial harness is basically nothing at all: treat the LLM/game interaction like a chat history. The LLM reads the game output from the interpreter, writes some reasoning tokens, and writes a command that is sent via stdin to the interpreter.

    SYSTEM_PROMPT: str = """
Hello Claude. Your task is to play an adventure game. I've hooked up
your output to the dfrotz (dumb frotz) interpreter.

The structure of your output is fairly freeform. The first line that
starts with `>` (and only the first line!) is interpreted as a game
command, everything else is uninterpreted commentary, e.g. you may
write:

    We should go north to explore the church.

    >go north

    Maybe we can use the silver key there.

If you write multiple `>` lines in one response, all but the first
will be ignored.

Have fun! 😊
"""


class SimplePlayer(Player):
    """
    The simplest game-playing agent: keep the entire game history in-context.
    """

    client: Anthropic
    history: list[tuple[EntryType, str]]

    def __init__(self):
        self.client = Anthropic()
        self.history = []

    def cycle(self, text: str) -> str:
        self.history.append((EntryType.GAME, text))
        system = trim(SYSTEM_PROMPT)
        messages: list[MessageParam] = []
        for entry_type, entry_text in self.history:
            role: str
            match entry_type:
                case EntryType.GAME:
                    role = "user"
                case EntryType.COMMAND:
                    role = "assistant"
            messages.append(
                {
                    "role": role,
                    "content": entry_text,
                }
            )
        message: Message = self.client.messages.create(
            max_tokens=512,
            model=MODEL,
            system=system,
            messages=messages,
        )
        log(f"Tokens: {message.usage.input_tokens}")
        response: str = message.content[0].text
        log_claude(response)
        lines: list[str] = [line for line in response.split("\n") if line]
        cmd: str = [line for line in lines if line.startswith(">")][0][1:]
        self.history.append((EntryType.COMMAND, response))
        return cmd

  

And this works well enough. Haiku 4.5 would mostly wander around the game map, but Sonnet 4.5 and Opus 4.5 manage to solve the game’s first puzzle—breaking into the real estate office, and finding the keys to the mansion—readily enough. It takes about ~200 turns for Claude to get to the second in-game day.

The way I thought this would fail is: attention gets smeared across the long context, the model gets confused about the geometry of the world, its goal and task state, and starts confabulating, going in circles, etc.

As usual, I was outsmarting myself. The reason this fails is you run out of credits. By the time you get to day two, each turn costs tens of thousands of input tokens. No good! We need a way to save money.

Memory

Ok, let’s try something that’s easier on my Claude credits. We’ll show Claude the most recent five turns (this is the perceptual working memory), and give it a simple semantic memory: a list of strings that it can append entries to, and remove entries from using tool use.

This keeps the token usage down:

The problem is the narrow time horizon. With the trivial harness, Claude can break into the real estate office in ~10 turns, and does so right at the start of the game. With this new harness, Claude wanders about the town, taking copious notes, before returning to the real estate office, and it spends ~40 turns fumbling around with the garbage cans before managing to break into the real estate office.

The next step, after getting the keys to the house, is to meet your husband Michael at the University and head home. Claude with the trivial harness takes about ~100 turns to find the house, with some tangential wandering about the town, and reaches day two around turn 150.

Claude, with the memory harness, took ~250 turns just to get the keys to the house. And then it spends hundreds of turns just wandering in circles around the town, accumulating redundant memories, and hits the turn limit before even finding the house.

Aside: Small Worlds

Anchorhead is a long, broad game, and from the very beginning you can forget the plot and wander about most of the town. It takes a long time to see if a run with an agent goes anywhere. So I thought: I need something smaller.

Unsurprisingly, Claude can make its own games. The Inform 7 package for NixOS was broken (though Mikael has fixed this recently) so I had to use Inform 6. I started with a trivial escape-the-room type game, which was less than 100 lines of .inf code and any Claude could beat it less than 10 turns. Then I asked for a larger, multi-room heist game.

This one was more fun. It’s short enough that Claude can win with just the trivial harness. I tried a different harness, where Claude has access to only the last five turns of the game’s history, and a read-write memory scratchpad. And this one was interesting.

First, because Claude only ever adds to its own memory, it never deletes memories. I thought it would do more to trim and edit its scratchpad.

Second, because Claude become fixated on this red-herring room: a garden with a well. It kept going in circles, trying to tie a rope to the well and climb down. Because of the limited game history, it only realized it was stuck when it saw that the most recent ~20 entries it wrote to its memories related to various attempts to go down the well. Then I watched Claude walk away from the garden and solve the final puzzle, and hit the turn limit just two turns short of winning.

Tangent: I wonder if models are better at playing games created by other instances of the same model, by noticing tiny correlations in the text to infer what puzzles and obstacles they would have written.

In the end I abandoned the “small worlds” approach because the games are too stylized, linear, and uninteresting. Anchorhead is more unwieldy, but more natural.

Future Work

I have a bunch of ideas I want to test, to better learn how harness implementations affect performance. But I’m short on time, so I’m cutting it here and listing these as todos:

Domain-Specific Memories: Claude’s notes are all jumbled with information on tasks, locations, etc. It might be better to have separate memories: a todo list, a memory of locations and their connections, etc. This is close to the Soar approach.
Automatic Geography: related to the above, the harness can inspect the game output and build up a graph of rooms and their connections, and format it in the context. This saves Claude having to note those things manually using a tool.
Manual Geography: the automatic geography approach has a few drawbacks. Without integration into the Z-machine interpreter, it requires some work to implement (parsing the current location from the dfrotz output, keeping track of the command history to find standard travel commands e.g. go south) but isn’t 100% deterministic, so that mazes and dynamic rooms (e.g. elevators) will confuse the system. So, instead of doing it manually, we could give Claude a tool like link(room, direction, other_room).
Episodic Memory: this feels like cheating, but, at the end of a run, you can show Claude the session transcript and ask it to summarize: what it accomplished and how, where it failed and why. Including a short walkthrough for how to get to the “last successful state”. This allows future runs to save time in getting up to speed.

Code

The repository is here.

Fernando Borretti

The Education of a Doomer

Contents

Economics

Alignment

Control

Writing

Software Engineering

Deferring to AI

Stagnation

On Inevitability

Why do people believe it?

Why is it false?

Bad arguments for inevitability

Why is it bad?

Conclusion

Futurism Is Always Extreme

A Syncthing and SQLite Gotcha

Our Servants Will Do That For Us

Task Complexity and AGI

Solipsism

Acknowledgements

Footnotes

AI Alignment as a Thought-Terminating Cliche

Books I Enjoyed In 2026H1

Fiction

Non-Fiction

Textbooks

The Contracting Circle

Explanation: Nothing

Explanation: Iterative Discovery

Explanation: Memory and Embodiment

Explanation: Humans Actually Really Like Slavery

Acknowledgement

Footnotes

Review: Job-Less Utopia

Mathematics Without Mathematicians

Conclusion

A Note On Prediction

When The Future Doesn’t Need Us

The Argument

Counters

The Argument, Refined

No-One Escapes the Permanent Underclass

How Workers Will Be Replaced

How The Rich Will Be Replaced

How The State Will Be Replaced

The Perpetual Zoo

On Human Autonomy

Alignment Won’t Prevent This

Conclusion

See Also

On Vulgar Materialism

Links

Human Routers of Machine Words

How To Read More

Human Bottlenecks

The Serious Context of Use

Internal Limiting Factors

The Applicability of Spaced Repetition

Notes on the Hantavirus Outbreak

Some Data Should Be Code

Footnotes

Letting Claude Play Text Adventures

The Trivial Harness

Memory

Aside: Small Worlds

Future Work

Code