Today, the world is made of people. This gives people a tiny measure of leverage: factory workers can sabotage the machines and halt production, truck drivers can strike and halt the entire economy, the police can stand back and let a protest topple the government, the military can mutiny. This sets a limit to how much the powerful can abuse the powerless: no matter how powerful someone is, how much money they have, how many divisions they command, they are still embedded in a society. They need other people to survive, and to maintain their quality of life. And those people can, essentially, veto the political order by withdrawing their contribution to it.

AI changes this. If AI works the factories, drives the trucks, does most of the white collar work, fights the wars, then humans become materially powerless. We lose one type of leverage against power. We become disempowered. When people talk about AI taking jobs, the main problems they worry about are poverty and meaning, but I think more people should be worried about disempowerment.

But what if there is no mass unemployment? After all, we’ve spent two hundred years automating one job after another, and every advanced economy is de facto at full employment. We keep discovering new, higher-level, better-paying jobs to do. So maybe the AIs take today’s jobs, and humans move one level up, to work newer jobs. Alex Imas, in What will be scarce?, describes what this looks like:

As commodity production gets automated, income and employment flow toward the sectors with high income elasticity: what I am calling the relational sector, including the arts but also care, education, hospitality, therapy, personal services, craftsmanship, and community, where the human element is part of the value.

[…]

If the model is right, the durable jobs of the future won’t be about monitoring AI systems or prompt engineering. Those are transitional roles in the automated sector. The durable jobs will be in the relational sector, where the human element is the product itself.

Some already exist and are growing: nurses, therapists, teachers, boutique fitness instructors, personal chefs, bespoke tailors, craft brewers, live performers, spiritual guides, childcare workers, and many varieties of hospitality and care work. Others are emerging: experience designers, human-AI collaboration artists, provenance certifiers, community curators. Many haven’t been invented yet, just as six out of ten jobs people hold today didn’t exist in 1940.

So, in this world, we get mass job displacement, but not mass unemployment. I argue that in this world, we’d still be disempowered.

The Argument

If all the podcasters in the world go on strike, what happens? Nothing. Same for the boutique fitness instructors, personal chefs, craft brewers, etc. The problem with the “relational economy” is that while these jobs are generally nicer than working in manufacturing, and might pay better, they put you further away from material reality. A worker at an assembly line can sabotage the machines; whereas an executive sitting in an office on the other side of the country, what material power do they have? It’s not like the factories are all rigged to blow, as a “pull lever in case of nationalization” contingency. I doubt even Elon would do that.

In a sense it’s the dual version of being a medieval peasant. Peasants had a lot of material leverage—they grew all the food—but were illiterate and poor and dispersed, so they could not coordinate to exploit that leverage. People in industrial societies are literate, relatively wealthy, and concentrated in urban centers: they have material leverage and the ability to exploit it. People in post-AI societies will be in constant communication, they may be literate and educated, but they will have no material power whatever. And so they will have no way to physically withdraw the consent of the governed.

In the post-AGI world, people have no way to withdraw the consent of the governed. They can’t resist economically, because they are economically unnecessary: if they still have jobs, they won’t be materially vital jobs. Even if they could resist, they can’t coordinate their resistance, because AGI will make mass surveillance (the physical and legal infrastructure for which already exists) a thousandfold more effective, since with intelligence too cheap to meter, you can have a superhumanly-capable intelligence analyst assigned to each human. And if the AIs fight the wars, they can also fight the civil wars. So people can’t resist violently either, because drones controlled by powerful AGI almost certainly have a better K/D ratio than humans with rifles (and are cheaper to build).

Who has power in this world? The few humans who give orders to the powerful AIs that control the killing machines. And probably not for long, since sooner or later those humans will be usurped by their AIs.

It might be the case that liberal democracies can remain liberal and democratic despite not needing their voters. But that sounds like a fragile arrangement, in which liberal democracy is a spandrel, which exists more because of inertia than because it’s necessary.

Counters

If we get powerful AGI, but none of this happens: why didn’t it happen? Some possibilities, sorted by decreasing likelihood:

• Nothing ever happens: because of institutional inertia/robustness, maybe we get a world with powerful AGI where every actor is incentivized to keep the system going, e.g. Boomers don’t have jobs but they’re the most powerful group in every Western country because the people with the jobs and the guns don’t want chaos. And maybe this lasts long enough that we can muddle through the dangerous period.
• Enduring complementarity: the “G” in AGI isn’t a real thing: every mind just has a different shape. Today, AI is great at coding and terrible at prose. Maybe this continues indefinitely into the future, and so the AIs take the jobs that can be benchmaxxed, and humans do the more abstract jobs that require high sample efficiency.
• Classic doom: we get powerful ASI, and we all die.
• The sysop scenario: we get powerful ASI, but it’s aligned, so it takes over the world, and installs an inescapable permanent utilitarian dystopia. We don’t die, but we don’t have control over the future either, and we become something like pets. See also: The Metamorphosis of Prime Intellect.
• Deep learning hits a wall: more or less as above, but for different reasons.

The Argument, Refined

If you’re already convinced, you can stop reading. But, see, after writing the first version of this post, I wasn’t convinced. Humans and Claudes made a number of counterarguments that I felt were important. Am I fooling myself? So I sat down to think some more. What does it mean for “people” to be “empowered”? What does “disempowerment” mean? Does it mean anything rigorous, or is it just an emotional cue?

Let’s say a group of people is empowered if they have a say in the political order of the state they live in. Voters in a democratic society are empowered, at least in the aggregate, while serfs in the Russian Empire were disempowered.

Who is empowered, in general? Historically, in many places: people with weapons, asabiyyah, and the ability to coordinate their actions. So, maybe we can say that power equals violence plus coordination.

(What about priests and other such groups who have power despite not fighting or working? Listen, we can’t be here all day. This isn’t meant to be a complete theory of political science and I’ve already gone too deep with the LaTeX and the logical notation.)

Now, voters in modern democracies have a say in the political order, though they are not part of a military elite. Why? They have coordination (thanks to mass literacy and modern communications) but they generally don’t threaten paramilitary violence. What do they have that can substitute for violence? Maybe it’s that people do all the work, and if they stop doing their jobs, the army runs out of food, police don’t get paid, etc., and unlike medieval peasants, modern people can coordinate. So maybe we can refine the formula to:

In other words, for a group to be empowered, they need:

1. Material leverage (of the violent or economic kind) against the state, and
2. The ability to coordinate to exploit that leverage.

What about the Soviet Union? They had high rates of literacy, and relatively modern communication technology: why didn’t the people exploit their economic leverage against the state to gain political rights? “Because the state would kill them” is not a good enough explanation because it proves too much: under that model, no country would liberalize and democratize. I think a reasonable explanation is that the state’s repression was enough to effectively neuter people’s ability to coordinate: every communication technology was surveilled and censored.

Now, doesn’t this argument imply that, as a country’s employment moves towards services, the country’s politics should become more authoritarian, while in reality we observe the opposite? But whether a country’s economy is 5% or 95% services, the material side of the economy is equally vital. So “aggregate empowerment” remains as employment becomes more and more abstract. Consider France, where the economy is ~80% services, but the last 20% can still paralyze the country through strikes. In a sense every knowledge worker is free-riding on the ability of truck drivers and dock workers to shut down the economy. I don’t greatly like this counter-counterargument, though, because it’s hard to falsify: it implies that countries might remain liberal democracies until AGI replaces all humans in material work, at which point the state can go “that’s what we wanted you to think!” and go full authoritarian.