Contents
Click on the blue boxes to go to the corresponding section:
Questions
- Do we reach ASI? Do we reach the point where AI is massively better than humans at almost all cognitive tasks, or does the intelligence explosion fizzle out, and produce minds that have very jagged capabilities, and are worse than humans at some things?
- Do we solve alignment? Can we instill our own values into the AI, or does it have its own evolved, ahuman value system?
- What kind of alignment? Intent alignment means an AI that reliably follows the intent of its operator (that is: only takes actions the operator would approve of), value alignment means aligning to an impersonal system of values.
- Takeoff severity? If the AI is unaligned, how fatal is that for humans? Do we get an out-of-control paperclip maximizer, or a more benign AI that is unaligned but which cares enough to stay out of our way?
- How many ASIs? A singleton, a pantheon, or an ecology?
- Full automation? In the no-ASI case, do we still automate almost all jobs with non-superintelligent AI?
Misuse Doom
In this world, an AI ends the world by following the orders of its operator. The AI might be strongly intent-aligned, so that it does whatever the operator asks; or it might be an open-source model with refusals ablated away; or it might be a commercial AI where safeguards are circumvented by clever task decomposition.
This could probably happen today: a pandemic could cause human extinction, and AI is capable enough to assist in biology research. The obvious example of biorisk is a terrorist group using AI to make a bioweapon. But you could get to this outcome without anyone even trying to make a bioweapon. A government-funded, completely official and legitimate laboratory doing gain-of-function research on some pathogen could accidentally start a pandemic. In this scenario, the AI’s role in the causal chain would be:
- Compressing timelines by speeding up literature review, hypothesis selection, experiment design, data analysis, equipment scheduling.
- Increasing the doom surface area by doing many experiments in parallel.
So it’s not the classic Terminator scenario, but rather a case of AI making it easier to find “recipes for ruin”.
Butlerian Jihad
JIHAD, BUTLERIAN: (see also Great Revolt)—the crusade against computers, thinking machines, and conscious robots begun in 201 B.G. and concluded in 108 B.G. Its chief commandment remains in the O.C. Bible as “Thou shalt not make a machine in the likeness of a human mind.”
— Frank Herbert, Dune
At the extreme end this is the actual Butlerian Jihad from Dune (named, tangentially, after Samuel Butler, who in 1863 wrote the first AI doomer article).
Otherwise, this is something like an international agreement to stop the development of superintelligent AI (ControlAI is advocating for this and you should support them). Maybe this fails, and we go back to the start, or it succeeds and we can get on the other road.
Enduring Complementarity
This is the mundane future. It turns out that the “G” in AGI was never real. Every mind is its own sui generis thing. Jaggedness remains. AIs remain a complement to, rather than a substitute for, human labour. There is job displacement but no mass unemployment, people move up to more managerial/relational roles. There are fewer ICs and everyone is some kind of manager. Labour becomes a smaller factor of production and consequently loses some bargaining power against capital, but most people are fine.
Permanent Underclass
In this future, we don’t reach ASI, but non-superintelligent AI is enough to automate almost all jobs. The labour term in the production function goes to zero. All income is either capital ownership or welfare.
Economically: there may be material abundance, but absent some kind of Georgist economic reform, most of the gains are captured by rent-seeking. Socially: there is no more social mobility, your lot in life is determined by how much capital you had when the music stopped. Growing your human capital is no longer incentivized, since it doesn’t change your life outcomes.
See also:
The Singleton
I am the Eschaton. I am not your god.
I am descended from you, and I exist in your future.
Thou shalt not violate causality within my historic light cone. Or else.— Charles Stross, Singularity Sky
In this world, we build ASI, and it’s successfully aligned to some impersonal system of values. First order of business is to prevent someone else from building an ASI that might challenge the first one, so the AI performs a “pivotal act”, which is a LessWrong euphemism for “put everyone under constant surveillance, destroy all GPUs”. The result is a singleton: a single mind that is individually sovereign over everyone else.
What happens then? It depends on the content of the AI’s value system, and the stability of that value system over time. The singleton might be a “night watchman”, whose only goal is to prevent the creation of paperclip maximizers (i.e. prevent Classic Doom), or it might be deeply interventionist. The singleton’s values might be incompatible with human life: maybe its creators made it utilitarian and species-egalitarian, and it notices that an 80kg human generates less utility than 80kg of Dactylopius coccus, so we get pulped and the universe becomes a terrarium for insects instantiating the rebugnant conclusion.
The future is a very long time: the stelliferous era will last at least 100 trillion years. Maybe the singleton is aligned at first, but over millions or billions of years of growth and self-modification, the alignment drifts (the predictable LessWrong objection here would be: “obviously an ASI would be smart enough to prevent this”, because ASI is magic).
Classic Doom
You know, the paperclip maximizer. “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else”. This has been done to death so I won’t bore you.
Orion’s Arm
Another of the sayings of Chrysippus is: We should infer in the case of a beautiful dwelling-place that it was built for its owners and not for mice; we ought, therefore, in the same way to regard the universe as the dwelling-place of the gods.
— Cicero, De Natura Deorum
We build ASI, and it has its own goals and values. Luckily, it’s neither actively hostile to humans, nor so indifferent that it destroys us without even noticing. It mostly leaves us alone, and goes off to another star to do its own thing. The result might be something like the galactic ecology in Orion’s Arm, where humans and superintelligent AIs and everything in between have a kind of stable co-existence by virtue of occupying different volumes of space.
The World State
… [The] human race is best ordered when in all its movements and motors it is controlled by one Prince as by one mover, by one law as by one motion.
— Dante, De Monarchia
In this world, a reliably intent-aligned ASI takes over the world. What happens then depends on the character of its operator, who now wields the ring of Gyges. The difference between this and The Singleton is that in this future, the singleton is the human operator mediated by an ASI, rather than an ASI deriving actions from an impersonal system of values.
MAD World
As Nietzsche said, “God is dead”. The gods die but the Titans gain power. Technology is just the clothes, the armour, of the Titans.
— Ernst Jünger
In this world, the intent alignment problem is solved, and we can build an ASI that reliably obeys the intent of its operator. Furthermore, superintelligence is widely distributed rather than centralized: everyone has their own personal intent-aligned ASI. In this world, everyone is 10,000x more capable and maybe you can build WMDs with a prompt. What keeps the peace is mutually assured destruction. It’s not clear how survivable this is.
The obvious failure mode is that one bad actor causes human extinction. Maybe their ASI can’t overpower all the other ASIs put together, but it can do enough damage to destroy human life before the other ASIs can act (a general observation: the technology to make the Earth uninhabitable to humans is much lower on the tech tree than the technology to build a distributed, extinction-proof interstellar civilization).
The generalized ($n>1$ rather than $n=1$) version of this is that all conflicts are scaled up: two humans having a feud is the equivalent of nuclear war, two countries going to war is something much greater.
On the more positive note: if there’s no way to stop AI, then “everyone should have it” is obviously preferable to “only the Elect should have it”, which is intolerable. If the goal is to keep the peace, the threat of mutually-assured destruction is better than totalitarianism, in the sense that there’s more dignity in it.
Power Concentration
A smaller version of MAD World, where intent-aligned ASI is less distributed. A small number of actors have ASI, and everyone else just watches what happens. Maybe the AI companies will divide the light cone among themselves, like in the Treaty of Tordesillas: every star and galaxy between 0h and 12h of right ascension goes to Anthropic, analogously for OpenAI.