Any rigorous thinking about the long-term future leads to extreme outcomes. It cannot, by construction, produce normal or mundane futures.
“Rigorous thinking” means you have a model: a formalism for representing the state of the world, and a transition function to advance the state. You put the world into the formalism, and turn the crank to get successive states of the world. Models can be qualitative or quantitative. Most thinking about the future is qualitative and informal: people think in terms of trends over vague qualities like “intelligence”. For example, your model might be:
- Currently, humans are smarter than computers.
- Computers are getting smarter faster than humans are getting smarter.
From this, you can derive:
- At some point in the future, computers will match human intelligence, and thereafter exceed it.
- In the limit, and given reasonable assumptions about at the physics of information processing, computers will vastly exceed human intelligence.
But you can’t say when (1) or (2) will happen. So, a qualitative model can tell you the world will pass through a sequence of discrete states, but not when.
Quantitative models usually have better time resolution: you turn the crank until you reach some desired time t, and examine the state of the world. You can talk about both the journey and the destination. For obvious reasons, models that make detailed, fine-grained predictions only work for the near future.
But, in an existential sense, the only thing that matters is the ultimate future, “ultimate” not as in “long-term” but in the sense of a normal form: a state for which no more rewrite rules apply. So you keep turning the crank until the model stabilizes. And there are only two stable states: heaven and hell. It’s either the spike, von Neumann probes, uploaded minds living at a million subjective years per second, galaxies dimmed to infrared with Matrioshka brains, the Omega Point; or we crash and go to zero.