ASI as a Black Box

An artificial superintelligence (ASI) is an AI that’s much better than the top humans at almost all cognitive tasks. This is a perfectly fine definition. But, from this definition and from our web of knowledge, we can deduce very little about what an ASI would actually look like.

Human geniuses provide a lower bound, but not an upper bound. In some areas, we have deductive theories that describe the space of what is possible for all agents. We know, for example, that a superintelligent AI won’t be able to travel faster than light, or travel back in time, because we have well-trod theories that prohibit these things. In most areas of life, we don’t have such theories, and so we don’t have solid reasons to say what is or is not possible.

Consider persuasion. Some humans are very charismatic. But what is the range of charisma? Is superpersuasion possible? I don’t know. The term is very loosely defined, but people just throw the word around like it means something. Without a deductive theory of human belief-change and communication, the possibility of “superpersuasion” is an empirical question.

Consider scientific discovery. Maybe ASI would vastly accelerate science, or not. You can think of scientific discovery as a function of intelligence, the existing state of knowledge, and access to empirical observation. But we don’t know the coefficients of those terms, the—so to speak—intelligence elasticity of scientific discovery. I would guess that progress would accelerate, since greater intelligence means better hypothesis generation, hypothesis selection, experiment design, and more efficient scheduling of lab equipment.

More generally, for a question of the form “would an ASI be able to do X?”, our inability to answer the question usually stems, not from our lack of knowledge about ASI, but from our lack of knowledge about X. “Things an ASI can do” is a subset of “things it is possible to do”, to know the former (deductively rather than empirically) we first need to know the latter, and to do that we need a complete, deductive theory of whatever area of knowledge X belongs to.

But don’t we have all these mathematical theories of rationality and optimal decision-making? Are those useful for predicting how a superintelligence would behave? The standard doctrine says superintelligence should converge on perfect Bayesian rationality. If that’s true, then all those papers about decision theory and all that ink spilled on Newcomb’s problem will finally pay off. If, on the other hand, these theories are just an unreachable, abstract normative standard, then we know nothing.

In short: we can’t predict the specific cognitive abilities an ASI would have, and we also can’t predict the higher-order consequences of those cognitive abilities.