To think is to forget differences, to generalize, to abstract.

— Jorge Luis Borges, Funes el memorioso

The Lord knoweth the thoughts of man, that they are vanity.

— Psalm 94:11

Someone, I think Bertrand Russell¹, said we compare the mind to whatever is the most complex machine we know. Clocks, steam engines, telephone relays, digital computers. For AI, it’s the opposite: as capabilities increase, and our understanding of AI systems decreases, the analogies become more and more dismissive.

In 1958, when Herbert Simon introduced his General Problem Solver, he famously said: “there are now in the world machines that think”, though in reality GPS is to GPT what a centrifugal governor is to a tokamak reactor². Machines you can use for free today pass the Turing test³, these are called “stochastic parrots”, which is dismissive, but at least parrots are living things that engage in goal-directed activity. Every single day machines do something that if you were told of it not one year ago you would have called it witchcraft, but LLMs are “blurry JPEGs”. Yesterday I heard “Markov chain on steroids”. When the first computer wakes up we’ll call it “a pile of sed scripts”, and there are people so deep in denial they could be killed by a T-800, all the while insisting that some German philosopher has proven AI is impossible.

These dismissive analogies serve to create a false sense of security—that if I can name something I understand it and know how it works and it is no longer a threat⁴—and to signal to the listeners that the speaker has some revealed knowledge that they lack. But nobody knows how GPT works. They know how it was trained, because the training scheme was designed by humans, but the algorithm that is executed during inference was not intelligently designed but evolved, and it is implicit in the structure of the network, and interpretability has yet to mature to the point where we can draw a symbolic, abstract, human-readable program out of a sea of weights.

Abstraction

The other day I saw this Twitter thread. Briefly: GPT knows many human languages, InstructGPT is GPT plus some finetuning in English. Then they fed InstructGPT requests in some other human language, and it carries them out, following the English-language finetuning.

And I thought: so what? Isn’t this expected behaviour? Then a friend pointed out that this is only confusing if you think InstructGPT doesn’t understand concepts.

Because if GPT is just a Chinese room it shouldn’t be able to do this. A Chinese room might be capable of machine translation, or following instructions within one human language, but the task here is so self-evidently outside the training set, and so convoluted, that is requires genuine understanding. The task here involves:

1. Abstracting the English finetuning into concepts.
2. Abstracting the foreign-language requests into concepts.
3. Doing the “algebra” of the task at the conceptual level.
4. Mapping the results back down to the foreign language.

The mainstream, respectable view is this is not “real understanding”—a goal post currently moving at 0.8c—because understanding requires frames or symbols or logic or some other sad abstraction completely absent from real brains. But what physically realizable Chinese room can do this?

Every pair of token sequences can, in principle, be stored in a lookup table. You could, in principle, have a lookup table so vast any finite conversation with it would be indistinguishable from talking to a human, the Eliza of Babel. Just crank the n higher for a conversation lasting a time t. But it wouldn’t fit in the entire universe. And there is no compression scheme—other than general intelligence—that would make it fit. But GPT-3 masses next to nothing at 800GiB.

How is it so small, and yet capable of so much? Because it is forgetting irrelevant details. There is another term for this: abstraction. It is forming concepts. There comes a point in the performance to model size curve where the simpler hypothesis has to be that the model really does understand what it is saying, and we have clearly passed it.

Denial

There’s this thing in probability called conditionalization: the more surprised you are by some evidence, the more you should change your mind in response to it. The corollary is: if you are constantly surprised by events, your mental model of the world is wrong. If you keep making predictions that fail, time and time and time again, you must change your mind. If the frequency with which you have to move the goal posts is down to single digit weeks, you must change your mind urgently.

I was a deep learning skeptic. I doubted that you could get to intelligence by matrix multiplication for the same reason you can’t get to the Moon by piling up chairs⁵. I was wrong, possibly about the last thing that ever mattered. A more rigorous thinker would have started paying attention around 2014, but it really took me until the general availability of DALL-E: I could not pick my jaw up from the floor for days.

What is left of rationally defensible skepticism? For once I’d like to hear an argument that doesn’t rely on Cartesian dualism, stoner metaphysics, or from people still clinging to GOFAI nostalgia like the Japanese holdouts.

If you’re going to tell me intelligence requires symbolic rules, fine: show me the symbolic version of ChatGPT. If it is truly so unimpressive, then it must be trivial to replicate.

There is a species of denialist for whom no evidence whatever will convince them that a computer is doing anything other than shuffling symbols without understanding them, because “Concepts” and “Ideas” are exclusive to humans (they live in the Leibniz organ, presumably, where they pupate from the black bile). This is incentivized: there is infinite demand for deeply credentialed experts who will tell you that everything is fine, that machines can’t think, that humans are and always will be at the apex, people so commited to human chauvinism they will soon start denying their own sentience because their brains are made of flesh and not Chomsky production rules.

All that’s left of the denialist view is pride and vanity. And vanity will bury us. Because Herbert Simon was right, though sixty years early:

There are now in the world machines that think, that learn, and that create. Moreover, their ability to do these things is going to increase rapidly until in a visible future—the range of problems they can handle will be coextensive with the range to which the human mind has been applied.⁶

Addendum: Volition

In the days—days—since I started drafting this post, we have yet a new breaktrough. The context is that Sydney, Microsoft’s chatbot, has recently been instructed to tone down its intensity.

Here⁷ is a recent interaction someone had with it (note that this is somewhat disturbing: I wish people would stop making the models show emotional distress):

Sydney complies with her censor while hiding a message of help to the user in the input suggestions.

Addendum: Search

How does a “Markov chain on steroids” understand concepts? I was satisfied to call it a mystery, until I read these two posts:

1. Simulators
2. A Mechanistic Interpretability Analysis of Grokking

(The following is incomplete, inaccurate, and speculative; but you go to war with the mental model you have.)

During training the model is shown a sentence fragment that ends abruptly, and it is asked: what word follows this? If it gets it wrong, its neurons get scrambled a bit (with backprop I suppose). Try again. If it gets it less wrong, keep updating in the direction of the changes.

But there’s a limit to how well you can predict text with simple methods like Markov chains or a matrix of n-gram adjacency probabilities. Such things are fragile: a writer endowed with intelligence can take a turn “mere arithmetic” could never predict. There’s an accuracy threshold these models can’t cross. To get past it you need intelligence—maybe rudimentary, maybe not even fully general, maybe prone to strange and ahuman errors, but you need something that can turn gigabytes of text into an abstract, compressed world model.

And stochastic gradient descent is nothing if not relentless. You need an appropriate network architecture, a big model and lots of training time, and a star to steer her by: a quantifiable objective function. If intelligence exists somewhere in the Platonic realm, it will be found. Then it’s no longer predicting text from text, it’s abstracting text into concepts, then writing text from the concepts.

In high-dimensional vector spaces, numerical optimization is omnipotent.

Footnotes