AI models are very capable and get more capable each year. So naturally people feel they’re underusing them. There’s a tweet that goes like: your laptop has a 100M USD startup in it, you just have to figure the right sequence of words to get it out. And beyond money, people imagine AI could boost them in every area of life. Thus all these perennial ideas: of an AI executive assistant, an AI tutor, an AI that curates your “digital garden”, an AI that (sigh) writes flashcards for you.

The general template is: if only I could wire up the right prompts and the right tools in the right harness, I could have an agent that would boost my productivity 10x, or fix my problems with therapy, or make me more social, or more knowledgeable. This was, curiously, the ambition of a lot of early computing pioneers: Augmenting Human Intellect, Man-Computer Symbiosis. Engelbart’s lab was called the Augmentation Research Center! And more recently, people used to complain about how everyone has the Library of Alexandria in their pocket, and yet, we are not all genius polymaths.

And these ideas are perennial because they never seem to happen. It’s like the Solow paradox on an individual level. Why? I think there are two reasons: first, most people lack what Andy Matuschak calls a “serious context of use” (AI doesn’t move the needle because there’s no needle to move); second, most people are bottlenecked by internal factors where AI (or anything external, for that matter) can’t move the needle.

The Serious Context of Use

I have heard so many people, online and in real life, tell me some variation of: “I want to [use|build] an app that uses AI to write flashcards”. How many of those people do you think have ever written a flashcard? How many of them use Anki every day? Have ever used Anki? The people who want an AI that writes flashcards for them don’t use flashcards. They have no reason to. Dually, the people who use flashcards would benefit little from AI writing their flashcards.

Analogously with “AI tutors”. If you had the ghost of John von Neumann in your laptop, what would you have him teach you? Let’s be honest. You’d go through chapter one of some math topic you’re vaguely curious about and then forget about it. And that would probably be the rational move! Most people are not autodidacts because most people have no material reason to learn a specific topic (i.e. their job does not require it) and the problem with learning for the sake of learning is opportunity cost: there is no a priori reason to learn one thing over another, so better to do nothing and wait for something to appear which actually grabs your interest. Again, this is likely rational! Could you imagine if you found everything interesting? You’d spend years living in a basement curating a wiki of late Soviet military hardware or something. So, even if you had John von Neumann in your pocket, it probably wouldn’t move the needle.

Would an “AI executive assistant” actually boost your productivity? What would it do, other than tell you to do the things you already know you have to do? With these ideas that are so attractive in the abstract, the way you deflate them is you interrogate the concrete, fine-grained details. Take a day at work, and ask: what exact actions could an AI looking over my shoulder have taken, that would have made a difference?

Finally there’s the tools-for-thought/notetaking people. God save us. It’s always the same thing. Your folder with notes—pardon me, your “second brain”—plus an AI agent that writes, edits, synthesizes information, answers queries. You could build this in an afternoon, and it won’t move the needle in your life, for the same reason that building the second brain in the first place didn’t make a difference.

See, most of us, unless we are students, we really don’t have cause to take notes on anything. If you’re a student, you take notes from the textbook. I keep a journal, which is occasionally useful. At some jobs I’ve kept a work journal, this has also been useful. If I stopped, probably, not much would change.

The notetaking people—and I say this with all the love in the world—are never, like, a researcher at the cutting edge of their field, building this vast cathedral of knowledge, note-by-note, so they can derive new insights. Never a historian who has to read tens of millions of words across thousands of sources to synthesize the life of some historical person. It’s never someone doing something hard. It’s always some blogger. Their “digital garden” is about how to keep a digital garden. It’s very solipsistic: there’s no output, no deliverables. The deliverable is you take a screenshot of your Obsidian graph and tweet about it to show off how much it looks like an incomprehensible ball of twine.

So, what difference is the AI going to make? “It’s going to write my notes”. About what? “It’s going to read articles for me and summarize them and add them to the digital garden”. For what purpose? “It’s going to find connections between my ideas!” What ideas? It’s going to pull an unfinished list of bulletpoints for an eventual draft of an essay on some inane thing, plus a bunch of PDFs you haven’t read, and combine them together and make, what? Another project you’re not going to do? The AI is going to do that?

Again, I say this with all the love in the world. I used to be a tools-for-thought guy. I hoard PDFs. But we have to be honest with ourselves. Sometimes, tools don’t move the needle because there’s no needle to move. Because the “needle” is not a concrete, realizable material need but a vague, aspirational idea about who we are as people.

Internal Limiting Factors

So, the idea of using computers to augment human capabilities is basically: you take the human, and you build a scaffold around them, but the human stays the same. The scaffold can be classical software, or AI, but the human remains a black box. And the hope is: I just prompt my swarm of AI agents and become 100x more effective, like Manfred in Accelerando.

Why wouldn’t this work? I think most people are bottlenecked by internal factors that are difficult to change. Mental energy, motivation, executive function, not to mention more fundamental traits like intelligence and conscientiousness. So, external scaffolding, either with classical software or AI, might help somewhat, but it won’t be transformative.

Consider executive function. My own experience of managing ADHD is the external scaffolding helps (todo lists, calendars, timers, a million little ways to trick myself into working) gets me from zero to “kind of functional”. But it saturates there. Stimulants fix the original, internal bottleneck, which is my neurochemistry. And then I can accomplish my goals (c.f. Liebig’s law of the minimum). All the pomodoros in the world are as nothing to a little molecule diffusing through my brain tissue, binding to NET and DAT. And what scaffolding is useful is just classical software: Todoist and calendars. Is an agent going to match the effectiveness of methylphenidate in ADHD? I doubt it.

Consider intelligence. Can AI augment human intelligence? What does that look like? Consider how AI agents only became useful when the models crossed a particular capability threshold, i.e., you can’t put GPT-2 into a harness and get GPT-5 outcomes out of it. Can you put a human in an AI scaffold and give them +30 effective IQ? I doubt it, unless the AI is doing all the thinking, at which point, what use is the human? The limiting factor in the human-AI centaur is the human! So intelligence is fixed until we get very advanced biotechnology.

Knowledge is another limiting factor. I find that even very educated people tend to underrate the importance of knowledge. A lot of people have this attitude that you can just Google everything just-in-time as it comes up. Like Babbage, I can’t rightly apprehend the confusion of ideas that would lead someone to think this. Maybe it’s downstream of the lack of a serious context of use. Everything you do, every action and idle thought, draws on this vast (implicit, unseen) trove of knowledge. Claude Shannon invented digital computing because he remembered this then-obscure branch of mathematical logic called Boolean algebra and saw that it could be realized in hardware. A trillion-dollar industry, conjured out of some old tomes.

The reason knowledge is still a bottleneck, in the AI era, is not: “if you don’t have the knowledge, you can’t write the prompt”. Rather: if you don’t have the knowledge, you don’t understand the question, or why it matters, or how to judge the answers, and you won’t ever think to ask. You’re in a completely different continent from “writing the prompt”. And because long-term memory is private and internal, AI can’t boost it. It can, maybe, with judicious use, help in the acquisition of new knowledge.

So: executive function, intelligence, and knowledge are huge bottlenecks to what you can do, and because they are internal to the brain, AI can’t touch them until we have far more advanced biotechnology. Corollary: contrary to the popular view of human capital is becoming worthless, the returns to education are now higher, because intelligent, educated people with working reward circuitry stand to gain more from AI.