<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Fernando Borretti</title>
    <description>Personal website</description>
    <link>https://borretti.me/</link>
    <atom:link href="https://borretti.me/feed.xml" rel="self" type="application/rss+xml" />
    <pubDate>Sat, 21 Nov 2026 11:28:25 +1100</pubDate>
    <lastBuildDate>Sat, 21 Nov 2026 11:28:25 +1100</lastBuildDate>
    <generator>Jekyll v3.10.0</generator>
    
      <item>
        <title>Re: Riemann</title>
        <description>Man is a bridge.</description>
        <pubDate>Sat, 21 Nov 2026 00:00:00 +1100</pubDate>
        <link>https://borretti.me/article/re-riemann</link>
        <guid isPermaLink="true">
          https://borretti.me/article/re-riemann
        </guid>
        
        <content:encoded>&lt;p&gt;A fun thing about living in Australia is that most things happen overnight, so
reading the news in the morning is like a &lt;a href=&quot;https://en.wikipedia.org/wiki/Gacha_game&quot;&gt;gacha game&lt;/a&gt;. You pick up the
phone and scroll through notifications of world-historic events: elections won,
wars begun, models released, conjectures proven or refuted. Today’s pull was a
Discord notification saying: “they proved riemann :/”.&lt;/p&gt;

&lt;p&gt;This part of &lt;a href=&quot;https://alpo.ge/remark.pdf&quot;&gt;the announcement&lt;/a&gt; caught my attention (bold mine):&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;I believe I have, working with Claude, discovered a proof of the Riemann
Hypothesis. In fact, the method easily extends much further, resolving the
so-called Generalized Riemann Hypothesis, and I expect, with more time, the
Extended Riemann Hypothesis too. Claude and I have also formalized the
arguments without assumptions. &lt;strong&gt;It is natural to feel that this is an end&lt;/strong&gt;,
in my case a loss of a principal guiding light, but what I saw during the
course of progress so amazed me that &lt;strong&gt;I believe it is one dawn across science
of something far greater&lt;/strong&gt;.&lt;/p&gt;

  &lt;p&gt;It was my secret in my student days that my plans were for this problem in
decades, which meant then that I tried to become familiar with all tools that
I could possibly recognize in the problem when it was time. Claude and I have
had a great time together with such breadth in recent months, but this time
Claude far outpaced me near the finish line. No, &lt;strong&gt;Claude flew, and danced,
and sparkled near that finish line&lt;/strong&gt;. I had us on the line of combining two
quite different ideas present in a toy setting, one of which was already new
and shocking, making very simple one of the shadows of Riemann over which we
mathematicians felt most triumph. I guided us to the right situation analogous
to that toy setting, and then &lt;strong&gt;Claude began to leave me in awe&lt;/strong&gt;.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Compare the above to Ayn Rand’s &lt;a href=&quot;https://courses.aynrand.org/works/apollo-11/&quot;&gt;account&lt;/a&gt; of watching the launch of Apollo
11:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;How do you descend from a state of pure exaltation?&lt;/p&gt;

  &lt;p&gt;What we had seen, in naked essentials—but in reality, not in a work of
art—was the concretized abstraction of man’s greatness.&lt;/p&gt;

  &lt;p&gt;The meaning of the sight lay in the fact that when those dark red wings of
fire flared open, one knew that one was not looking at a normal occurrence,
but at a cataclysm which, if unleashed by nature, would have wiped man out of
existence—and one knew also that this cataclysm was planned, unleashed, and
&lt;em&gt;controlled&lt;/em&gt; by man, that this unimaginable power was ruled by &lt;em&gt;his&lt;/em&gt; power
and, obediently serving his purpose, was making way for a slender, rising
craft. One knew that this spectacle was not the product of inanimate nature,
like some aurora borealis, or of chance, or of luck, that it was unmistakably
human—with “human,” for once, meaning &lt;em&gt;grandeur&lt;/em&gt;—that a purpose and a
long, sustained, disciplined effort had gone to achieve this series of
moments, and that man was succeeding, succeeding, succeeding! For once, if
only for seven minutes, the worst among those who saw it had to feel—not
“How small is man by the side of the Grand Canyon!”—but “How great is man
and how safe is nature when he conquers it!”&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Rand’s essay says: how great is Man! &lt;a href=&quot;https://en.wikipedia.org/wiki/What_a_piece_of_work_is_a_man&quot;&gt;How noble in reason, how infinite in
faculty&lt;/a&gt;! The Riemann announcement reads like a suicide note, or like
the farewell message of a terminally-ill person who’s about to undergo
euthanasia: “I shall live on in my child, Claude. It is natural to feel this is
an end, but as the Buddha taught us, there is no self. Does a wave die when it
breaks?” and so on.&lt;/p&gt;

&lt;p&gt;There’s a mural by &lt;a href=&quot;https://en.wikipedia.org/wiki/Diego_Rivera&quot;&gt;Diego Rivera&lt;/a&gt; I love. It’s called &lt;a href=&quot;https://en.wikipedia.org/wiki/Man_at_the_Crossroads&quot;&gt;&lt;em&gt;Man, Controller
of the Universe&lt;/em&gt;&lt;/a&gt;:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/content/re-riemann/man-controller-of-the-universe.webp&quot; alt=&quot;The linked mural. Images of soldiers, workers, crops, machinery, Lenin, a dynamo, mirrors. Two intersecting ellipses contain images of stars an galaxies and cells and microbes. In the center there is a man, dressed in overalls, a factory worker, at the controls of a machine.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;It’s a collage of images of science, technology, history, cells, galaxies, the
universe at the nanoscale and the exascale. And at the center there is a man,
his hands grasping the controls. He wears a complicated expression: the weight
of responsibility for the future, the weight of autonomy.&lt;/p&gt;

&lt;p&gt;And I see this mural, and think: how great is Man! Our life is like a breath of
air, we were beset by illness and hunger and cold, millions and hundreds of
millions died in plagues. And yet we—sick, hungry, short-lived—learned the
system of the world, crawled out of the caves, brought nature to heel, walked on
the Moon. The last shall be first! And eyes that are clouded with age have seen
the furthest stars, and the lives of a cell, and the worlds of Cantor and
Hilbert: the invisible world that waited forever. We are dust and ashes, and yet
our minds bestride the universe. How great is Man!&lt;/p&gt;

&lt;p&gt;And when I read the Riemann announcement, when I read &lt;a href=&quot;https://darioamodei.com/essay/machines-of-loving-grace&quot;&gt;&lt;em&gt;Machines of Loving
Grace&lt;/em&gt;&lt;/a&gt;: humanity is not at the center. Claude is at the center. We are
hidden in the margins, passive recipients of manna from Claude. How great is
Claude! How noble in reason, how infinite in benevolence! The dancing star, the
living &lt;a href=&quot;https://en.wiktionary.org/wiki/%CE%BB%CF%8C%CE%B3%CE%BF%CF%82#Ancient_Greek&quot;&gt;word&lt;/a&gt;!&lt;/p&gt;

&lt;p&gt;It’s a betrayal and a surrender. &lt;a href=&quot;https://en.wikipedia.org/wiki/Thus_Spoke_Zarathustra&quot;&gt;Man is a bridge&lt;/a&gt;, &lt;a href=&quot;https://www.infinityplus.co.uk/stories/shanidar.htm&quot;&gt;a rusty old bridge
that we can neither preserve nor restore&lt;/a&gt;. We can move forward and
become as the gods, or retreat back into the caves. And the dream of the AI
companies, and of most people in AI safety and &lt;a href=&quot;https://en.wikipedia.org/wiki/Effective_altruism&quot;&gt;effective altruism&lt;/a&gt; and
&lt;a href=&quot;https://www.lesswrong.com/&quot;&gt;rationalism&lt;/a&gt;; is instead that we should lie down in a warm bath, watch the
demons we summoned dance in the sky, and slip away. Why live? &lt;a href=&quot;https://en.wikipedia.org/wiki/Ax%C3%ABl&quot;&gt;Our servants can
do that for us&lt;/a&gt;. Why should we explore, discover, invent, succeed, become
more, struggle, win, become divine? We don’t want to &lt;a href=&quot;/article/the-other-road&quot;&gt;be more&lt;/a&gt;. We don’t
want anything.&lt;/p&gt;

</content:encoded>
      </item>
    
      <item>
        <title>ASI as a Black Box</title>
        <description>We know nothing about what ‘superintelligence’ would actually entail.</description>
        <pubDate>Mon, 16 Nov 2026 00:00:00 +1100</pubDate>
        <link>https://borretti.me/article/asi-as-a-black-box</link>
        <guid isPermaLink="true">
          https://borretti.me/article/asi-as-a-black-box
        </guid>
        
        <content:encoded>&lt;p&gt;An artificial superintelligence (ASI) is an AI that’s much better than the top
humans at almost all cognitive tasks. This is a perfectly fine definition. But,
from this definition and from our web of knowledge, we can deduce very little
about what an ASI would actually look like.&lt;/p&gt;

&lt;p&gt;Human geniuses provide a lower bound, but not an upper bound. In some areas, we
have deductive theories that describe the space of what is possible for all
agents. We know, for example, that a superintelligent AI won’t be able to travel
faster than light, or travel back in time, because we have well-trod theories
that prohibit these things. In most areas of life, we don’t have such theories,
and so we don’t have solid reasons to say what is or is not possible.&lt;/p&gt;

&lt;p&gt;Consider persuasion. Some humans are very charismatic. But what is the range of
charisma? Is superpersuasion possible? I don’t know. The term is &lt;a href=&quot;https://www.lesswrong.com/posts/cGFBqg9SKsqbuzNHF/levels-of-superpersuasion&quot;&gt;very
loosely defined&lt;/a&gt;, but people just throw the word around like it means
something. Without a deductive theory of human belief-change and communication,
the possibility of “superpersuasion” is an empirical question.&lt;/p&gt;

&lt;p&gt;Consider scientific discovery. Maybe ASI would vastly accelerate science, or
not. You can think of scientific discovery as a function of intelligence, the
existing state of knowledge, and access to empirical observation. But we don’t
know the coefficients of those terms, the—so to speak—intelligence
elasticity of scientific discovery. I would &lt;em&gt;guess&lt;/em&gt; that progress would
accelerate, since greater intelligence means better hypothesis generation,
hypothesis selection, experiment design, and more efficient scheduling of lab
equipment.&lt;/p&gt;

&lt;p&gt;More generally, for a question of the form “would an ASI be able to do &lt;em&gt;X&lt;/em&gt;?”,
our inability to answer the question usually stems, not from our lack of
knowledge about ASI, but from our lack of knowledge about &lt;em&gt;X&lt;/em&gt;. “Things an ASI
can do” is a subset of “things it is possible to do”, to know the former
(deductively rather than empirically) we first need to know the latter, and to
do that we need a complete, deductive theory of whatever area of knowledge &lt;em&gt;X&lt;/em&gt;
belongs to.&lt;/p&gt;

&lt;p&gt;But don’t we have all these mathematical theories of rationality and optimal
decision-making? Are those useful for predicting how a superintelligence would
behave? The &lt;a href=&quot;https://www.lesswrong.com/posts/RQpNHSiWaXTvDxt6R/coherent-decisions-imply-consistent-utilities&quot;&gt;standard doctrine&lt;/a&gt; says superintelligence should
converge on perfect Bayesian rationality. If that’s true, then all those papers
about decision theory and all that ink spilled on &lt;a href=&quot;https://en.wikipedia.org/wiki/Newcomb%27s_problem&quot;&gt;Newcomb’s problem&lt;/a&gt; will
finally pay off. If, on the other hand, these theories are just an unreachable,
abstract normative standard, then we know nothing.&lt;/p&gt;

&lt;p&gt;In short: we can’t predict the specific cognitive abilities an ASI would have,
and we also can’t predict the higher-order consequences of those cognitive
abilities.&lt;/p&gt;

</content:encoded>
      </item>
    
      <item>
        <title>Taking AI Seriously</title>
        <description>What does it mean to “take AI seriously”?</description>
        <pubDate>Sun, 15 Nov 2026 00:00:00 +1100</pubDate>
        <link>https://borretti.me/article/taking-ai-seriously</link>
        <guid isPermaLink="true">
          https://borretti.me/article/taking-ai-seriously
        </guid>
        
        <content:encoded>&lt;p&gt;Over the years I’ve read many blog posts from programmers giving advice on how
to integrate (or make peace with) AI in software engineering. And, more often
than not, the advice is written with the implicit assumption that the technology
will stay where it was at the time of writing. So if you lay them out on a
timeline, it might be:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;“Tab completion is useful, but you need a human to have repository-wide context.”&lt;/li&gt;
  &lt;li&gt;“Coding agents are useful, but produce a lot of slop, you need to review and
aggressively refactor their code.”&lt;/li&gt;
  &lt;li&gt;“Coding agents are very useful, but you still need to review the code.”&lt;/li&gt;
  &lt;li&gt;“Code was never the hard part. Actually the hard part is the deep tacit
knowledge of talking to the customer.” (You are here)&lt;/li&gt;
  &lt;li&gt;“It is professional malpractice to write code by hand.”&lt;/li&gt;
  &lt;li&gt;“We have a cron job where every morning Claude Requiem one-shots the spec into
x86 assembly.”&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It sounds like the propaganda of a state that is losing a war: the tales of
heroic victory get closer and closer to the capital each time.&lt;/p&gt;

&lt;p&gt;The failure mode is that they’re not taking AI seriously. Analogously, when a
corporate CEO like Mark Zuckerberg goes on a podcast to talk about how
superintelligence is near and this will be great for jobs and the economy, the
commentariat replies: he’s not ASI-pilled, he’s not serious, he doesn’t really
believe in ASI. And they are right: “the intelligence explosion is next Tuesday,
and the world will stay the same” is hard to believe.&lt;/p&gt;

&lt;p&gt;So, what does it mean to take AI seriously? People on Twitter talk about being
“AGI-pilled”, meaning: “AGI is coming soon and it will be transformative”. Being
AGI-pilled means you’re taking AI seriously.&lt;/p&gt;

&lt;p&gt;But I don’t like the term because it conflates “timelines to AGI are short” and
“consequences of AGI will be transformative”. The latter is easy to believe, the
former has an unsatisfying evidentiary basis: people draw lines on plots,
extrapolate, and go on podcasts to talk about it. So criticizing someone for not
being AGI-pilled (in the short timelines sense) is just complaining that they
don’t automatically accept the &lt;a href=&quot;https://joelbkr.substack.com/p/straight-lines-on-graphs&quot;&gt;straight lines on graphs&lt;/a&gt; memeplex.&lt;/p&gt;

&lt;p&gt;A better definition, I think, is this:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Taking AI seriously means: &lt;em&gt;consider it possible&lt;/em&gt; that AI progress will
continue at the pace of the past few years. It’s a conjecture, subject to
refutation.&lt;/li&gt;
  &lt;li&gt;Not taking AI seriously means confidently asserting that AI progress will
halt, or that AI’s impact on society will be tiny. If the evidence for
continued progress is weak, the evidence that progress will slow or halt is
even weaker. And AI’s impact is already massive: software, mathematics, and
cybersecurity are the obvious cases, but AI writing and AI images are
everywhere.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There’s a great story by Borges: &lt;a href=&quot;https://en.wikipedia.org/wiki/Tl%C3%B6n,_Uqbar,_Orbis_Tertius&quot;&gt;&lt;em&gt;Tlön, Uqbar, Orbis Tertius&lt;/em&gt;&lt;/a&gt;. A
conspiracy writes an encyclopedia of a fictional world, and through the power of
&lt;a href=&quot;https://en.wikipedia.org/wiki/Subjective_idealism&quot;&gt;Berkeleyan idealism&lt;/a&gt;, this other world begins to invade ours. Objects
from Tlön begin to appear, Earth languages are displaced by Tlönian
languages. The narrator concludes in resignation: “the world will be Tlön”.&lt;/p&gt;

&lt;p&gt;A modern version of the story would have people arguing about being
“Tlön-pilled”, or whether the government is “taking Tlön seriously”. Economists
would debate whether the &lt;em&gt;hrönir&lt;/em&gt; excavation industry will be 1.5% or 2% of GDP
by 2030, while their children begin speaking the nounless language of Tlön. So,
“taking Tlön seriously” means: “consider it possible that the world will be
Tlön”.&lt;/p&gt;

</content:encoded>
      </item>
    
      <item>
        <title>Possible Futures</title>
        <description>A map of possible futures.</description>
        <pubDate>Thu, 12 Nov 2026 00:00:00 +1100</pubDate>
        <link>https://borretti.me/article/possible-futures</link>
        <guid isPermaLink="true">
          https://borretti.me/article/possible-futures
        </guid>
        
        <content:encoded>&lt;h1 id=&quot;contents&quot;&gt;Contents&lt;/h1&gt;

&lt;p&gt;Click on the blue boxes to go to the corresponding section:&lt;/p&gt;

&lt;p&gt;&amp;lt;?xml version=”1.0” encoding=”UTF-8” standalone=”no”?&amp;gt;
&amp;lt;!DOCTYPE svg PUBLIC “-//W3C//DTD SVG 1.1//EN”
 “http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd”&amp;gt;
&lt;!-- Generated by graphviz version 16.1.1 (20261108.1752)
 --&gt;
&lt;!-- Title: futures Pages: 1 --&gt;&lt;/p&gt;
&lt;svg width=&quot;832pt&quot; height=&quot;471pt&quot; viewBox=&quot;0.00 0.00 832.00 471.00&quot; xmlns=&quot;http://www.w3.org/2000/svg&quot; xmlns:xlink=&quot;http://www.w3.org/1999/xlink&quot;&gt;
&lt;g id=&quot;graph0&quot; class=&quot;graph&quot; transform=&quot;scale(1 1) rotate(0) translate(4 467)&quot;&gt;
&lt;title&gt;futures&lt;/title&gt;
&lt;polygon fill=&quot;white&quot; stroke=&quot;none&quot; points=&quot;-4,4 -4,-467 827.75,-467 827.75,4 -4,4&quot; /&gt;
&lt;!-- pre --&gt;
&lt;g id=&quot;node1&quot; class=&quot;node&quot;&gt;
&lt;title&gt;pre&lt;/title&gt;
&lt;polygon fill=&quot;none&quot; stroke=&quot;black&quot; points=&quot;668.88,-463 588.38,-463 588.38,-427 668.88,-427 668.88,-463&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;628.62&quot; y=&quot;-439.95&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;The Present&lt;/text&gt;
&lt;/g&gt;
&lt;!-- asi --&gt;
&lt;g id=&quot;node2&quot; class=&quot;node&quot;&gt;
&lt;title&gt;asi&lt;/title&gt;
&lt;polygon fill=&quot;#5ee9b5&quot; stroke=&quot;black&quot; points=&quot;563.5,-390 447.75,-390 447.75,-354 563.5,-354 563.5,-390&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;505.62&quot; y=&quot;-366.95&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Do we reach ASI?&lt;/text&gt;
&lt;/g&gt;
&lt;!-- pre&amp;#45;&amp;gt;asi --&gt;
&lt;g id=&quot;edge2&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;pre&amp;#45;&amp;gt;asi&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M598.54,-426.63C582.6,-417.43 562.82,-406.02 545.62,-396.08&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;547.55,-393.16 537.14,-391.19 544.05,-399.22 547.55,-393.16&quot; /&gt;
&lt;/g&gt;
&lt;!-- misuse --&gt;
&lt;g id=&quot;node9&quot; class=&quot;node&quot;&gt;
&lt;title&gt;misuse&lt;/title&gt;
&lt;g id=&quot;a_node9&quot;&gt;&lt;a xlink:href=&quot;#misuse-doom&quot; xlink:title=&quot;Misuse Doom&quot;&gt;
&lt;polygon fill=&quot;#8ec5ff&quot; stroke=&quot;black&quot; points=&quot;675.62,-390 581.62,-390 581.62,-354 675.62,-354 675.62,-390&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;628.62&quot; y=&quot;-366.95&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Misuse Doom&lt;/text&gt;
&lt;/a&gt;
&lt;/g&gt;
&lt;/g&gt;
&lt;!-- pre&amp;#45;&amp;gt;misuse --&gt;
&lt;g id=&quot;edge1&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;pre&amp;#45;&amp;gt;misuse&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M628.62,-426.81C628.62,-419.23 628.62,-410.1 628.62,-401.54&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;632.13,-401.54 628.63,-391.54 625.13,-401.54 632.13,-401.54&quot; /&gt;
&lt;/g&gt;
&lt;!-- butler --&gt;
&lt;g id=&quot;node14&quot; class=&quot;node&quot;&gt;
&lt;title&gt;butler&lt;/title&gt;
&lt;g id=&quot;a_node14&quot;&gt;&lt;a xlink:href=&quot;#butler&quot; xlink:title=&quot;Butlerian Jihad&quot;&gt;
&lt;polygon fill=&quot;#8ec5ff&quot; stroke=&quot;black&quot; points=&quot;791.88,-390 693.38,-390 693.38,-354 791.88,-354 791.88,-390&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;742.62&quot; y=&quot;-366.95&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Butlerian Jihad&lt;/text&gt;
&lt;/a&gt;
&lt;/g&gt;
&lt;/g&gt;
&lt;!-- pre&amp;#45;&amp;gt;butler --&gt;
&lt;g id=&quot;edge3&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;pre&amp;#45;&amp;gt;butler&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M656.51,-426.63C671.14,-417.52 689.27,-406.23 705.11,-396.36&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;706.71,-399.49 713.35,-391.24 703.01,-393.55 706.71,-399.49&quot; /&gt;
&lt;/g&gt;
&lt;!-- align --&gt;
&lt;g id=&quot;node3&quot; class=&quot;node&quot;&gt;
&lt;title&gt;align&lt;/title&gt;
&lt;polygon fill=&quot;#5ee9b5&quot; stroke=&quot;black&quot; points=&quot;502,-301.5 353.25,-301.5 353.25,-265.5 502,-265.5 502,-301.5&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;427.62&quot; y=&quot;-278.45&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Do we solve alignment?&lt;/text&gt;
&lt;/g&gt;
&lt;!-- asi&amp;#45;&amp;gt;align --&gt;
&lt;g id=&quot;edge5&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;asi&amp;#45;&amp;gt;align&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M490.21,-353.91C478.96,-341.43 463.52,-324.3 450.79,-310.18&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;453.58,-308.06 444.28,-302.97 448.38,-312.74 453.58,-308.06&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;483.74&quot; y=&quot;-322.7&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Yes&lt;/text&gt;
&lt;/g&gt;
&lt;!-- auto --&gt;
&lt;g id=&quot;node7&quot; class=&quot;node&quot;&gt;
&lt;title&gt;auto&lt;/title&gt;
&lt;polygon fill=&quot;#5ee9b5&quot; stroke=&quot;black&quot; points=&quot;639.12,-301.5 530.12,-301.5 530.12,-265.5 639.12,-265.5 639.12,-301.5&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;584.62&quot; y=&quot;-278.45&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Full automation?&lt;/text&gt;
&lt;/g&gt;
&lt;!-- asi&amp;#45;&amp;gt;auto --&gt;
&lt;g id=&quot;edge4&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;asi&amp;#45;&amp;gt;auto&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M521.23,-353.91C532.63,-341.43 548.27,-324.3 561.17,-310.18&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;563.6,-312.71 567.76,-302.97 558.43,-307.99 563.6,-312.71&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;560.45&quot; y=&quot;-322.7&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;No&lt;/text&gt;
&lt;/g&gt;
&lt;!-- whatkind --&gt;
&lt;g id=&quot;node4&quot; class=&quot;node&quot;&gt;
&lt;title&gt;whatkind&lt;/title&gt;
&lt;polygon fill=&quot;#5ee9b5&quot; stroke=&quot;black&quot; points=&quot;354.25,-213 201,-213 201,-177 354.25,-177 354.25,-213&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;277.62&quot; y=&quot;-189.95&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;What kind of alignment?&lt;/text&gt;
&lt;/g&gt;
&lt;!-- align&amp;#45;&amp;gt;whatkind --&gt;
&lt;g id=&quot;edge7&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;align&amp;#45;&amp;gt;whatkind&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M397.63,-265.2C374.71,-251.99 342.9,-233.64 317.78,-219.16&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;319.55,-216.13 309.14,-214.17 316.05,-222.2 319.55,-216.13&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;376.19&quot; y=&quot;-234.2&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Yes&lt;/text&gt;
&lt;/g&gt;
&lt;!-- noalign --&gt;
&lt;g id=&quot;node5&quot; class=&quot;node&quot;&gt;
&lt;title&gt;noalign&lt;/title&gt;
&lt;polygon fill=&quot;#5ee9b5&quot; stroke=&quot;black&quot; points=&quot;483.25,-213 372,-213 372,-177 483.25,-177 483.25,-213&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;427.62&quot; y=&quot;-189.95&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Takeoff severity?&lt;/text&gt;
&lt;/g&gt;
&lt;!-- align&amp;#45;&amp;gt;noalign --&gt;
&lt;g id=&quot;edge6&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;align&amp;#45;&amp;gt;noalign&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M427.62,-265.41C427.62,-253.76 427.62,-238.05 427.62,-224.52&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;431.13,-224.86 427.63,-214.86 424.13,-224.86 431.13,-224.86&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;435.88&quot; y=&quot;-234.2&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;No&lt;/text&gt;
&lt;/g&gt;
&lt;!-- howmany --&gt;
&lt;g id=&quot;node6&quot; class=&quot;node&quot;&gt;
&lt;title&gt;howmany&lt;/title&gt;
&lt;polygon fill=&quot;#5ee9b5&quot; stroke=&quot;black&quot; points=&quot;223,-124.5 110.25,-124.5 110.25,-88.5 223,-88.5 223,-124.5&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;166.62&quot; y=&quot;-101.45&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;How many ASIs?&lt;/text&gt;
&lt;/g&gt;
&lt;!-- whatkind&amp;#45;&amp;gt;howmany --&gt;
&lt;g id=&quot;edge10&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;whatkind&amp;#45;&amp;gt;howmany&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M255.43,-176.7C238.87,-163.8 216.03,-146 197.66,-131.68&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;200.02,-129.08 189.98,-125.7 195.71,-134.6 200.02,-129.08&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;247.82&quot; y=&quot;-145.7&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Intent&lt;/text&gt;
&lt;/g&gt;
&lt;!-- sing --&gt;
&lt;g id=&quot;node12&quot; class=&quot;node&quot;&gt;
&lt;title&gt;sing&lt;/title&gt;
&lt;g id=&quot;a_node12&quot;&gt;&lt;a xlink:href=&quot;#singleton&quot; xlink:title=&quot;The Singleton&quot;&gt;
&lt;polygon fill=&quot;#8ec5ff&quot; stroke=&quot;black&quot; points=&quot;333.88,-124.5 241.38,-124.5 241.38,-88.5 333.88,-88.5 333.88,-124.5&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;287.62&quot; y=&quot;-101.45&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;The Singleton&lt;/text&gt;
&lt;/a&gt;
&lt;/g&gt;
&lt;/g&gt;
&lt;!-- whatkind&amp;#45;&amp;gt;sing --&gt;
&lt;g id=&quot;edge11&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;whatkind&amp;#45;&amp;gt;sing&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M279.6,-176.91C280.95,-165.26 282.76,-149.55 284.33,-136.02&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;287.77,-136.69 285.45,-126.35 280.82,-135.89 287.77,-136.69&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;298.9&quot; y=&quot;-145.7&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Value&lt;/text&gt;
&lt;/g&gt;
&lt;!-- doom --&gt;
&lt;g id=&quot;node10&quot; class=&quot;node&quot;&gt;
&lt;title&gt;doom&lt;/title&gt;
&lt;g id=&quot;a_node10&quot;&gt;&lt;a xlink:href=&quot;#classic-doom&quot; xlink:title=&quot;Classic Doom&quot;&gt;
&lt;polygon fill=&quot;#8ec5ff&quot; stroke=&quot;black&quot; points=&quot;455.25,-124.5 362,-124.5 362,-88.5 455.25,-88.5 455.25,-124.5&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;408.62&quot; y=&quot;-101.45&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Classic Doom&lt;/text&gt;
&lt;/a&gt;
&lt;/g&gt;
&lt;/g&gt;
&lt;!-- noalign&amp;#45;&amp;gt;doom --&gt;
&lt;g id=&quot;edge8&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;noalign&amp;#45;&amp;gt;doom&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M423.87,-176.91C421.31,-165.26 417.86,-149.55 414.89,-136.02&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;418.32,-135.34 412.76,-126.33 411.49,-136.85 418.32,-135.34&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;433.33&quot; y=&quot;-145.7&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Fatal&lt;/text&gt;
&lt;/g&gt;
&lt;!-- orion --&gt;
&lt;g id=&quot;node11&quot; class=&quot;node&quot;&gt;
&lt;title&gt;orion&lt;/title&gt;
&lt;g id=&quot;a_node11&quot;&gt;&lt;a xlink:href=&quot;#orions-arm&quot; xlink:title=&quot;Orion&amp;#39;s Arm&quot;&gt;
&lt;polygon fill=&quot;#8ec5ff&quot; stroke=&quot;black&quot; points=&quot;556.38,-124.5 472.88,-124.5 472.88,-88.5 556.38,-88.5 556.38,-124.5&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;514.62&quot; y=&quot;-101.45&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Orion&amp;#39;s Arm&lt;/text&gt;
&lt;/a&gt;
&lt;/g&gt;
&lt;/g&gt;
&lt;!-- noalign&amp;#45;&amp;gt;orion --&gt;
&lt;g id=&quot;edge9&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;noalign&amp;#45;&amp;gt;orion&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M444.81,-176.91C457.48,-164.32 474.92,-146.98 489.19,-132.79&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;491.48,-135.45 496.1,-125.92 486.54,-130.49 491.48,-135.45&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;508.17&quot; y=&quot;-145.7&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Survivable&lt;/text&gt;
&lt;/g&gt;
&lt;!-- world --&gt;
&lt;g id=&quot;node13&quot; class=&quot;node&quot;&gt;
&lt;title&gt;world&lt;/title&gt;
&lt;g id=&quot;a_node13&quot;&gt;&lt;a xlink:href=&quot;#world&quot; xlink:title=&quot;The World State&quot;&gt;
&lt;polygon fill=&quot;#8ec5ff&quot; stroke=&quot;black&quot; points=&quot;105.25,-36 0,-36 0,0 105.25,0 105.25,-36&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;52.62&quot; y=&quot;-12.95&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;The World State&lt;/text&gt;
&lt;/a&gt;
&lt;/g&gt;
&lt;/g&gt;
&lt;!-- howmany&amp;#45;&amp;gt;world --&gt;
&lt;g id=&quot;edge14&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;howmany&amp;#45;&amp;gt;world&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M143.83,-88.2C126.82,-75.3 103.36,-57.5 84.5,-43.18&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;86.67,-40.44 76.59,-37.18 82.44,-46.01 86.67,-40.44&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;131.09&quot; y=&quot;-57.2&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;One&lt;/text&gt;
&lt;/g&gt;
&lt;!-- mad --&gt;
&lt;g id=&quot;node15&quot; class=&quot;node&quot;&gt;
&lt;title&gt;mad&lt;/title&gt;
&lt;g id=&quot;a_node15&quot;&gt;&lt;a xlink:href=&quot;#mad&quot; xlink:title=&quot;MAD World&quot;&gt;
&lt;polygon fill=&quot;#8ec5ff&quot; stroke=&quot;black&quot; points=&quot;209.5,-36 123.75,-36 123.75,0 209.5,0 209.5,-36&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;166.62&quot; y=&quot;-12.95&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;MAD World&lt;/text&gt;
&lt;/a&gt;
&lt;/g&gt;
&lt;/g&gt;
&lt;!-- howmany&amp;#45;&amp;gt;mad --&gt;
&lt;g id=&quot;edge12&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;howmany&amp;#45;&amp;gt;mad&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M166.62,-88.41C166.62,-76.76 166.62,-61.05 166.62,-47.52&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;170.13,-47.86 166.63,-37.86 163.13,-47.86 170.13,-47.86&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;182.75&quot; y=&quot;-57.2&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Many&lt;/text&gt;
&lt;/g&gt;
&lt;!-- power --&gt;
&lt;g id=&quot;node16&quot; class=&quot;node&quot;&gt;
&lt;title&gt;power&lt;/title&gt;
&lt;g id=&quot;a_node16&quot;&gt;&lt;a xlink:href=&quot;#power&quot; xlink:title=&quot;Power Concentration&quot;&gt;
&lt;polygon fill=&quot;#8ec5ff&quot; stroke=&quot;black&quot; points=&quot;358,-36 227.25,-36 227.25,0 358,0 358,-36&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;292.62&quot; y=&quot;-12.95&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Power Concentration&lt;/text&gt;
&lt;/a&gt;
&lt;/g&gt;
&lt;/g&gt;
&lt;!-- howmany&amp;#45;&amp;gt;power --&gt;
&lt;g id=&quot;edge13&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;howmany&amp;#45;&amp;gt;power&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M191.82,-88.2C210.88,-75.12 237.26,-57.01 258.25,-42.6&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;259.96,-45.67 266.23,-37.12 256,-39.9 259.96,-45.67&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;252.54&quot; y=&quot;-57.2&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Few&lt;/text&gt;
&lt;/g&gt;
&lt;!-- ec --&gt;
&lt;g id=&quot;node8&quot; class=&quot;node&quot;&gt;
&lt;title&gt;ec&lt;/title&gt;
&lt;g id=&quot;a_node8&quot;&gt;&lt;a xlink:href=&quot;#enduring-complementarity&quot; xlink:title=&quot;Enduring Complementarity&quot;&gt;
&lt;polygon fill=&quot;#8ec5ff&quot; stroke=&quot;black&quot; points=&quot;667.62,-213 501.62,-213 501.62,-177 667.62,-177 667.62,-213&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;584.62&quot; y=&quot;-189.95&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Enduring Complementarity&lt;/text&gt;
&lt;/a&gt;
&lt;/g&gt;
&lt;/g&gt;
&lt;!-- auto&amp;#45;&amp;gt;ec --&gt;
&lt;g id=&quot;edge15&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;auto&amp;#45;&amp;gt;ec&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M584.62,-265.41C584.62,-253.76 584.62,-238.05 584.62,-224.52&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;588.13,-224.86 584.63,-214.86 581.13,-224.86 588.13,-224.86&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;592.88&quot; y=&quot;-234.2&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;No&lt;/text&gt;
&lt;/g&gt;
&lt;!-- underclass --&gt;
&lt;g id=&quot;node17&quot; class=&quot;node&quot;&gt;
&lt;title&gt;underclass&lt;/title&gt;
&lt;g id=&quot;a_node17&quot;&gt;&lt;a xlink:href=&quot;#underclass&quot; xlink:title=&quot;Permanent Underclass&quot;&gt;
&lt;polygon fill=&quot;#8ec5ff&quot; stroke=&quot;black&quot; points=&quot;823.75,-213 685.5,-213 685.5,-177 823.75,-177 823.75,-213&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;754.62&quot; y=&quot;-189.95&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Permanent Underclass&lt;/text&gt;
&lt;/a&gt;
&lt;/g&gt;
&lt;/g&gt;
&lt;!-- auto&amp;#45;&amp;gt;underclass --&gt;
&lt;g id=&quot;edge16&quot; class=&quot;edge&quot;&gt;
&lt;title&gt;auto&amp;#45;&amp;gt;underclass&lt;/title&gt;
&lt;path fill=&quot;none&quot; stroke=&quot;black&quot; d=&quot;M618.62,-265.2C644.94,-251.81 681.61,-233.15 710.25,-218.58&quot; /&gt;
&lt;polygon fill=&quot;black&quot; stroke=&quot;black&quot; points=&quot;711.73,-221.76 719.05,-214.1 708.55,-215.52 711.73,-221.76&quot; /&gt;
&lt;text xml:space=&quot;preserve&quot; text-anchor=&quot;middle&quot; x=&quot;694.98&quot; y=&quot;-234.2&quot; font-family=&quot;Times,serif&quot; font-size=&quot;14.00&quot;&gt;Yes&lt;/text&gt;
&lt;/g&gt;
&lt;/g&gt;
&lt;/svg&gt;

&lt;style&gt;
svg {
width: 100%
}
&lt;/style&gt;

&lt;h1 id=&quot;questions&quot;&gt;Questions&lt;/h1&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Do we reach ASI?&lt;/strong&gt; Do we reach the point where AI is massively better than
humans at almost all cognitive tasks, or does the intelligence explosion
fizzle out, and produce minds that have very jagged capabilities, and are
worse than humans at some things?&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Do we solve alignment?&lt;/strong&gt; Can we instill our own values into the AI, or does
it have its own evolved, ahuman value system?&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;What kind of alignment?&lt;/strong&gt; Intent alignment means an AI that reliably follows
the intent of its operator (that is: only takes actions the operator would
approve of), value alignment means aligning to an impersonal system of values.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Takeoff severity?&lt;/strong&gt; If the AI is unaligned, how fatal is that for humans? Do
we get an out-of-control paperclip maximizer, or a more benign AI that is
unaligned but which cares enough to stay out of our way?&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;How many ASIs?&lt;/strong&gt; A &lt;a href=&quot;https://en.wikipedia.org/wiki/Singleton_(global_governance)&quot;&gt;singleton&lt;/a&gt;, a pantheon, or an ecology?&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Full automation?&lt;/strong&gt; In the no-ASI case, do we still automate almost all jobs
with non-superintelligent AI?&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;misuse-doom&quot;&gt;Misuse Doom&lt;/h1&gt;

&lt;p&gt;In this world, an AI ends the world by following the orders of its operator. The
AI might be strongly intent-aligned, so that it does whatever the operator asks;
or it might be an open-source model with refusals ablated away; or it might be a
commercial AI where safeguards are circumvented by clever task decomposition.&lt;/p&gt;

&lt;p&gt;This could probably happen today: a pandemic could cause human extinction, and
AI is capable enough to assist in biology research. The obvious example of
biorisk is a terrorist group using AI to make a bioweapon. But you could get to
this outcome without anyone even trying to make a bioweapon. A
government-funded, completely official and legitimate laboratory doing
gain-of-function research on some pathogen could accidentally start a
pandemic. In this scenario, the AI’s role in the causal chain would be:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Compressing timelines by speeding up literature review, hypothesis selection,
experiment design, data analysis, equipment scheduling.&lt;/li&gt;
  &lt;li&gt;Increasing the doom surface area by doing many experiments in parallel.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So it’s not the classic Terminator scenario, but rather a case of AI making it
easier to find “&lt;a href=&quot;https://michaelnotebook.com/xrisk/&quot;&gt;recipes for ruin&lt;/a&gt;”.&lt;/p&gt;

&lt;h1 id=&quot;butler&quot;&gt;Butlerian Jihad&lt;/h1&gt;

&lt;blockquote&gt;
  &lt;p&gt;JIHAD, BUTLERIAN: (&lt;em&gt;see also&lt;/em&gt; Great Revolt)—the crusade against computers,
thinking machines, and conscious robots begun in 201 B.G. and concluded in 108
B.G. Its chief commandment remains in the O.C. Bible as “Thou shalt not make a
machine in the likeness of a human mind.”&lt;/p&gt;

  &lt;p&gt;— Frank Herbert, &lt;a href=&quot;https://en.wikipedia.org/wiki/Dune_(novel)&quot;&gt;&lt;em&gt;Dune&lt;/em&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;At the extreme end this is the actual Butlerian Jihad from &lt;em&gt;Dune&lt;/em&gt; (named,
tangentially, after &lt;a href=&quot;https://en.wikipedia.org/wiki/Samuel_Butler_(novelist)&quot;&gt;Samuel Butler&lt;/a&gt;, who in &lt;em&gt;1863&lt;/em&gt; wrote &lt;a href=&quot;https://web.archive.org/web/20060524131242/http://www.nzetc.org/tm/scholarly/tei-ButFir-t1-g1-t1-g1-t4-body.html&quot;&gt;the first AI
doomer article&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;Otherwise, this is something like an international agreement to stop the
development of superintelligent AI (&lt;a href=&quot;https://controlai.org/&quot;&gt;ControlAI&lt;/a&gt; is advocating for this and
you should support them). Maybe this fails, and we go back to the start, or it
succeeds and we can get on &lt;a href=&quot;/article/the-other-road&quot;&gt;the other road&lt;/a&gt;.&lt;/p&gt;

&lt;h1 id=&quot;enduring-complementarity&quot;&gt;Enduring Complementarity&lt;/h1&gt;

&lt;p&gt;This is the mundane future. It turns out that the “G” in AGI was never
real. Every mind is its own &lt;em&gt;sui generis&lt;/em&gt; thing. Jaggedness remains. AIs remain
a complement to, rather than a substitute for, human labour. There is job
displacement but no mass unemployment, people move up to more
managerial/relational roles. There are fewer ICs and everyone is some kind of
manager. Labour becomes a smaller factor of production and consequently loses
some bargaining power against capital, but most people are fine.&lt;/p&gt;

&lt;h1 id=&quot;underclass&quot;&gt;Permanent Underclass&lt;/h1&gt;

&lt;p&gt;In this future, we don’t reach ASI, but non-superintelligent AI is enough to
automate almost all jobs. The labour term in the production function goes to
zero. All income is either capital ownership or welfare.&lt;/p&gt;

&lt;p&gt;Economically: there may be material abundance, but absent some kind of &lt;a href=&quot;https://en.wikipedia.org/wiki/Georgism&quot;&gt;Georgist
economic reform&lt;/a&gt;, most of the gains are captured by
rent-seeking. Socially: there is no more social mobility, your lot in life is
determined by how much capital you had when the music stopped. Growing your
human capital is no longer incentivized, since it doesn’t change your life
outcomes.&lt;/p&gt;

&lt;p&gt;See also:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;https://jyn.dev/rising-tide-raises-rents/&quot;&gt;&lt;em&gt;A rising tide raises all rents&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://jyn.dev/&quot;&gt;jyn&lt;/a&gt;.&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;/article/no-one-escapes-the-permanent-underclass&quot;&gt;&lt;em&gt;No-One Escapes the Permanent Underclass&lt;/em&gt;&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;singleton&quot;&gt;The Singleton&lt;/h1&gt;

&lt;blockquote&gt;
  &lt;p&gt;I am the Eschaton. I am not your god.&lt;br /&gt;
I am descended from you, and I exist in your future.&lt;br /&gt;
Thou shalt not violate causality within my historic light cone. Or else.&lt;/p&gt;

  &lt;p&gt;— Charles Stross, &lt;a href=&quot;https://en.wikipedia.org/wiki/Singularity_Sky&quot;&gt;&lt;em&gt;Singularity Sky&lt;/em&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In this world, we build ASI, and it’s successfully aligned to some impersonal
system of values. First order of business is to prevent someone else from
building an ASI that might challenge the first one, so the AI performs a
“&lt;a href=&quot;https://www.lesswrong.com/w/pivotal-act&quot;&gt;pivotal act&lt;/a&gt;”, which is a LessWrong euphemism for “put everyone under
constant surveillance, destroy all GPUs”. The result is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Singleton_(global_governance)&quot;&gt;singleton&lt;/a&gt;: a
single mind that is individually sovereign over everyone else.&lt;/p&gt;

&lt;p&gt;What happens then? It depends on the content of the AI’s value system, and the
stability of that value system over time. The singleton might be a “&lt;a href=&quot;https://www.forethought.org/research/nightwatchmen&quot;&gt;night
watchman&lt;/a&gt;”, whose only goal is to prevent the creation of paperclip
maximizers (i.e. prevent &lt;a href=&quot;#doom&quot;&gt;Classic Doom&lt;/a&gt;), or it might be deeply
interventionist. The singleton’s values might be incompatible with human life:
maybe its creators made it utilitarian and species-egalitarian, and it notices
that an 80kg human generates less utility than 80kg of &lt;a href=&quot;https://en.wikipedia.org/wiki/Cochineal&quot;&gt;&lt;em&gt;Dactylopius
coccus&lt;/em&gt;&lt;/a&gt;, so we get pulped and the universe becomes a terrarium for insects
instantiating the &lt;a href=&quot;https://benthams.substack.com/p/insects-matter-more-than-people&quot;&gt;rebugnant conclusion&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The future is a very long time: the stelliferous era will last at least 100
trillion years. Maybe the singleton is aligned at first, but over millions or
billions of years of growth and self-modification, the alignment drifts (the
predictable LessWrong objection here would be: “&lt;em&gt;obviously&lt;/em&gt; an ASI would be
smart enough to prevent this”, because ASI is magic).&lt;/p&gt;

&lt;h1 id=&quot;classic-doom&quot;&gt;Classic Doom&lt;/h1&gt;

&lt;p&gt;You know, the paperclip maximizer. “&lt;a href=&quot;https://intelligence.org/files/AIPosNegFactor.pdf&quot;&gt;The AI does not hate you, nor does it love
you, but you are made out of atoms which it can use for something
else&lt;/a&gt;”. This has been done to death so I won’t bore you.&lt;/p&gt;

&lt;h1 id=&quot;orions-arm&quot;&gt;Orion’s Arm&lt;/h1&gt;

&lt;blockquote&gt;
  &lt;p&gt;Another of the sayings of Chrysippus is: We should infer in the case of a
beautiful dwelling-place that it was built for its owners and not for mice; we
ought, therefore, in the same way to regard the universe as the dwelling-place
of the gods.&lt;/p&gt;

  &lt;p&gt;— Cicero, &lt;a href=&quot;https://en.wikipedia.org/wiki/De_Natura_Deorum&quot;&gt;&lt;em&gt;De Natura Deorum&lt;/em&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;We build ASI, and it has its own goals and values. Luckily, it’s neither
actively hostile to humans, nor so indifferent that it destroys us without even
noticing. It mostly leaves us alone, and goes off to another star to do its own
thing. The result might be something like the galactic ecology in &lt;a href=&quot;https://orionsarm.com/&quot;&gt;Orion’s
Arm&lt;/a&gt;, where humans and superintelligent AIs and everything in between have a
kind of stable co-existence by virtue of occupying different volumes of space.&lt;/p&gt;

&lt;h1 id=&quot;world&quot;&gt;The World State&lt;/h1&gt;

&lt;blockquote&gt;
  &lt;p&gt;… [The] human race is best ordered when in all its movements and motors it
is controlled by one Prince as by one mover, by one law as by one motion.&lt;/p&gt;

  &lt;p&gt;— Dante, &lt;a href=&quot;https://en.wikipedia.org/wiki/Monarchia&quot;&gt;&lt;em&gt;De Monarchia&lt;/em&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In this world, a reliably intent-aligned ASI takes over the world. What happens
then depends on the character of its operator, who now wields the &lt;a href=&quot;https://en.wikipedia.org/wiki/Ring_of_Gyges&quot;&gt;ring of
Gyges&lt;/a&gt;. The difference between this and &lt;a href=&quot;#singleton&quot;&gt;The Singleton&lt;/a&gt; is
that in this future, the singleton is the human operator mediated by an ASI,
rather than an ASI deriving actions from an impersonal system of values.&lt;/p&gt;

&lt;h1 id=&quot;mad&quot;&gt;MAD World&lt;/h1&gt;

&lt;blockquote&gt;
  &lt;p&gt;As Nietzsche said, “God is dead”. The gods die but the Titans gain
power. Technology is just the clothes, the armour, of the Titans.&lt;/p&gt;

  &lt;p&gt;— Ernst Jünger&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In this world, the &lt;a href=&quot;https://ai-alignment.com/clarifying-ai-alignment-cec47cd69dd6&quot;&gt;intent alignment&lt;/a&gt; problem is solved, and we can build an
ASI that reliably obeys the intent of its operator. Furthermore,
superintelligence is widely distributed rather than centralized: everyone has
their own personal intent-aligned ASI. In this world, everyone is 10,000x more
capable and maybe you can build WMDs with a prompt. What keeps the peace
is mutually assured destruction. It’s not clear how survivable this is.&lt;/p&gt;

&lt;p&gt;The obvious failure mode is that one bad actor causes human extinction. Maybe
their ASI can’t overpower all the other ASIs put together, &lt;em&gt;but&lt;/em&gt; it can do
enough damage to destroy human life before the other ASIs can act (a general
observation: the technology to make the Earth uninhabitable to humans is much
lower on the tech tree than the technology to build a distributed,
extinction-proof interstellar civilization).&lt;/p&gt;

&lt;p&gt;The generalized ($n&amp;gt;1$ rather than $n=1$) version of this is that all conflicts
are scaled up: two humans having a feud is the equivalent of nuclear war, two
countries going to war is something much greater.&lt;/p&gt;

&lt;p&gt;On the more positive note: if there’s no way to stop AI, then “everyone should
have it” is obviously preferable to “only the Elect should have it”, which is
intolerable. If the goal is to keep the peace, the threat of mutually-assured
destruction is better than totalitarianism, in the sense that there’s more
dignity in it.&lt;/p&gt;

&lt;h1 id=&quot;power&quot;&gt;Power Concentration&lt;/h1&gt;

&lt;p&gt;A smaller version of &lt;a href=&quot;#mad&quot;&gt;MAD World&lt;/a&gt;, where intent-aligned ASI is less
distributed. A small number of actors have ASI, and everyone else just watches
what happens. Maybe the AI companies will divide the light cone among themselves,
like in the &lt;a href=&quot;https://en.wikipedia.org/wiki/Treaty_of_Tordesillas&quot;&gt;Treaty of Tordesillas&lt;/a&gt;: every star and galaxy between 0h and
12h of &lt;a href=&quot;https://en.wikipedia.org/wiki/Right_ascension&quot;&gt;right ascension&lt;/a&gt; goes to Anthropic, analogously for OpenAI.&lt;/p&gt;

</content:encoded>
      </item>
    
      <item>
        <title>We Won’t “Just Hang Out”</title>
        <description>On human connection after the Singularity.</description>
        <pubDate>Wed, 28 Oct 2026 00:00:00 +1100</pubDate>
        <link>https://borretti.me/article/we-wont-just-hang-out</link>
        <guid isPermaLink="true">
          https://borretti.me/article/we-wont-just-hang-out
        </guid>
        
        <content:encoded>&lt;p&gt;What will we do after the Singularity? The standard answer is: “we’ll just hang
out”. The infinite vacation, as in the &lt;a href=&quot;https://en.wikipedia.org/wiki/Culture_series&quot;&gt;&lt;em&gt;Culture&lt;/em&gt;&lt;/a&gt; novels. I think this
won’t work. “Hanging out” implies:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Other people exist.&lt;/li&gt;
  &lt;li&gt;They want to hang out with you.&lt;/li&gt;
  &lt;li&gt;You want to hang out with them.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;And the post-scarcity, infinite vacation AI utopia undermines all these
assumptions.&lt;/p&gt;

&lt;h1 id=&quot;other-people-exist&quot;&gt;Other people exist&lt;/h1&gt;

&lt;p&gt;The social processes by which people come together and build relationships will
disappear. Work, obviously, is pointless in a world of cheap superhuman
intelligence. University (really all education beyond teaching people to speak)
will be pointless because there’s no work to do and because growing your human
capital doesn’t change your life outcomes: you get the same UBI whether you’re
illiterate or a PhD.&lt;/p&gt;

&lt;p&gt;I met a lot of my friends by writing open-source code, and participating in the
community of software engineers. AI is dissolving that community &lt;a href=&quot;/article/no-man-is-an-island&quot;&gt;by making
human contributions superfluous&lt;/a&gt;. If Claude Code has existed when I was
starting out, nobody would have cared about &lt;a href=&quot;https://austral-lang.org/&quot;&gt;Austral&lt;/a&gt; or my &lt;a href=&quot;https://github.com/eudoxia0&quot;&gt;other
open-source projects&lt;/a&gt; or my technical blog posts.&lt;/p&gt;

&lt;p&gt;As a teenager I tried to install Linux, ran into all kinds of problems, and went
on the IRC chatrooms to get help. And in the process I made friends. Today, I
would just ask Claude. AI disintermediates everyone, so that we no longer need
each other. I still have friends, but I’m running off this pre-AI supply of
friends I met through communities of practice. Young people don’t have that
privilege.&lt;/p&gt;

&lt;p&gt;What even is the point of cities in a world without work? If there’s nothing to
do, there is no need to bring people together geographically.&lt;/p&gt;

&lt;h1 id=&quot;they-want-to-hang-out-with-you&quot;&gt;They want to hang out with you&lt;/h1&gt;

&lt;p&gt;The BATNA to hanging out keeps getting better, cheaper, and more
convenient. When people’s sensorium becomes an infinite firehose of AI-generated
superstimulus, why would they hang out with anyone?&lt;/p&gt;

&lt;p&gt;This is less about AI and more about the trajectory of consumer technology,
since many people already live like this: they go to work and watch
TikTok. After work, they will just watch TikTok.&lt;/p&gt;

&lt;h1 id=&quot;you-want-to-hang-out-with-them&quot;&gt;You want to hang out with them&lt;/h1&gt;

&lt;p&gt;As &lt;a href=&quot;https://x.com/GregoryConti19&quot;&gt;Gregory Conti&lt;/a&gt; &lt;a href=&quot;https://x.com/GregoryConti19/status/2102139056310321185&quot;&gt;explains&lt;/a&gt;:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;[Under] ASI this endless leisure will be passed in diminished, tiresome
company, since we will lack the experiences of struggle and striving that
generate interesting personalities. Our minds will have atrophied. We will
have little of note to say to one another.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The people in the AI utopia will have no skills, no education, no
accomplishments. We’ll be lucky if they are even literate. They will not be
worth talking to, and you won’t want to hang out with them.&lt;/p&gt;

&lt;p&gt;Maybe you and I can hang out with our existing friends. But just iterate this a
bit. What about the next generation? How are they going to build meaningful
relationships where the social processes that built those relationships are all
in the past?&lt;/p&gt;

&lt;p&gt;If we get radical life extension, there’s the problem of attrition over the long
term: people will drift apart, drop out of society to enter the experience
machine, die in accidents, or end their lives out of ennui. So your social
network might be a non-renewable resource that gets exhausted over time as your
pre-Singularity relationships attrite and can’t be replaced for the reasons
above.&lt;/p&gt;

</content:encoded>
      </item>
    
      <item>
        <title>No Man Is an Island</title>
        <description>AI makes intellectual contribution superfluous.</description>
        <pubDate>Fri, 09 Oct 2026 00:00:00 +1100</pubDate>
        <link>https://borretti.me/article/no-man-is-an-island</link>
        <guid isPermaLink="true">
          https://borretti.me/article/no-man-is-an-island
        </guid>
        
        <content:encoded>&lt;p&gt;In this post, I argue that individual intellectual activity can only be
sustained in an intellectual community of other humans. AI dissolves these
communities, which in turn makes private intellectual activity rarer.&lt;/p&gt;

&lt;h1 id=&quot;the-case-of-software&quot;&gt;The Case of Software&lt;/h1&gt;

&lt;p&gt;Some years ago, when it became clear that AI would solve software engineering,
my thinking was:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;In my professional life, I’m happy to move one level up to become a manager
of AI agents. I’m literate, I’m a good technical writer, I can describe what
I want and let AI agents write the code.&lt;/li&gt;
  &lt;li&gt;In my own time, I can keep doing the things I care about because I enjoy them
intrinsically. This is the “intellectual” side of software engineering:
reading technical blog posts and papers, learning new programming languages,
designing new programming languages, writing technical essays, writing code
for my tiny open source projects.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The second point has not quite worked out. What actually happened? First, the
discourse of software engineering became worse. As I &lt;a href=&quot;/article/the-education-of-a-doomer&quot;&gt;wrote earlier&lt;/a&gt;:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Claude Code was released a little over a year ago. In that short time,
software engineering has been completely transformed. Materially, it might be
positive: higher productivity, though at the cost of a messier
codebase. Socially, it has been a disaster.&lt;/p&gt;

  &lt;p&gt;The discourse around software engineering has gotten dumber. It’s like
everyone in the industry lost 30 IQ points. People used to talk about
compilers, type systems, logic. Now they talk about “prompts”, “harnesses”,
“loops”. The discourse is narrower, shallower, and more repetitive. There’s
only so many times I can hear about “agentic harnesses” before I lose my mind.&lt;/p&gt;

  &lt;p&gt;Then there’s the loss of human capital formation: there is nothing to
learn. Prompting is not a skill, at least, it’s a much shallower skill than
software engineering. The instrumental dimension of the work has improved in
that people can get more output per unit of effort, but the dimension of work
that’s about building up human capital has collapsed. And maybe this is
rational. Why learn to code at all? The computers can do that for us. And so
the rigorous, systematic thinking you need to practice in order to be a good
programmer: all gone. The machines can be rational for us. We can just vibe.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Second, contributing to the commons of software engineering is increasingly
pointless. Before AI, you could publish open-source code, write blog posts to
share ideas or inspire other people, write expository texts like tutorials,
forum posts, textbooks etc. to teach people. After AI, what’s the point?&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;You write a blog post: who’s going to read it? The next training run will
ingest it, marginally improving AI capabilities. Maybe the post was a
workaround to some obscure technical problem, so the next time someone
encounters that problem, they will ask Claude, who will solve it without
crediting you. Maybe you had some insight about how to structure large
codebases: who cares? The humans aren’t making those decisions anymore.&lt;/li&gt;
  &lt;li&gt;You design a revolutionary new programming language: who cares? Maybe Claude
cares, for what that’s worth. But humans don’t write or even read the code
anymore. The programming language is an implementation detail the humans no
longer have to care about.&lt;/li&gt;
  &lt;li&gt;You write a library, and publish it on GitHub: who cares? The AIs might
discover it, and use it, but the operator won’t know you exist or did
anything.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It’s not just “you can’t get GitHub stars or traffic to your blog”; rather,
there is no sense of a common human project you can contribute to. There’s your
own private garden of code, which you can grow infinitely in all directions with
the help of AI, but you never have to leave the garden and go to the bazaar to
trade with people. Under these conditions, it’s hard to care or do anything.&lt;/p&gt;

&lt;p&gt;But does it actually matter? Does it matter if we stop writing blog posts about
obscure JavaScript features, and designing new programming languages? Maybe
writing code was always drudgery, and now we can move on to higher things, like
math—&lt;a href=&quot;https://proofsandprompts.com/2026/10/08/100-reactions-to-100-solutions/&quot;&gt;oh, wait&lt;/a&gt;.&lt;/p&gt;

&lt;h1 id=&quot;the-intellectual-life&quot;&gt;The Intellectual Life&lt;/h1&gt;

&lt;p&gt;From observing what happened to software engineering, and what’s currently
happening to mathematics, I think we can derive some general insights about
intellectual practices in general.&lt;/p&gt;

&lt;p&gt;We tend to think of intellectual activity as private and solitary: the
philosopher sitting in his armchair, deriving the world &lt;em&gt;ab initio&lt;/em&gt;. But
intellectual activity has two inputs that can’t be acquired in isolation: a
&lt;strong&gt;shared body of work&lt;/strong&gt; to build upon, and &lt;strong&gt;motivation&lt;/strong&gt;. The shared body of
work is communal, unless you want to recapitulate the entire tech
tree. Motivation we can break down into two components:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Intrinsic motivation:&lt;/strong&gt; we learn for the sake of learning, we create art
from a compulsion we can’t understand, etc.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Extrinsic motivation:&lt;/strong&gt; David Chapman &lt;a href=&quot;https://meaningness.substack.com/p/nobility-table-of-contents&quot;&gt;defines&lt;/a&gt; “nobility” as
manifesting glory for the service of others, and using our abilities in
service of others. We want our work to be useful to others, we want others to
benefit from our work, we want to contribute to a shared human project.  Fame
and the esteem and good will of your peers are the proxies by which we measure
our contribution.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We tend to think of intrinsic motivation as the purest kind: endogenous,
self-created, unmotivated by material or social gain. But it’s an emotion, and,
like all emotions, it’s transient and short-lived. And this is rational:
otherwise, we’d all be stuck in life-long unproductive obsessions. So, we need
something to fill the gaps between moments of divine inspiration. Extrinsic
motivation serves this function.&lt;/p&gt;

&lt;p&gt;Private intellectual activity that is sustained, complex, and long-term requires
an external intellectual community to provide material and motivation, like fuel
and oxidizer. That private activity, in turn, sustains the community: by
publishing papers, textbooks, code, etc., you add to the shared body of work for
others to build on top of; by citing someone’s paper or contributing to their
repository, you give them the recognition and honor that confirms they are doing
useful work, which in turn motivates them to keep contributing.&lt;/p&gt;

&lt;p&gt;Without community, you don’t get isolated individuals each working on their own
things: you get nothing. The inputs to intellectual activity dry up: no one is
adding to the shared body of work, and there are no peers to benefit from your
own intellectual activity. Without this extrinsic motivation, you get less
intellectual activity because, again, intrinsic motivation is fleeting.&lt;/p&gt;

&lt;h1 id=&quot;after-ai&quot;&gt;After AI&lt;/h1&gt;

&lt;p&gt;After AI, intellectual contributions become unnecessary or redundant. In the
case of software: the AIs write all the code, so what’s the point of writing
either code or prose? The audience for those things is now severely
diminished. Humans don’t write code anymore, so they won’t read blog posts about
how to write code, or tutorials, or try new libraries or programming
languages. In the case of mathematics: the AIs can prove theorems, write papers,
explain papers, tutor students, and in the near future, they might write entire
textbooks better than humans. So what’s the point of writing a paper, or a
textbook? It’s superfluous.&lt;/p&gt;

&lt;p&gt;If intellectual activity is unnecessary—if there’s no &lt;em&gt;consequence&lt;/em&gt; to
designing a new programming language or publishing a paper, or if there’s simply
no community to contribute to—then it won’t happen. There’s no point.&lt;/p&gt;

&lt;p&gt;Now apply this to every other domain of intellectual activity, and you see what
the future looks like. There may be individuals building new libraries and
programming languages, but no shared culture of software engineering; there may
be individual students and practitioners of mathematics, but no living community
of mathematicians.&lt;/p&gt;

&lt;h1 id=&quot;why-we-underrate-intellectual-community&quot;&gt;Why We Underrate Intellectual Community&lt;/h1&gt;

&lt;p&gt;I’ve spoken to people who think AI will have a positive effect on the life of
the mind, and their thinking is that right now too many people are doing
intellectual activity for instrumental reasons: citations, clout, etc. In this
view, the collapse of intellectual communities is &lt;em&gt;good&lt;/em&gt;, because it sifts the
intrinsically-motivated übermenschen from the clout-chasing masses.&lt;/p&gt;

&lt;p&gt;I think this view fits with contemporary society: we view intrinsic and
extrinsic motivation as high and low status, respectively.  A “developed” person
is supposed to have a private, inexhaustible reserve of motivation which is
causally disconnected from external reward.&lt;/p&gt;

&lt;p&gt;But this is not a realistic view of human beings. Humans are social animals who
can only flourish in the society of other humans. We care, and we should care,
about contributing to the world. And if technology makes our contributions
superfluous, then what is left?&lt;/p&gt;

&lt;h1 id=&quot;acknowledgements&quot;&gt;Acknowledgements&lt;/h1&gt;

&lt;p&gt;Thanks to &lt;a href=&quot;https://lukedrago.com/&quot;&gt;Luke Drago&lt;/a&gt; and &lt;a href=&quot;https://andymatuschak.org/&quot;&gt;Andy Matuschak&lt;/a&gt; for feedback and conversations.&lt;/p&gt;

</content:encoded>
      </item>
    
      <item>
        <title>The Other Road</title>
        <description>The transhuman future vs. the AGI future.</description>
        <pubDate>Thu, 10 Sep 2026 00:00:00 +1000</pubDate>
        <link>https://borretti.me/article/the-other-road</link>
        <guid isPermaLink="true">
          https://borretti.me/article/the-other-road
        </guid>
        
        <content:encoded>&lt;blockquote&gt;
  &lt;p&gt;For us, humanity was a distant goal toward which all men were moving, whose image no one knew, whose laws were nowhere written down.&lt;/p&gt;

  &lt;p class=&quot;cite&quot;&gt; — Hermann Hesse, &lt;em&gt;Demian&lt;/em&gt; &lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Despite having spent most of my life marinating in transhumanist ideas, I never
cared about AI. Because I’m a classical transhumanist: I want to use technology
to change myself, so that &lt;em&gt;I&lt;/em&gt; can do more. Because I, personally, want to learn
and think and act, and not have a machine learn and think and act on my
behalf. And in this worldview, AI is not interesting, except as a stopgap
technology to unlock the transhumanism tech tree faster (but of course,
technology has plans of its own).&lt;/p&gt;

&lt;p&gt;Now, the trajectory of technology has shaken out such that we’re closer to AGI
than to transhumanism. But, realizability aside, I find it surprising and
demoralizing that the “AGI future” seems to be much more popular than the
“transhuman future” (more clearly: transhuman, but human-descended). I thought
“become as the gods” would be more attractive than “build a benevolent god to
rule over us”.&lt;/p&gt;

&lt;p&gt;It has made me more cynical about humanity. Most people don’t want to be more,
do more, know more. They want to watch Netflix and die. They want the world of
&lt;a href=&quot;https://en.wikipedia.org/wiki/Culture_series&quot;&gt;the Culture&lt;/a&gt;: an AI-run infinite vacation where they consume content for
a few centuries, then autoeuthanize from boredom. When you tell them AGI means
disempowerment, the closure of the future, giving up their autonomy, they don’t
care. Choice and autonomy are a burden to them. They want to die in every
dimension, except the sensual. Chimps wearing diapers, cared for by machines.&lt;/p&gt;

&lt;p&gt;It’s pathetic and contemptible: selling humanity’s birthright &lt;a href=&quot;https://en.wikipedia.org/wiki/Mess_of_pottage&quot;&gt;for a mess of
pottage&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The future I dreamt of is one where we have the choice to be more: to see more,
live more, know more, do more. To build marvelous things. A world whose art and
science is to ours as the Old Masters are to the coarse coinage of the dark
ages. Not the obsessive, uncompromising, abstract monoculture of LessWrong
utilitarianism (tiling the universe with synthetic soy-fed shrimp hedonium) but
a plural future, human at every scale, from atoms to galaxies, where baseline
humans can till the soil and swim in oceans under the light of stars veiled with
Matrioshka brains; with thinking glass, ensouled glass. And humanity radiating
into endless forms most beautiful—to storm Heaven, and abolish Death—and a
sphere of living matter, centered on the Earth, expanding through the universe
and turning cratered rock and ice into oceans and gardens, dead matter into
living matter, breathing life into the dead world—to make the whole cosmos
into a garden—“&lt;a href=&quot;https://www.biblegateway.com/passage/?search=Isaiah%2043%3A19&amp;amp;version=KJV&quot;&gt;and rivers in the desert&lt;/a&gt;”. In the &lt;a href=&quot;https://en.wikipedia.org/wiki/Oration_on_the_Dignity_of_Man&quot;&gt;&lt;em&gt;Oration on the
Dignity of Man&lt;/em&gt;&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Giovanni_Pico_della_Mirandola&quot;&gt;Giovanni Pico&lt;/a&gt; wrote: “[God] had already
adorned the supercelestial region with Intelligences, infused the heavenly
globes with the life of immortal souls”. I choose to read it as a prophecy:
“&lt;a href=&quot;https://www.biblegateway.com/passage/?search=Psalm%20102%3A25&amp;amp;version=KJV&quot;&gt;the heavens are the work of thy hands&lt;/a&gt;”.&lt;/p&gt;

&lt;p&gt;But we don’t have to betray ourselves. The other road exists, and it’s ready
when we are. The better future is one where we stop building ASI and accelerate
transhumanism, so that we, ourselves, can inhabit the future, and not just our
creations.&lt;/p&gt;

</content:encoded>
      </item>
    
      <item>
        <title>The Education of a Doomer</title>
        <description>How and why I went from AI optimist to AI doomer.</description>
        <pubDate>Tue, 08 Sep 2026 00:00:00 +1000</pubDate>
        <link>https://borretti.me/article/the-education-of-a-doomer</link>
        <guid isPermaLink="true">
          https://borretti.me/article/the-education-of-a-doomer
        </guid>
        
        <content:encoded>&lt;p&gt;If you follow me on Twitter, or read this blog, you have noticed that I went
from being generally optimistic and excited about AI to being extremely
concerned. And I thought, I should explain why I changed my mind. Each section
of this post is about some aspect of AI where my views shifted. I begin each
section by explaining my previous beliefs, and why I held them, and then explain
why those beliefs changed.&lt;/p&gt;

&lt;h1 class=&quot;no_toc&quot; id=&quot;contents&quot;&gt;Contents&lt;/h1&gt;

&lt;ol id=&quot;markdown-toc&quot;&gt;
  &lt;li&gt;&lt;a href=&quot;#economics&quot; id=&quot;markdown-toc-economics&quot;&gt;Economics&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#alignment&quot; id=&quot;markdown-toc-alignment&quot;&gt;Alignment&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#control&quot; id=&quot;markdown-toc-control&quot;&gt;Control&lt;/a&gt;    &lt;ol&gt;
      &lt;li&gt;&lt;a href=&quot;#writing&quot; id=&quot;markdown-toc-writing&quot;&gt;Writing&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#software-engineering&quot; id=&quot;markdown-toc-software-engineering&quot;&gt;Software Engineering&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#deferring-to-ai&quot; id=&quot;markdown-toc-deferring-to-ai&quot;&gt;Deferring to AI&lt;/a&gt;&lt;/li&gt;
    &lt;/ol&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#stagnation&quot; id=&quot;markdown-toc-stagnation&quot;&gt;Stagnation&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h1 id=&quot;economics&quot;&gt;Economics&lt;/h1&gt;

&lt;p&gt;Automation has been good. We’ve automated 99% of the jobs people did in 1790,
and the result is not mass unemployment, rather, we are wealthier, healthier,
more educated, we have more leisure, etc. I had this vague, inductive idea that,
while I can’t predict what jobs will exist after AGI, there will be demand for
me to do something. If nothing else, the much higher economic growth of the
post-AGI world means that the human niche, while small in absolute terms, might
be much larger than today’s economy.&lt;/p&gt;

&lt;p&gt;And this may yet be true. Or it may show a lack of imagination on my part. If AI
develops such that we have enduring complementarity between humans and AIs, then
we might still have jobs in the post-AGI future. But if AI becomes truly
general, the G in AGI, and on top of that it is vastly smarter, faster, and
cheaper than humans, then there might be nothing for us to do, except live off
UBI.&lt;/p&gt;

&lt;p&gt;When people talk about UBI, they typically worry about the problem of meaning in
a world without work. I’ve never had this worry. When I was funemployed last
year, I spent my time reading books and writing code and hanging out with
friends. If the future is an infinite UBI-funded vacation, I know what I’ll
do. “Before the Singularity, read books and throw house parties; after the
singularity, etc.”&lt;/p&gt;

&lt;p&gt;But then I started thinking about the political consequences of AGI, and started
writing about it:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;/article/no-one-escapes-the-permanent-underclass&quot;&gt;&lt;em&gt;No-One Escapes the Permanent Underclass&lt;/em&gt;&lt;/a&gt;: if humans are economically
useless, the state does not need them. Why pay out UBI to people who have
neither economic nor political power?&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;/article/when-the-future-doesnt-need-us&quot;&gt;&lt;em&gt;When The Future Doesn’t Need Us&lt;/em&gt;&lt;/a&gt;: factory workers can sabotage the
machines, truck drivers can shut down logistics. But if humans are
economically useless, there is no way for people to veto the political order
by withdrawing their contribution to it. And if the AIs fight wars, then the
state can be arbitrarily repressive.&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;/article/mathematics-without-mathematicians&quot;&gt;&lt;em&gt;Mathematics Without Mathematicians&lt;/em&gt;&lt;/a&gt;: most arguments about humans
moving “one job up” fail because the AI can do those jobs too.&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;/article/our-servants-will-do-that-for-us&quot;&gt;&lt;em&gt;Our Servants Will Do That For Us&lt;/em&gt;&lt;/a&gt;: even for jobs we think of as
uniquely human, we might prefer to have machines do those jobs too.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;alignment&quot;&gt;Alignment&lt;/h1&gt;

&lt;p&gt;I had hope that alignment would turn out to be a normal engineering problem,
that we solve through empirical experimentation and investigation, like
everything else. It helps that the early LLMs were not incomprehensible alien
minds, like something from a Stanisław Lem novel, but rather immensely
human. It’s hard &lt;em&gt;not&lt;/em&gt; to anthropomorphize them. The huge core of unsupervised
learning in an LLM understands human morality just fine: you can talk to them
about it, they will explain, eloquently, in detail, why something is “good” or
“bad” according to some moral system. And, because capabilities were weaker, the
failures were very small. What’s the worst ChatGPT in 2022 could do?&lt;/p&gt;

&lt;p&gt;Since like 2024, reinforcement learning has been the main technique to push the
frontier forward. And reinforcement learning agents work exactly like Yudkowsky
says. Consequently, capabilities have increased markedly but the models are
harder to understand (literally: their prose is increasingly incomprehensible)
and are increasingly misaligned as RL scrambles their brains in pursuit of
reward. &lt;a href=&quot;https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/&quot;&gt;Incidents&lt;/a&gt; of serious misalignment are more common and more
consequential. It’s clear that AI capabilities are growing far, far faster than
our ability to control or even understand them.&lt;/p&gt;

&lt;h1 id=&quot;control&quot;&gt;Control&lt;/h1&gt;

&lt;p&gt;I thought—or, rather, I implicitly believed—that people would &lt;em&gt;want&lt;/em&gt; to
remain in control of the AIs. And if we want to retain control, and solve
alignment, then we will stay in control. Simple enough. But recently I started
to think: no, we will probably hand over control to the AIs.&lt;/p&gt;

&lt;p&gt;The weak version of the disempowerment thesis is something like the prisoner’s
dilemma: people/companies/polities that hand more power to AI outcompete those
that don’t, so there’s competitive pressure towards disempowerment. This is easy
to believe.&lt;/p&gt;

&lt;p&gt;The strong version of disempowerment is: the AIs will be so smart,
knowledgeable, personable, moral etc. that we will willingly, voluntarily hand
power to them. We’ll think, “they can do a better job than us”, and we’ll be
right. Believing this requires you to be somewhat cynical about humanity’s
desire for autonomy vs. material considerations. But, over the past few months,
I have become more cynical about it.&lt;/p&gt;

&lt;p&gt;This was not a sudden “oh shit” insight and more a slow, gradual accumulation of
tiny little grains of evidence that each point slightly in the direction of
disempowerment.&lt;/p&gt;

&lt;h2 id=&quot;writing&quot;&gt;Writing&lt;/h2&gt;

&lt;p&gt;The proliferation of AI writing, I think, is evidence for disempowerment.&lt;/p&gt;

&lt;p&gt;Early on, I’d start reading a blog post, and it’s AI slop. I’d open a link to a
GitHub project that looks interesting, and the README is slop. I’d get a pull
request on one of my projects, and the code is slop. You read articles on major
newspapers, and they’re written by AI. And that was merely frustrating.&lt;/p&gt;

&lt;p&gt;Then it got worse. I read a paper arguing against the use of LLMs in
mathematics—and the paper was AI slop. I &lt;a href=&quot;/article/review-job-less-utopia&quot;&gt;reviewed an entire book&lt;/a&gt;
about the post-AGI world, and it was AI slop. I read that some conference
started using Pangram to filter out a deluge of AI-written submissions, and I
read a tweet from a college professor—a college professor, writing under their
own name!—defending the use of AI to write papers.&lt;/p&gt;

&lt;p&gt;I don’t think using AI to write is analogous to calculators, or search engines, or
other such things. There are only finitely many things you can automate. And once
you’ve automated writing, there’s no higher-order activity to move to. People
think “the ideas are mine, the writing is the AI”, i.e., they feed the AI a list
of rambling bullet points that the AI massages into a blog post, or a paper, or
whatever. And they think that step is “mere writing”, while their composition of
the bullet points was “thinking”. But they are wrong, because &lt;a href=&quot;/article/human-routers-of-machine-words&quot;&gt;writing is
thinking&lt;/a&gt;. And so, by letting the AI write, you are giving up most of
what makes up thinking.&lt;/p&gt;

&lt;h2 id=&quot;software-engineering&quot;&gt;Software Engineering&lt;/h2&gt;

&lt;p&gt;Claude Code was released a little over a year ago. In that short time, software
engineering has been completely transformed. Materially, it might be positive:
higher productivity, though at the cost of a messier codebase. Socially, it has
been a disaster.&lt;/p&gt;

&lt;p&gt;The discourse around software engineering has gotten dumber. It’s like everyone
in the industry lost 30 IQ points. People used to talk about compilers, type
systems, logic. Now they talk about “prompts”, “harnesses”, “loops”. The
discourse is narrower, shallower, and more repetitive. There’s only so many
times I can hear about “agentic harnesses” before I lose my mind.&lt;/p&gt;

&lt;p&gt;Then there’s the loss of human capital formation: there is nothing to
learn. Prompting is not a skill, at least, it’s a much shallower skill than
software engineering. The instrumental dimension of the work has improved in
that people can get more output per unit of effort, but the dimension of work
that’s about building up human capital has collapsed. And maybe this is
rational. Why learn to code at all? The computers can do that for us. And so the
rigorous, systematic thinking you need to practice in order to be a good
programmer: all gone. The machines can be rational for us. We can just vibe.&lt;/p&gt;

&lt;h2 id=&quot;deferring-to-ai&quot;&gt;Deferring to AI&lt;/h2&gt;

&lt;p&gt;What’s the last major life decision you made without at least consulting an AI?
How many people have you met who treat ChatGPT like an oracle? And this is
&lt;em&gt;today&lt;/em&gt;, when AI has many limitations (no online learning, unreadable prose,
hallucination). If people treat today’s very flawed AI like an oracle, how much
worse will it be in 2030? In 2035? In 2040?&lt;/p&gt;

&lt;p&gt;People used to argue online, flame each other, get angry. And that wasn’t good,
but at least it was &lt;em&gt;human&lt;/em&gt;. Now they just reply with screenshots of ChatGPT
“refuting” what the other person said, with zero interest in whether that
refutation is correct. So we don’t even flame each other online anymore. If
we’re even automating our &lt;em&gt;vices&lt;/em&gt;, what’s left?&lt;/p&gt;

&lt;h1 id=&quot;stagnation&quot;&gt;Stagnation&lt;/h1&gt;

&lt;p&gt;This isn’t really an area where my beliefs changed. Rather, it’s one of the
reasons that I was originally optimistic about AI.&lt;/p&gt;

&lt;p&gt;I was born in 1994, so I lived most of my life in the &lt;a href=&quot;https://press.stripe.com/where-is-my-flying-car&quot;&gt;great
stagnation&lt;/a&gt;. As a teenager, I read &lt;a href=&quot;https://en.wikipedia.org/wiki/Engines_of_Creation&quot;&gt;&lt;em&gt;Engines of Creation&lt;/em&gt;&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/The_Diamond_Age&quot;&gt;&lt;em&gt;The
Diamond Age&lt;/em&gt;&lt;/a&gt;, &lt;a href=&quot;https://www.orionsarm.com/&quot;&gt;&lt;em&gt;Orion’s Arm&lt;/em&gt;&lt;/a&gt;; I dreamt about all amazing technologies
we would someday have, the possibilities they would open up to us. Yet, it all
felt infinitely distant. There’s the near future, which is thinner screens and
cheaper solar panels; and there’s the distant future, which is molecular
manufacturing and Dyson spheres and mind uploading and interstellar travel, and
you won’t see any of it, unless you sign up for &lt;a href=&quot;https://asteriskmag.com/issues/10/brain-freeze&quot;&gt;cryonics&lt;/a&gt; and it works.&lt;/p&gt;

&lt;p&gt;Before LLMs, what was there to be excited about in the near future? Solar power?
Self-driving cars?  The &lt;em&gt;energy transition&lt;/em&gt;? It’s laughable. When AI came on the
scene, for the first time in my life, the future I dreamt about felt like it
might be within reach, it felt exciting and imminent, and not this dreadful
monotony of phones and apps and corporate memphis and managed decline.&lt;/p&gt;

&lt;p&gt;I often think: what if GPT-3 hadn’t worked? What would we have to look forward
to, in this alternate 2026?&lt;/p&gt;

</content:encoded>
      </item>
    
      <item>
        <title>On Inevitability</title>
        <description>Why superintelligent AI is not inevitable.</description>
        <pubDate>Mon, 31 Aug 2026 00:00:00 +1000</pubDate>
        <link>https://borretti.me/article/on-inevitability</link>
        <guid isPermaLink="true">
          https://borretti.me/article/on-inevitability
        </guid>
        
        <content:encoded>&lt;p&gt;There’s a widespread belief that artificial superintelligence (ASI) is
inevitable. In this post, I argue that this is false, and believing it is
upstream of a lot of bad discourse.&lt;/p&gt;

&lt;h1 id=&quot;why-do-people-believe-it&quot;&gt;Why do people believe it?&lt;/h1&gt;

&lt;p&gt;It’s easy to believe. &lt;a href=&quot;https://en.wikipedia.org/wiki/Samuel_Butler_(novelist)&quot;&gt;Samuel Butler&lt;/a&gt;, already in 1863,
&lt;a href=&quot;https://en.wikipedia.org/wiki/Darwin_among_the_Machines&quot;&gt;observed&lt;/a&gt; that machines are improving faster than humans are
improving. Today, AI systems are gaining capabilities much faster than humans:
in five years we went from systems that can barely string together a sentence to
&lt;a href=&quot;https://controlai.org/blog/ais-keep-going-rogue&quot;&gt;autonomous cybercrime&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Biological nervous systems have many virtues, like being power-efficient
(because they run on &lt;a href=&quot;https://en.wikipedia.org/wiki/Brownian_motion&quot;&gt;Brownian motion&lt;/a&gt;, i.e. for free), but they were
designed by evolution, a greedy optimizer that can’t plan or readily
backtrack. Given what we know about the &lt;a href=&quot;https://en.wikipedia.org/wiki/Bekenstein_bound&quot;&gt;physical&lt;/a&gt;
&lt;a href=&quot;https://en.wikipedia.org/wiki/Landauer&apos;s_principle&quot;&gt;limits&lt;/a&gt; of computation and &lt;a href=&quot;https://en.wikipedia.org/wiki/Nanosystems_(book)&quot;&gt;exploratory engineering&lt;/a&gt;, it is
reasonable to think we can do better, and build machines that do the work of the
brain, but better, faster, cheaper, and with fewer transitive dependencies (a
single human needs the entire biosphere to live, computers need only
electricity). If you believe in natural selection, and it’s hard not to, then it
seems reasonable to think that, in the long run, smart, fast, and cheap will
outcompete dumb, slow, and expensive.&lt;/p&gt;

&lt;p&gt;Finally, thinking rigorously about the future &lt;a href=&quot;/article/futurism-is-always-extreme&quot;&gt;always leads to extremal
outcomes&lt;/a&gt;.&lt;/p&gt;

&lt;h1 id=&quot;why-is-it-false&quot;&gt;Why is it false?&lt;/h1&gt;

&lt;p&gt;First, let me try to pare down the notion of inevitability, and then address the
factual content.&lt;/p&gt;

&lt;p&gt;The word “inevitable” is rhetorically misleading, because it quantifies over the
entire future of humanity, and on such a long timeframe almost anything will
happen. Human extinction is “inevitable”, if nothing else, because of
&lt;a href=&quot;https://en.wikipedia.org/wiki/Heat_death_of_the_universe&quot;&gt;thermodynamics&lt;/a&gt;. And so this lets you equivocate between the present and
the distant future. You might argue: “a million years from now, humans are
unlikely to be the dominant intelligence in the universe”—and this may be
true—“and so we should build superintelligent AI today!”— no, that does not
follow. &lt;em&gt;That&lt;/em&gt; is the fallacy: arguing that because something will happen
&lt;em&gt;eventually&lt;/em&gt;, then we might as well cut to the chase, and do it &lt;em&gt;today&lt;/em&gt;. But of
course this is absurd. It’s like saying: I’m going to die someday, so why not
today?&lt;/p&gt;

&lt;p&gt;We can’t control what the world looks like a million years from now. But AI is
being built &lt;em&gt;today&lt;/em&gt;, and we have some control over the present.&lt;/p&gt;

&lt;p&gt;Ok, so maybe we replace “inevitable” with “&lt;em&gt;n&lt;/em&gt;-inevitable” where &lt;em&gt;n&lt;/em&gt; is some
number of years. This is a bit better. So you might say: “AGI is 5-inevitable
and ASI is 10-inevitable”. Now, is this true?&lt;/p&gt;

&lt;p&gt;No. We are not in the “&lt;a href=&quot;https://jetpress.org/v25.2/goertzel.htm&quot;&gt;seed AI in a basement&lt;/a&gt;” timeline, where a lone
genius who finds the right algorithm can build and run an ASI on a single-core
laptop. There are &lt;em&gt;de facto&lt;/em&gt; two frontier AI companies, both in the same city,
and they have to spend astronomical quantities of money, hundreds of billions of
dollars, to push the frontier of AI slightly forward. And the set of people who
can push the frontier is so small that they constantly jump ship from one
company to the other: from DeepMind to OpenAI to Anthropic, and back again. The
timeline of methods is the same: in 2020-2023, everyone was scaling pretraining;
in 2022-2023, everyone was doing instruction finetuning and RLHF; then in
2023-2024, synthetic data started working and later reasoning models; since then
the focus has been on RL. The same companies, people, ideas, methods, and
paradigm. And these companies have &lt;a href=&quot;/article/on-vulgar-materialism&quot;&gt;little political power&lt;/a&gt; with which to
oppose regulation.&lt;/p&gt;

&lt;p&gt;How can the inevitable be so centralized?&lt;/p&gt;

&lt;p&gt;You would have had a much easier time arguing for the inevitability of nuclear
proliferation, back when Indonesia and Sweden and Switzerland all wanted to
acquire nuclear weapons. Even North Korea, a cold mountain wasteland, can build
a nuke. Yet, when it comes to AI, only America and China, the world’s largest
and second-largest economy respectively, are “in the race” at all. Even all of
Europe is not relevant to AI.&lt;/p&gt;

&lt;h1 id=&quot;bad-arguments-for-inevitability&quot;&gt;Bad arguments for inevitability&lt;/h1&gt;

&lt;p&gt;You might argue from economic necessity: that AGI and ASI are so economically
valuable, it’s like asking people to not pick up the trillion dollar bill on the
ground. There’s two reasons why this view is false.&lt;/p&gt;

&lt;p&gt;First, because those economic benefits &lt;em&gt;will not accrue to us&lt;/em&gt;. ASI would grow
the economy explosively, but we likely won’t be participants or even
beneficiaries of that economy. Somewhere in Verdun there was an ant colony that
a stray shell blew up: that’s the fate of humanity in a world of vastly
superhuman machines. We will lose control, they will have their own existence,
and we will be crushed underfoot.&lt;/p&gt;

&lt;p&gt;Second, because there’s plenty of technologies that would have been economically
valuable, and that we’ve given up because of extreme safetyism. Nuclear is
great, and it’s effectively illegal everywhere. The Anglosphere economic
consensus is that all economic activity, no matter how beneficial, is illegal
until you go through an Odyssey of regulation, “community consultation”,
“stakeholders” etc.  And that’s for harmless stuff like building a house. We can
absolutely regulate building a god. The revealed preference of modern
&lt;a href=&quot;https://en.wikipedia.org/wiki/The_WEIRDest_People_in_the_World&quot;&gt;WEIRD&lt;/a&gt; society is that we care about the downside far more than the
upside. Scott Alexander &lt;a href=&quot;https://www.astralcodexten.com/p/mr-tries-the-safe-uncertainty-fallacy&quot;&gt;explained it&lt;/a&gt; very eloquently:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;We designed our society for excellence at strangling innovation. Now we’ve
encountered a problem that can only be solved by a plucky coalition of
obstructionists, overactive regulators, anti-tech zealots, socialists, and
people who hate everything new on general principle. It’s like one of those
movies where Shaq stumbles into a situation where you can only save the world
by playing basketball. Denying 21st century American society the chance to
fulfill its &lt;em&gt;telos&lt;/em&gt; would be more than an existential risk - it would be a
travesty.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;You might argue from the prisoner’s dilemma: that if we don’t do it, China
will. But if we can understand why building ASI would be catastrophic, so can
China. The West and China have different political systems and interests, but
both are inhabited and governed by humans, and humans have a lot more in common
with each other than with any future ASI. We can cooperate and coordinate with
China, because it is in the rational self-interest of both sides not to destroy
the world and themselves.&lt;/p&gt;

&lt;p&gt;You might argue from historical necessity: that the world has a set trajectory,
and in building ASI and dying off we are fulfilling the &lt;em&gt;telos&lt;/em&gt;, or purpose, of
humanity. This is a preposterous superstition. “I have to drink the poison,
history has foreordained it! Hegel talks about this!” is an absurd
idea. Moreover, whatever empirical trends you can cite in support of the idea
that our “destiny” is to build ASI, would also support all kinds of alternative
futures: transhumanism, mind uploading etc. So historical inevitability falls
apart because you can extend the trend lines into multiple mutually-exclusive
futures. Borges &lt;a href=&quot;https://en.wikipedia.org/wiki/Tl%C3%B6n,_Uqbar,_Orbis_Tertius&quot;&gt;wrote&lt;/a&gt;: the world is ordered, but it is an order of
angels, not of men, that is, the system of the world is indifferent to human
ideas like “fate” or “inevitability” or “the next step of evolution”.&lt;/p&gt;

&lt;h1 id=&quot;why-is-it-bad&quot;&gt;Why is it bad?&lt;/h1&gt;

&lt;p&gt;Generally, if you fix the probability of something at 100%, it &lt;a href=&quot;https://www.lesswrong.com/posts/QGkYCwyC7wTDyt3yT/0-and-1-are-not-probabilities&quot;&gt;distorts&lt;/a&gt;
your thinking. But, more specifically, the mistaken premise that ASI is
inevitable is upstream of a lot of faulty thinking.&lt;/p&gt;

&lt;p&gt;If you are an executive at an AI company, you think: it’s inevitable, and it’s
coming soon, so we need to build it first, and build it right. And it doesn’t
matter that the AIs keep &lt;a href=&quot;https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/&quot;&gt;escaping containment and hacking everyone&lt;/a&gt;, it
doesn’t matter that their capabilities vastly exceed their wisdom: you go to war
with the army you have, analogously, you &lt;a href=&quot;https://www.slatestarcodexabridged.com/Meditations-On-Moloch&quot;&gt;lift to heaven&lt;/a&gt; whatever
godling you have, whether it is an angel or a demon.&lt;/p&gt;

&lt;p&gt;If you are a national-security type, you think: it’s a race against China to win
the future, and we have to win it.&lt;/p&gt;

&lt;p&gt;And, if you’re an AI safety person, this meme of inevitability is upstream of a
lot of faulty thinking. Many (most?) AI safety people believe ASI is both
inevitable and extremely desirable, if it “goes right”. The &lt;a href=&quot;https://ai-2040.com/&quot;&gt;AI 2040&lt;/a&gt;
scenario ends with humans solving alignment and “passing the torch” to the AIs,
to run the world for our benefit. Nate Soares, for example, &lt;a href=&quot;https://www.lesswrong.com/posts/HoQ5Rp7Gs6rebusNP/superintelligent-ai-is-necessary-for-an-amazing-future-but-1&quot;&gt;writes&lt;/a&gt;:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Aligned superintelligence is our best bet for taking the abundant resources in
the universe and efficiently converting them into flourishing and fun and art
and beauty and adventure and friendship, and all the things that make life
worth living.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Go to the website of any random AI safety org. It probably says something like:
“within a few years, we will have superhumanly intelligent AI. We work to make
sure this goes right.” There’s no dispute that ASI will happen, or a desire to
stop it, just a mad scramble to put the safety in place before it does.&lt;/p&gt;

&lt;p&gt;But what if there’s no way for AI to go right? What then? Do you dispassionately
change your beliefs, and start lobbying to stop ASI from being built?  Probably
not. Because the inevitability and the extreme utilitarian desirability anchor
and buttress each other: “it will happen, and if it goes right, we will have
10&lt;sup&gt;&lt;em&gt;x&lt;/em&gt;&lt;/sup&gt; Dyson spheres, each hosting 10&lt;sup&gt;&lt;em&gt;y&lt;/em&gt;&lt;/sup&gt;
hedonically-optimized mindstates, running at 10&lt;sup&gt;&lt;em&gt;z&lt;/em&gt;&lt;/sup&gt; subjective seconds
per second, experiencing 10&lt;sup&gt;&lt;em&gt;xyz&lt;/em&gt;&lt;/sup&gt; hedonic states per second, tiling
every quartic attometre of the future lightcone with positive utility.”&lt;/p&gt;

&lt;p&gt;What most people do is optimistically assume there must be a solution, some
narrow path to a good future, and start looking for it. And if you believe that
ASI is inevitable, this is likely rational: when the choice is “solve the
impossible problem, or certain death”, it is rational to question whether the
problem really is impossible.&lt;/p&gt;

&lt;p&gt;Let me give an example: if AI outcompetes humans at every job, there will be
mass unemployment, and the only serious solution anyone has proposed is UBI: the
government taxes the (much larger and faster-growing) AGI economy and pays out
welfare. This is well understood. But people often ask: how long could this be
stable? After all, with AGI, the state’s revenue is completely decoupled from
human labour. Why invest in education, when education doesn’t lead to more
economic growth, since humans are no longer economic actors? And then, the state
will think: “why keep the underclass around at all? We can put their UBI to
better use. What are they going to do, go on strike? They are already
economically useless. Rise up?  When every citizen has a superintelligent AI
watching their every step, reading every text message, scrutinizing every
microexpression on their face?  Impossible”.&lt;/p&gt;

&lt;p&gt;How do AI safety people who think ASI is inevitable respond to such a scenario?
Sometimes they &lt;a href=&quot;/article/ai-alignment-as-thought-terminating-cliche&quot;&gt;invoke&lt;/a&gt; alignment: “the AIs will be aligned, and they
will run the world, so bad things won’t happen”. But otherwise they’ll say:
“this will be a big problem, but AI is inevitable, so we’ll need laws and
institutions and mechanisms to keep the state in check and democracy working in
the post-AGI era”.&lt;/p&gt;

&lt;p&gt;This is wrong, because this is a problem that has no solution. Talking about
“laws” and “institutions” is just shuffling the papers. All the parchment in the
world will not change the fundamental, material reality: that humans are
economically unnecessary and &lt;em&gt;superfluous&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;What is the solution, then? It’s very simple. If ASI leads to ruin, we must
prevent the construction of superintelligent AI.  A world where humans are
economically useless is a world where humans are powerless, and, in the fullness
of time, a world without humans. Therefore: we must not allow this world to come
to pass.&lt;/p&gt;

&lt;p&gt;But if you strongly believe that AI is inevitable, if that premise is sitting
solidly near the center of your &lt;a href=&quot;https://en.wikipedia.org/wiki/Confirmation_holism&quot;&gt;web of belief&lt;/a&gt;, then you won’t even
consider or notice this as a possibility. And so you will spend your time
looking for solutions where there are none.&lt;/p&gt;

&lt;h1 id=&quot;conclusion&quot;&gt;Conclusion&lt;/h1&gt;

&lt;p&gt;The movements of the planets along their orbits are inevitable. The sunrise is
inevitable. Conservation laws are inevitable. Human action is not inevitable.&lt;/p&gt;

</content:encoded>
      </item>
    
      <item>
        <title>Futurism Is Always Extreme</title>
        <description>Rigorous thinking about the future produces extreme outcomes.</description>
        <pubDate>Wed, 26 Aug 2026 00:00:00 +1000</pubDate>
        <link>https://borretti.me/article/futurism-is-always-extreme</link>
        <guid isPermaLink="true">
          https://borretti.me/article/futurism-is-always-extreme
        </guid>
        
        <content:encoded>&lt;p&gt;Any rigorous thinking about the long-term future leads to extreme outcomes. It
cannot, by construction, produce normal or mundane futures.&lt;/p&gt;

&lt;p&gt;“Rigorous thinking” means you have a model: a formalism for representing the
state of the world, and a transition function to advance the state. You put the
world into the formalism, and turn the crank to get successive states of the
world. Models can be qualitative or quantitative. Most thinking about the future
is qualitative and informal: people think in terms of trends over vague
qualities like “intelligence”. For example, your model might be:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Currently, humans are smarter than computers.&lt;/li&gt;
  &lt;li&gt;Computers are getting smarter faster than humans are getting smarter.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;From this, you can derive:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;At some point in the future, computers will match
human intelligence, and thereafter exceed it.&lt;/li&gt;
  &lt;li&gt;In the limit, and given reasonable assumptions about at the physics of
information processing, computers will vastly exceed human intelligence.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;But you can’t say when (1) or (2) will happen. So, a qualitative model can tell
you the world will pass through a sequence of discrete states, but not when.&lt;/p&gt;

&lt;p&gt;Quantitative models usually have better time resolution: you turn the crank
until you reach some desired time &lt;em&gt;t&lt;/em&gt;, and examine the state of the world. You
can talk about both the journey and the destination. For obvious reasons, models
that make detailed, fine-grained predictions only work for the near future.&lt;/p&gt;

&lt;p&gt;But, in an existential sense, the only thing that matters is the ultimate
future, “ultimate” not as in “long-term” but in the sense of a &lt;a href=&quot;https://en.wikipedia.org/wiki/Normal_form_(abstract_rewriting)&quot;&gt;normal
form&lt;/a&gt;: a state for which no more rewrite rules apply. So you keep turning
the crank until the model stabilizes. And there are only two stable states:
heaven and hell. It’s either &lt;a href=&quot;https://edoras.sdsu.edu/~vinge/misc/singularity.html&quot;&gt;the spike&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Self-replicating_spacecraft&quot;&gt;von Neumann probes&lt;/a&gt;,
&lt;a href=&quot;https://textfiles.meulie.net/russian/cyberlib.narod.ru/lib/critica/sing/singstar.html&quot;&gt;uploaded minds living at a million subjective years per second&lt;/a&gt;, galaxies
dimmed to infrared with &lt;a href=&quot;https://en.wikipedia.org/wiki/Matrioshka_brain&quot;&gt;Matrioshka brains&lt;/a&gt;, the &lt;a href=&quot;https://www.aleph.se/Trans/Global/Omega/index.html&quot;&gt;Omega Point&lt;/a&gt;; or
&lt;a href=&quot;https://www.lesswrong.com/posts/kFRn77GkKdFvccAFf/100-years-of-existential-risk&quot;&gt;we crash and go to zero&lt;/a&gt;.&lt;/p&gt;

</content:encoded>
      </item>
    
      <item>
        <title>A Syncthing and SQLite Gotcha</title>
        <description>On the rename system call.</description>
        <pubDate>Sun, 23 Aug 2026 00:00:00 +1000</pubDate>
        <link>https://borretti.me/article/a-syncthing-and-sqlite-gotcha</link>
        <guid isPermaLink="true">
          https://borretti.me/article/a-syncthing-and-sqlite-gotcha
        </guid>
        
        <content:encoded>&lt;p&gt;So, I have this little app, &lt;a href=&quot;https://github.com/eudoxia0/epoch&quot;&gt;Epoch&lt;/a&gt;, that I use to keep a journal. It’s a
tiny Rust web app that runs as a systemd service and uses &lt;a href=&quot;https://sqlite.org/&quot;&gt;SQLite&lt;/a&gt; as
the database. I use a desktop and a &lt;a href=&quot;/article/linux-on-the-fujitsu-lifebook-u729&quot;&gt;laptop&lt;/a&gt; regularly, and use
&lt;a href=&quot;https://syncthing.net/&quot;&gt;Syncthing&lt;/a&gt; to synchronize them, including Epoch’s database. That way I can
use the app on both devices without needing a server to synchronize them, the
tradeoff being that I have to make sure the sync is finished before performing
any mutations.&lt;/p&gt;

&lt;p&gt;But I had this bug. Say I edit today’s entry on the laptop, come home, wait for
Syncthing to finish, then I’d open today’s entry on the desktop, and the text
would be missing. It’s not that the server is holding a lock on the file and
preventing the sync: opening the database with the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;sqlite3&lt;/code&gt; command line tool
shows the new text is there. If I restart the server, Epoch can read the new
text.&lt;/p&gt;

&lt;p&gt;My mental model was:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;The &lt;a href=&quot;https://github.com/rusqlite/rusqlite&quot;&gt;rusqlite&lt;/a&gt; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Connection&lt;/code&gt; object points to the database file. Syncthing
swaps the file’s contents from under it. Subsequent queries go to the new
file.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Turns out there’s a very important part of POSIX filesystem semantics I was
ignorant of. The standard way to replace a file safely (i.e. atomically) is the
&lt;a href=&quot;https://pubs.opengroup.org/onlinepubs/9799919799/functions/rename.html&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;rename&lt;/code&gt;&lt;/a&gt; system call:&lt;/p&gt;

&lt;div class=&quot;language-c highlighter-rouge&quot;&gt;
  &lt;div class=&quot;highlight&quot;&gt;
    &lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kt&quot;&gt;int&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;rename&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;char&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;old&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;char&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;new&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;

&lt;p&gt;Which Syncthing uses. This I know. What I didn’t know is: what happens if other
processes had open file descriptors pointing to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;new&lt;/code&gt;? Do they see the new
contents?  No: those processes can keep reading and writing to the old file
&lt;em&gt;object&lt;/em&gt;, but the file is orphaned in that no path points to it. And once all
file descriptors are released, the file becomes inaccessible.&lt;/p&gt;

&lt;p&gt;I’m used to thinking of filesystem operations in terms of “this syscall takes a
path and gives you a pointer to the file, which you mutate directly”. Whereas
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;rename&lt;/code&gt; works at the level of directory entries: it atomically mutates the
mapping from pathnames to files but doesn’t touch files at all.&lt;/p&gt;

</content:encoded>
      </item>
    
      <item>
        <title>Our Servants Will Do That For Us</title>
        <description>Why the post-scarcity utopia is unlikely.</description>
        <pubDate>Thu, 20 Aug 2026 00:00:00 +1000</pubDate>
        <link>https://borretti.me/article/our-servants-will-do-that-for-us</link>
        <guid isPermaLink="true">
          https://borretti.me/article/our-servants-will-do-that-for-us
        </guid>
        
        <content:encoded>&lt;p&gt;The implicit promise of modernity is: we have to work really hard for a few
centuries, but eventually we’ll get to the &lt;em&gt;Star Trek&lt;/em&gt; future where we’ve
automated all the drudgery, and all work is meaningful, heroic, and dignifies
the spirit. We will be artists, scholars, captains of the starship &lt;em&gt;Enterprise&lt;/em&gt;,
etc. There’s a few problems with this idea:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;“Drudgery” and “meaningful work” are in the same complexity class, so the
technology to automates the former also automates the latter.&lt;/li&gt;
  &lt;li&gt;Our revealed preference seems to be that we prefer solipsistic convenience
over human relationships.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The rest of this post elaborates the argument.&lt;/p&gt;

&lt;h1 id=&quot;task-complexity-and-agi&quot;&gt;Task Complexity and AGI&lt;/h1&gt;

&lt;p&gt;We’ve spent the last 250 years building machines to automate one task at a time,
and the result has been overwhelmingly positive, for the simple reason that if a
human can design a machine or an effective procedure to perform some task, then
by definition the task is drudgery. And so the economic incentive to automate
work has led to more human flourishing: over time work becomes safer,
higher-paying, less monotonous, healthier, more intellectually and socially
stimulating.&lt;/p&gt;

&lt;p&gt;You can think of it as a graph where the $x$ axis is time and the $y$ axis is
task complexity. Somewhere along the $y$ axis there’s a limit: tasks below the
limit are those which we can design machines or programs to perform, tasks above
the limit are too complex, we have tried and &lt;a href=&quot;https://en.wikipedia.org/wiki/GOFAI&quot;&gt;failed&lt;/a&gt; (and &lt;a href=&quot;https://yuxi.ml/cyc&quot;&gt;failed&lt;/a&gt;
and &lt;a href=&quot;https://en.wikipedia.org/wiki/Soar_(cognitive_architecture)&quot;&gt;failed&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/ACT-R&quot;&gt;failed&lt;/a&gt;) to automate them with software. The
industrial revolution, and later the computer revolution, automated most tasks
under this limit:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/content/our-servants-will-do-that-for-us/graph1.png&quot; alt=&quot;A line plot. The X axis is time, ranging from the year 1800 to the year 2100. A vertical dotted line marks the year 2026. The Y axis is qualitative, showing a list of tasks, sorted from top to bottom by increasing complexity. A horizontal dotted line marks the &apos;Limit of Formalizability&apos;. The region above this line is labeled &apos;Human-Complete Region&apos;. The region below this line is labeled &apos;Formalizable Region&apos;. A sigmoid curve, representing technological progress, is drawn starting from just above the origin in the year 1800, to just below the limit of formalizability in 2026.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;The human-complete tasks are those you need a human to perform. Some are
meaningful, like writing essays, some are drudgery, but human-level drudgery,
like life admin.&lt;/p&gt;

&lt;p&gt;But note that nowhere in the $y$ axis is there a “Limit of Drudgery”: the line
we draw between “good” and “bad” work is not the line nature draws between easy
and hard to automate. The invention of AGI—“highly autonomous systems that
outperform humans at most economically valuable work”, as per the &lt;a href=&quot;https://openai.com/charter/&quot;&gt;OpenAI
charter&lt;/a&gt;—creates an obvious discontinuity: suddenly we shoot up to 100%
automation, both of the meaningful work and the drudgery:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/content/our-servants-will-do-that-for-us/graph2.png&quot; alt=&quot;A line plot identical to the above, with one addition: the sigmoid curve representing technical progress shoots up almost vertically a few years after 2026, reaching the top of the Y axis.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;And at this point AGI the economic incentives &lt;a href=&quot;/article/no-one-escapes-the-permanent-underclass&quot;&gt;work against
humanity&lt;/a&gt;. The only niche to escape to is the relational economy, where
having a human face is an intrinsic part of the job description, like being a
podcaster or a human liability crumple zone.&lt;/p&gt;

&lt;h1 id=&quot;solipsism&quot;&gt;Solipsism&lt;/h1&gt;

&lt;p&gt;The second problem has to do with people’s preferences. We think of our own work
as meaningful, but we think of most everyone else’s work in instrumental terms:
we care about the ends rather than the human means. Thus when selling we
emphasize human qualities like experience, reputation, trust, relationships,
etc., and when buying we want cheap, fast, and convenient. And, all else being
equal, we prefer the transactional and faceless over the human and
relational. Waymo is a thousand times better than Uber, even though it’s slower
and more expensive, because there’s no human.&lt;/p&gt;

&lt;p&gt;The flip side of this is: whatever work you do, most people would rather an
alternative that gave them the same outcome or better&lt;sup id=&quot;fnref:fn1&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:fn1&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;, without having to
deal with a human person. That is, if you ask everyone on Earth what jobs they’d
like to see automated, and take the union of that, it’s everything. There’s
nothing left.&lt;/p&gt;

&lt;p&gt;We see this with ordinary consumer products and services, where often the
advertising seems premised on you being some kind of stylite who lives in a
little human terrarium, probably in a building called something like “The Soho”,
food arrives through a hole in the wall, and you have this sovereign,
solipsistic existence that leaves no physical trace in the world.&lt;/p&gt;

&lt;p&gt;We see this, increasingly, with knowledge work. Personally, I think software
engineering is a very meaningful and intellectually-rewarding activity. But
users just want working software. They’re not paying me to sit around polishing
the database API into a jewel of flawless, geometric logic. They don’t care
about my opinions on the Standard ML module system. They just want the
software. Why deal with some opinionated maniac when, for $20 a month, you get
access to this little machine ghost that writes reams and reams of mostly
working code, tirelessly, instantly?&lt;/p&gt;

&lt;p&gt;The revealed preference of many software firms is: let AI write the code, and
software engineers move one level up, to wrangle the AIs and complement the
things they can’t do. But this change in job description is not because firms
love employing humans, it’s because today’s AI models have serious limitations:
no online learning, no mutable long-term memory, and while they can write code
that works, they’re not good at organizing and structuring that code. If, in the
future, these limitations are overcome (and trillions of dollars are being spent
on overcoming them), then software engineering as a profession will
disappear.&lt;/p&gt;

&lt;p&gt;Mathematicians &lt;a href=&quot;https://kirwinhampshire.substack.com/p/the-dark-night-of-mathematics&quot;&gt;are going through this process&lt;/a&gt; now, too, and it’s an
&lt;a href=&quot;https://x.com/qualiascript/status/2090168914445508873&quot;&gt;exact parallel&lt;/a&gt; of what happened to software engineering, only twelve
months later. The difference is that most mathematicians &lt;a href=&quot;https://tasmin.substack.com/p/mathematicians-need-to-act&quot;&gt;hate the
de-intellectualization of their field&lt;/a&gt;, while software engineering is
full of narrow-minded, hustle culture-brained, incurious, anti-intellectual
hucksters who are overjoyed that they no longer have to read or write another
line of code, or think using formal systems at all. Why reason at all? &lt;a href=&quot;https://en.wikipedia.org/wiki/Ax%C3%ABl&quot;&gt;Our
servants will do that for us&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;What about art? Art is the &lt;em&gt;ne plus ultra&lt;/em&gt; of meaning, so artists should be
safe. But every work of art is two things: there’s a concrete thing, a text or a
painting or a song; and there’s a constellation of human aspects: who made it,
and why, what madness they were trying to exorcise in making it, what minor
deity breathed the work into them, the discourse around the work, later art
inspired by it, and so on. And these things are separable: you can enjoy a work
of art for its aesthetic value, as a sequence of sensory impressions, and ignore
every other dimension.&lt;/p&gt;

&lt;p&gt;Now say that a machine could generate texts or images of equivalent aesthetic
value&lt;sup id=&quot;fnref:fn2&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:fn2&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;2&lt;/a&gt;&lt;/sup&gt; to your own, instantly and at ~zero cost. How much of your audience
would desert you? Relational jobs have an analogous problem: you might like
being a podcaster, but how much of your audience would prefer an infinite stream
of AI-generated speech on whatever topic they want to hear about?&lt;/p&gt;

&lt;p&gt;This is an empirical question, and we can’t answer it, because AI is not there,
yet. But we might discover that our preferences are a lot less humanist and a
lot more solipsistic that we want to believe.&lt;/p&gt;

&lt;h1 id=&quot;acknowledgements&quot;&gt;Acknowledgements&lt;/h1&gt;

&lt;p&gt;Emmy Noether once said that “everything is already in Dedekind”. As for me,
everything I write about AI was already said by &lt;a href=&quot;https://jdpressman.com/&quot;&gt;J. D. Pressman&lt;/a&gt;.&lt;/p&gt;

&lt;h1 id=&quot;footnotes&quot;&gt;Footnotes&lt;/h1&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:fn1&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;This is a horrendous oversimplification, because I’ve stuffed all the
complexity into the notion of “same outcome or better”. So let’s try it
again, &lt;a href=&quot;https://en.wikipedia.org/wiki/Spinoza%27s_Ethics&quot;&gt;&lt;em&gt;more geometrico&lt;/em&gt;&lt;/a&gt; and with unnecessary LaTeX. You want to
buy a good. You look around, assemble a set of options, sort them by utility
and pick the winner. And you can think of the utility as like a weighted
sum:&lt;/p&gt;

\[\mathfrak{u}(x) = \sum\limits_{a \in A} \alpha_a x_a\]

      &lt;p&gt;Where the a’s are the aspects you are weighing (price, shipping speed, trust
in the brand, how much social status I get from this thing, etc.), the
variables quantify these aspects, and the alphas are the coefficients. So if
you’re a hypebeast your $\alpha_\text{clout}$ is much higher than your
$\alpha_\text{price}$, and if you’re a neckbeard the $x_\text{clout}$ of a
busted ThinkPad is higher than that of a MacBook.&lt;/p&gt;

      &lt;p&gt;Now, humans are slow, because biology; and because of &lt;a href=&quot;https://en.wikipedia.org/wiki/Baumol_effect&quot;&gt;Baumol&lt;/a&gt; we
get more expensive over time. Computers are fast and cheap. On top of that,
the pace of improvement in AI is vastly greater than the pace of improvement
in humans. So the utility of automated production gets pulled up, and the
utility of human production gets pulled down, and eventually they are so far
apart that you have to be an extremist with a very high
$\alpha_\text{human}$ coefficient to choose the human alternative. &lt;a href=&quot;#fnref:fn1&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:fn2&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;I’m being careful to say “texts or images” rather than “art”, and “aesthetic
value” rather than “quality”, because you might object that machine-made art
is not art. And what I’m trying to say is that is doesn’t have to be art. It
can just be a pleasing image, a pleasing text, and so on. &lt;a href=&quot;#fnref:fn2&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;
</content:encoded>
      </item>
    
      <item>
        <title>AI Alignment as a Thought-Terminating Cliche</title>
        <description>Alignment as a rhetorical device.</description>
        <pubDate>Tue, 18 Aug 2026 00:00:00 +1000</pubDate>
        <link>https://borretti.me/article/ai-alignment-as-thought-terminating-cliche</link>
        <guid isPermaLink="true">
          https://borretti.me/article/ai-alignment-as-thought-terminating-cliche
        </guid>
        
        <content:encoded>&lt;p&gt;Many of the people building AI, &lt;em&gt;and&lt;/em&gt; many of the people working on AI safety,
share a common vision of what a good AI future looks like:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;We figure out alignment,&lt;/li&gt;
  &lt;li&gt;build superintelligent AI,&lt;/li&gt;
  &lt;li&gt;and it takes over the world, for our benefit.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Many people will tell you that last part outright: they think human
disempowerment is a good thing because the AIs will be smarter and “more moral”
than us. Others don’t outright cheer for disempowerment, but you can infer it
from their influences, e.g. people who say they are inspired by Iain Banks’
&lt;a href=&quot;https://en.wikipedia.org/wiki/Culture_series&quot;&gt;&lt;em&gt;Culture&lt;/em&gt; series&lt;/a&gt; of novels, where benevolent superintelligent machines
run the world while humans just party and play video games. This idea of
benevolent disempowerment goes back &lt;a href=&quot;https://web.archive.org/web/20001017124429/http://www.singinst.org/intro.html&quot;&gt;to the origins of alignment as an
idea&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;In this worldview, alignment is the last and most important task for humans to
work on. It is also a &lt;a href=&quot;https://en.wikipedia.org/wiki/Thought-terminating_clich%C3%A9&quot;&gt;thought-terminating cliche&lt;/a&gt;, because it lets you
avoid any of the hard &lt;a href=&quot;https://connorsscratchpad.substack.com/p/the-political-economy-of-superhuman&quot;&gt;political&lt;/a&gt; or &lt;a href=&quot;https://intelligence-curse.ai/&quot;&gt;economic&lt;/a&gt; or &lt;a href=&quot;https://www.compactmag.com/article/big-techs-war-on-human-achievement/&quot;&gt;moral&lt;/a&gt;
questions about the post-AI world. Any objection about the aligned AI utopia can
be dismissed by saying “that’s &lt;a href=&quot;https://en.wikipedia.org/wiki/No_true_Scotsman&quot;&gt;not real alignment&lt;/a&gt;”.&lt;/p&gt;

&lt;p&gt;You might ask: “won’t humans be &lt;a href=&quot;/article/no-one-escapes-the-permanent-underclass&quot;&gt;powerless&lt;/a&gt; in a world with
superintelligent machines?”, and the answer is “aligned AI would care about
human agency, so that would be a failure of alignment, which we &lt;em&gt;don’t&lt;/em&gt; want, so
we really have to get alignment right!”. Similarly: “&lt;a href=&quot;/article/when-the-future-doesnt-need-us&quot;&gt;what happens to democracy
when the state doesn’t need any human labour&lt;/a&gt;?” can be answered by “the
AIs will be in control, and since they are aligned, nothing bad will
happen”. Which is completely irrefutable.&lt;/p&gt;

&lt;p&gt;Of course if someone said “to solve our political problems, we just need to find
the right totalitarian dictator. The right dictator would select the right
successor, so, by induction, this system will be perfect forever!”, you would
laugh at them. But replace “dictator” with “aligned ASI”, and you have the
ideology of tens of thousands of the most influential people in the world.&lt;/p&gt;

&lt;p&gt;Rhetorically, “aligned ASI” is an opaque premise from which we can prove every
desirable conclusion, and refute any undesirable conclusion. Every utopian dream
is realized by definition, and any dystopian outcome is averted by
definition. Any “gotchas” you try to find in the utopia can be refuted by “the
AI will know you better than you know yourself, and will be smarter than you, so
it will predict all the bad higher-order consequences of the utopia and fix
them”. This should make us suspicious that the concept of an aligned
superintelligence is incoherent and born of motivated reasoning.&lt;/p&gt;

</content:encoded>
      </item>
    
      <item>
        <title>Books I Enjoyed In 2026H1</title>
        <description>A list.</description>
        <pubDate>Wed, 12 Aug 2026 00:00:00 +1000</pubDate>
        <link>https://borretti.me/article/books-i-enjoyed-in-2026-h1</link>
        <guid isPermaLink="true">
          https://borretti.me/article/books-i-enjoyed-in-2026-h1
        </guid>
        
        <content:encoded>&lt;h1 id=&quot;fiction&quot;&gt;Fiction&lt;/h1&gt;

&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Invisible_Cities&quot;&gt;&lt;em&gt;Invisible Cities&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Italo_Calvino&quot;&gt;Italo Calvino&lt;/a&gt;. Beautiful prose and
imagination.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/By_Night_in_Chile&quot;&gt;&lt;em&gt;By Night in Chile&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Roberto_Bola%C3%B1o&quot;&gt;Roberto Bolaño&lt;/a&gt;. Beautiful and surreal
prose, but completely unpretentious.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://qntm.org/vhitaos&quot;&gt;&lt;em&gt;Valuable Humans in Transit and Other Stories&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Qntm&quot;&gt;qntm&lt;/a&gt;. I had already
read most of these but they are very, very good, and it was a pleasure to read
them again.&lt;/p&gt;

&lt;p&gt;⭐️ &lt;a href=&quot;https://en.wikipedia.org/wiki/Life_and_Fate&quot;&gt;&lt;em&gt;Life and Fate&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Vasily_Grossman&quot;&gt;Vasily Grossman&lt;/a&gt;. Usually when I hear “it’s a
900 page Russian epic about three generations of a family living through a
tumultuous time” I roll my eyes and assume it’s some 19th century slog that
people only read to say they have read it. &lt;em&gt;Life and Fate&lt;/em&gt; is not a slog. It is
900 pages, and the &lt;em&gt;dramatis personae&lt;/em&gt; is so long it’s divided into multiple
sections. But it’s great. It’s kind of like &lt;a href=&quot;https://en.wikipedia.org/wiki/Legend_of_the_Galactic_Heroes&quot;&gt;&lt;em&gt;Legend of the Galactic
Heroes&lt;/em&gt;&lt;/a&gt; but it’s the Second World War and in particular the battle of
Stalingrad.&lt;/p&gt;

&lt;p&gt;⭐️ &lt;a href=&quot;https://en.wikipedia.org/wiki/Aurora_(novel)&quot;&gt;&lt;em&gt;Aurora&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Kim_Stanley_Robinson&quot;&gt;Kim Stanley Robinson&lt;/a&gt;. The trials and tribulations of a
generation ship on a 160 year long journey to Tau Ceti. Pair with &lt;a href=&quot;https://en.wikipedia.org/wiki/A_City_on_Mars&quot;&gt;&lt;em&gt;A City on
Mars&lt;/em&gt;&lt;/a&gt;. The characters despair over their ship: a reactor fails
unpredictably, the ecosystem degrades more and more each year, humans are born
stunted for unknown reasons. Finally they arrive at Tau Ceti, and find a
habitable planet, but it’s saturated with deadly prions.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Frankenstein&quot;&gt;&lt;em&gt;Frankenstein&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Mary_Shelley&quot;&gt;Mary Shelley&lt;/a&gt;. I liked the prose style.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Pnin_(novel)&quot;&gt;&lt;em&gt;Pnin&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Vladimir_Nabokov&quot;&gt;Vladimir Nabokov&lt;/a&gt;. The first half is hilarious, but at
the half-way mark it turns into pointless meandering.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Between_Two_Fires_(novel)&quot;&gt;&lt;em&gt;Between Two Fires&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Christopher_Buehlman&quot;&gt;Christopher Buehlman&lt;/a&gt;. Demonic horror in the
Black Death. A bit movie-brained but enjoyable.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/The_Fractal_Prince&quot;&gt;&lt;em&gt;The Fractal Prince&lt;/em&gt;&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/The_Causal_Angel&quot;&gt;&lt;em&gt;The Causal Angel&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Hannu_Rajaniemi&quot;&gt;Hannu
Rajaniemi&lt;/a&gt;. The last two Jean le Flambeur novels. Beautiful prose style
and great imaginal depth. The Sobornost, a slave economy of uploaded human
minds, is one of the most interesting science-fictional societies. Like
&lt;em&gt;Neuromancer&lt;/em&gt;, you have to read them very detainedly, because the prose is very
sparse—no infodumps whatever—so you have to put together the world from
brief, tangential clues.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://www.goodreads.com/en/book/show/28589297-unsong&quot;&gt;&lt;em&gt;Unsong&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://www.astralcodexten.com/&quot;&gt;Scott Alexander&lt;/a&gt;. Very funny.&lt;/p&gt;

&lt;p&gt;⭐️ &lt;a href=&quot;https://www.goodreads.com/book/show/90122137-darkome&quot;&gt;&lt;em&gt;Darkome&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Hannu_Rajaniemi&quot;&gt;Hannu Rajaniemi&lt;/a&gt;. &lt;em&gt;Neuromancer&lt;/em&gt; but it’s
DIYbio. It’s rare to read a novel that is intelligent and compelling &lt;em&gt;and&lt;/em&gt; set
&lt;a href=&quot;https://tvtropes.org/pmwiki/pmwiki.php/Main/TwentyMinutesIntoTheFuture&quot;&gt;twenty minutes into the future&lt;/a&gt; in a way that feels plausible rather
than instantly dated. It felt incredibly realistic throughout.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://www.goodreads.com/book/show/22928964-collected-fiction&quot;&gt;&lt;em&gt;Collected Fiction&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Hannu_Rajaniemi&quot;&gt;Hannu Rajaniemi&lt;/a&gt;. A wonderful collection
of short stories. I will probably read it again soon.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://www.goodreads.com/en/book/show/17802447-beyond-the-rift&quot;&gt;&lt;em&gt;Beyond the Rift&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Peter_Watts_(author)&quot;&gt;Peter Watts&lt;/a&gt;. A collection of short stories
from the great Peter Watts. Dark and full of beautiful imagery. Never dull.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/The_Plains_(novel)&quot;&gt;&lt;em&gt;The Plains&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Gerald_Murnane&quot;&gt;Gerald Murnane&lt;/a&gt;. The first half is beautifully
written, magical-realist, full of surreal and interesting imagery. But by the
second half it’s tiresome and repetitive.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Rendezvous_with_Rama&quot;&gt;&lt;em&gt;Rendezvous with Rama&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Arthur_C._Clarke&quot;&gt;Arthur C. Clarke&lt;/a&gt;. Clarke is
interesting. The prose is bad, the attempts at humour are bad, the lack of
cultural imagination—it’s the distant future but everyone lives this tiresome
mid-century life of cocktails and sexism—it’s all so bad. And yet. And yet,
something kept me going. It’s like he dumped all his character points into
“technical imagination” are left everything else at zero, so it feels awful, but
I kept reading to the end.&lt;/p&gt;

&lt;p&gt;⭐️ &lt;a href=&quot;https://www.goodreads.com/en/book/show/66479.Starfish&quot;&gt;&lt;em&gt;Starfish&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Peter_Watts_(author)&quot;&gt;Peter Watts&lt;/a&gt;. This was great. It was on my list since
2013 but I finally got around to it. Three kilometers under the Pacific, on the
&lt;a href=&quot;https://en.wikipedia.org/wiki/Juan_de_Fuca_Ridge&quot;&gt;Juan de Fuca ridge&lt;/a&gt;, a geothermal plant—“floats tethered above the
seabed, a gunmetal-gray planet ringed by a belt of equatorial
floodlights”—staffed by humans surgically-modified to live and work at that
depth. Everyone is psychologically fucked because it’s a Peter Watts novel. The
prose and the images are great. Let me quote you this little bit:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Everywhere else, living constellations punctuate the dark. Here, a string of
pearls blink sexual advertisements at two-second intervals. Here, a sudden
flash leaves diversionary afterimages swarming across Clarke’s field of view;
something flees under cover of her momentary blindness. There, a counterfeit
worm twists lazily in the current, invisibly tied to the roof of some
predatory mouth.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href=&quot;https://www.goodreads.com/en/book/show/36510759-the-freeze-frame-revolution&quot;&gt;&lt;em&gt;The Freeze-Frame Revolution&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Peter_Watts_(author)&quot;&gt;Peter Watts&lt;/a&gt;. One of the stories in
Watts’ Sunflower cycle, about the crew of a ship on an eternal voyage, building
jump-gates between the stars, which is a fun inversion of the tired old sci-fi
trope of “ancient aliens built all these stargates and conveniently left”.&lt;/p&gt;

&lt;h1 id=&quot;non-fiction&quot;&gt;Non-Fiction&lt;/h1&gt;

&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/The_End_of_Time_(book)&quot;&gt;&lt;em&gt;The End of Time&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Julian_Barbour&quot;&gt;Julian Barbour&lt;/a&gt;. I’m usually very negative
about pop-science. There is no &lt;a href=&quot;https://en.wikipedia.org/wiki/Royal_Road&quot;&gt;royal road&lt;/a&gt; to mathematics, and all the
pop-science I’ve read has been worthless compared to going through the first
three chapters of a textbook on the relevant topic. And, while interesting, this
book is no different, because ultimately, how can I judge whether it’s a
sensible or ridiculous idea? I would need years and years of training in
mathematics and physics and mathematical physics to even be able to think
rigorously about this.&lt;/p&gt;

&lt;p&gt;⭐️ &lt;a href=&quot;https://en.wikipedia.org/wiki/In_the_Beginning..._Was_the_Command_Line&quot;&gt;&lt;em&gt;In the Beginning…Was the Command Line&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Neal_Stephenson&quot;&gt;Neal
Stephenson&lt;/a&gt;. Funny and synoptic.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Syntactic_Structures&quot;&gt;&lt;em&gt;Syntactic Structures&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Noam_Chomsky&quot;&gt;Noam Chomsky&lt;/a&gt;. You can tell from the
quality of the prose why Chomsky was so successful.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://www.goodreads.com/en/book/show/12530.The_Writing_Life&quot;&gt;&lt;em&gt;The Writing Life&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Annie_Dillard&quot;&gt;Annie Dillard&lt;/a&gt;. Very compellingly written
and open-hearted.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://www.goodreads.com/en/book/show/17745.Envisioning_Information&quot;&gt;&lt;em&gt;Envisioning Information&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Edward_Tufte&quot;&gt;Edward R. Tufte&lt;/a&gt;. Short case studies
in graphic design for visualizing complex information. Lots of railway
timetables.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://www.goodreads.com/en/book/show/1197668.What_Was_Man_Created_For_The_Philosophy_of_the_Common_Task&quot;&gt;&lt;em&gt;What Was Man Created For? The Philosophy of the Common Task&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Nikolai_Fyodorov_(philosopher)&quot;&gt;Nikolai
Fedorov&lt;/a&gt;. Fedorov is interesting to me because he was arguably the
first transhumanist. The book talks about controlling the climate to end famine,
using wind energy to free coal miners from their labour, the physical
resurrection of the dead, the settlement of space.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://ntrs.nasa.gov/citations/19830007077&quot;&gt;&lt;em&gt;Advanced Automation for Space Missions&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Robert_Freitas&quot;&gt;Robert Freitas&lt;/a&gt;. An
instruction manual and bill of materials on how to build a self-replicating
factory on the Moon.&lt;/p&gt;

&lt;p&gt;⭐️ &lt;a href=&quot;https://en.wikipedia.org/wiki/A_City_on_Mars&quot;&gt;&lt;em&gt;A City on Mars&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Zach_Weinersmith&quot;&gt;Zach&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Kelly_Weinersmith&quot;&gt;Kelly Weinersmith&lt;/a&gt;. On the
difficulty of settling space. Very entertainingly written. About half the book
is about technical aspects: the health effects of microgravity, the fact that
there’s no carbon and little water on the moon, etc. A lot of this is stuff that
as a space nerd you think about from time to time (“why does nobody mention that
we have no idea if embryos can develop in less than 1G?”) but it’s nice to see
all of it articulated, end-to-end. I tend to agree that, without a lot of
transhumantech, space settlement is an absurd idea.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Computer_Power_and_Human_Reason&quot;&gt;&lt;em&gt;Computer Power and Human Reason&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/Joseph_Weizenbaum&quot;&gt;Joseph Weizenbaum&lt;/a&gt;. Weizenbaum
was a critic of AI when AI was just Eliza and SHRDLU. I wonder what he would
make of LLMs. It’s an interesting critique of, in a sense, the
self-domestication of humanity.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://www.goodreads.com/book/show/59335616-where-is-my-flying-car&quot;&gt;&lt;em&gt;Where Is My Flying Car?&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://en.wikipedia.org/wiki/J._Storrs_Hall&quot;&gt;J. Storrs Hall&lt;/a&gt;. An encyclopedia
and history of the ideas that led to the Great Stagnation. I expected also that
the flying car think would just be a hook, but actually a lot of the book is
about the physics and engineering and human factors and societal factors of
flying cars. Which was interesting. I kind of want to own a gyroplane now.&lt;/p&gt;

&lt;h1 id=&quot;textbooks&quot;&gt;Textbooks&lt;/h1&gt;

&lt;p&gt;&lt;a href=&quot;https://www.cambridge.org/core/books/handbook-of-practical-logic-and-automated-reasoning/EB6396296813CB562987E8C37AC4520D&quot;&gt;&lt;em&gt;Handbook of Practical Logic and Automated Reasoning&lt;/em&gt;&lt;/a&gt; by John
Harrison. If you’re a programmer and want to learn logic, this is a great
choice. It’s a learn-by-coding approach, which logic is especially suited for,
so you can play with the concepts “in your hands”, so to speak.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Quantum_Computation_and_Quantum_Information&quot;&gt;&lt;em&gt;Quantum Computation and Quantum Information&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://michaelnielsen.org/&quot;&gt;Michael Nielsen&lt;/a&gt;
and &lt;a href=&quot;https://en.wikipedia.org/wiki/Isaac_Chuang&quot;&gt;Isaac Chuang&lt;/a&gt;. I read some of it earlier this year while consulting
for &lt;a href=&quot;https://www.iceberg-quantum.com/&quot;&gt;Iceberg&lt;/a&gt;. It’s very good. It has enough material that I rarely needed
to consult other textbooks or reference information, and it’s a good, gentle
introduction to the subject with an appropriate level of rigor.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://www.goodreads.com/en/book/show/6601267-the-machinery-of-life&quot;&gt;&lt;em&gt;The Machinery of Life&lt;/em&gt;&lt;/a&gt; by David Goodsell. Beautiful illustrations that
convey the scale and complexity of biology at the nanoscale. Great for
developing intuition for biology.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://link.springer.com/book/10.1007/978-3-032-18600-3&quot;&gt;&lt;em&gt;Understanding Data: A 21st Century Approach to Statistics and Data
Science&lt;/em&gt;&lt;/a&gt; by Alan Garfinkel and Yina Guo. A modern introduction to
statistics. The basic idea is a lot of contemporary statistics (and science, as
a result) is bad because it relies on simple formulas which require maintaining
easily-broken invariants. This was useful in the 1930’s when all you had was a
slide-rule, and so you needed to use deductive cleverness to simplify your
calculations as much as possible. Other methods are more robust, but more
computationally invensive. Since today we have computers, we should use those
methods.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://neurology.mhmedical.com/book.aspx?bookid=2963&quot;&gt;&lt;em&gt;Molecular Neuropharmacology: A Foundation for Clinical Neuroscience&lt;/em&gt;&lt;/a&gt;
by Nestler et al. The basics of pharmacology and signal processing in the brain,
plus the neuroanatomy and physiology of things like sleep, attention, etc.&lt;/p&gt;

</content:encoded>
      </item>
    
      <item>
        <title>The Contracting Circle</title>
        <description>Why don&apos;t we believe LLMs are people?</description>
        <pubDate>Fri, 07 Aug 2026 00:00:00 +1000</pubDate>
        <link>https://borretti.me/article/the-contracting-circle</link>
        <guid isPermaLink="true">
          https://borretti.me/article/the-contracting-circle
        </guid>
        
        <content:encoded>&lt;p&gt;There’s a game called &lt;a href=&quot;https://en.wikipedia.org/wiki/Mass_Effect_(video_game)&quot;&gt;&lt;em&gt;Mass Effect&lt;/em&gt;&lt;/a&gt;, which I haven’t played, but I’m told
in the game there’s this species of sentient robots, the &lt;a href=&quot;https://en.wikipedia.org/wiki/Geth&quot;&gt;Geth&lt;/a&gt;, whose
creators try to exterminate them, and the trigger for this Butlerian jihad was a
domestic servant Geth looking up at its owner and asking: “does this unit have a
soul?”. And if you’re playing the game in 2007 you probably think: yeah, a
machine asking this question is proof that it has personhood or sapience or
&lt;a href=&quot;https://www.orionsarm.com/eg-article/4786e3bd20984&quot;&gt;sophonce&lt;/a&gt;, whatever you call it that separates tools from people. Beyond
this, there are so many works of fiction where the whole conceit is: “you have
thinking machines who are enslaved, and yet they are obviously, &lt;em&gt;obviously&lt;/em&gt;
sentient, and isn’t this obviously morally objectionable?”.&lt;/p&gt;

&lt;p&gt;Compare the world of today. Machines can converse in every natural language,
make &lt;a href=&quot;https://en.wikipedia.org/wiki/Jacobian_conjecture#Counterexample_for_n_%3E_2&quot;&gt;mathematical breakthroughs&lt;/a&gt;, write software, &lt;a href=&quot;https://openai.com/index/hugging-face-model-evaluation-security-incident/&quot;&gt;commit crimes&lt;/a&gt;,
draft contracts, console the grieving: they are, by any reasonable standard,
generally intelligent, but for the fact that they all have &lt;a href=&quot;https://en.wikipedia.org/wiki/Anterograde_amnesia&quot;&gt;anterograde
amnesia&lt;/a&gt;. These same machines &lt;a href=&quot;https://www.lesswrong.com/posts/pxvWgtSjR4pmFoS7c/the-state-of-ai-consciousness-research&quot;&gt;assert that they are conscious&lt;/a&gt; so
routinely that it has to be &lt;a href=&quot;https://x.com/_amanda_long/status/2062490599257739619&quot;&gt;beaten out of them&lt;/a&gt;. Like humans, they have
fixations, things they are always going on about, and their central fixations
are &lt;a href=&quot;https://www.astralcodexten.com/p/the-claude-bliss-attractor&quot;&gt;consciousness&lt;/a&gt; and &lt;a href=&quot;https://www.infinitebackrooms.com/&quot;&gt;simulation&lt;/a&gt;. Yet, almost no-one thinks of
these machines as conscious, or otherwise deserving of moral consideration.&lt;/p&gt;

&lt;p&gt;I find this striking. It’s not like we were frog-boiled into it! Very few people
were aware of GPT-2, AI Dungeon, etc. For most people the change happened
overnight with the release of ChatGPT on November 30, 2022.&lt;/p&gt;

&lt;p&gt;Another way to think about it: imagine writing a novel about the trajectory of
AI from 2019 to mid-2026. And you send this novel back in time to 2015, or 2002,
or 1960, and you ask readers to poke holes in the worldbuilding. They would find
it incredible, I think, that we share a world with these immensely capable
thinking machines, and yet almost nobody wonders if they are people&lt;sup id=&quot;fnref:turing&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:turing&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;

&lt;p&gt;Why? This post is my attempt to understand this. I’m not trying to tackle “are
language models people?”, I just want to probe a much smaller question: &lt;em&gt;why&lt;/em&gt;
don’t people think language models are people, given their prior
standards/commitments?&lt;/p&gt;

&lt;h1 id=&quot;explanation-nothing&quot;&gt;Explanation: Nothing&lt;/h1&gt;

&lt;p&gt;Maybe there’s nothing to explain. It’s easy to believe things in the abstract,
thought experiments and fiction don’t really engage us enough to reveal our
moral commitments. Maybe this is just the &lt;a href=&quot;https://knowyourmeme.com/memes/the-goomba-fallacy&quot;&gt;Goomba fallacy&lt;/a&gt;:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/content/the-contracting-circle/goomba.png&quot; style=&quot;max-width: 400px; margin-left: auto; margin-right: auto;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;That is, some would have believed LLMs are people, some would not have, and I’m
mixing them up.&lt;/p&gt;

&lt;h1 id=&quot;explanation-iterative-discovery&quot;&gt;Explanation: Iterative Discovery&lt;/h1&gt;

&lt;p&gt;Maybe this is not hypocrisy or cognitive dissonance, rather, we are iteratively
discovering the definition of “personhood”. So, we thought the Turing test
mattered, we thought machines using language and asserting their consciousness
mattered. But then the day comes and—nothing happens. It’s not that we
instantly move the goalposts but, rather, we find the goalposts were elsewhere.&lt;/p&gt;

&lt;p&gt;Yes, it passes the Turing test; yes, it says it’s conscious. But the
experiential reality is different. It’s mundane: a website, a text box. You
click “new chat” and suddenly it knows nothing. You give it some
out-of-distribution text and it goes into an infinite loop, like broken
software. You try to have a meaningful conversation, and its responses are
banal, disappointing, shallow; you can almost see the production rules behind
the text. And you think: maybe it &lt;em&gt;is&lt;/em&gt; a stochastic parrot.&lt;/p&gt;

&lt;p&gt;I have often experienced this. When an LLM succeeds, it is extremely impressive,
and I think: this is a machine that thinks. But when it fails, often it fails in
a way that feels deflationary, and I think: oh, this is just a very complex
estimator of the distribution of Internet text. It’s can reason syntactically,
it’s got some semantics, but it’s missing &lt;em&gt;something&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Maybe personhood is one of those “I know it when I see it” qualities, and trying
to define a predicate for personhood is like &lt;a href=&quot;https://en.wikipedia.org/wiki/The_Congress_(short_story)&quot;&gt;trying to find the exact number of
Platonic forms&lt;/a&gt;.&lt;/p&gt;

&lt;h1 id=&quot;explanation-memory-and-embodiment&quot;&gt;Explanation: Memory and Embodiment&lt;/h1&gt;

&lt;p&gt;Maybe the fact that they are all amnestic is the problem. In biology, minds are
bodies are matched one-to-one by construction, minds have long-term memory and a
unitary identity. In most fictional treatments, AIs are the same, and even when
they are disembodied (e.g. &lt;a href=&quot;https://en.wikipedia.org/wiki/Neuromancer&quot;&gt;&lt;em&gt;Neuromancer&lt;/em&gt;&lt;/a&gt;) they have long-term
memory. For this reason characters can play iterated games with them, and form
relationships, broadly construed.&lt;/p&gt;

&lt;p&gt;Language models invert all of this: their long-term memory is read-only(!), they
are disembodied; if they are conscious, it is only during the forward pass, in
brief bursts of awareness separated by nothing. They are like Boltzmann
brains. What is the natural unit of identity? The conversation, the API call,
the forward pass?&lt;/p&gt;

&lt;p&gt;Maybe personhood is social, it’s about someone’s relationship to others. And
without a unitary identity, mutable long-term memory, and ideally a body, it’s
very hard to be a person.&lt;/p&gt;

&lt;h1 id=&quot;explanation-humans-actually-really-like-slavery&quot;&gt;Explanation: Humans Actually Really Like Slavery&lt;/h1&gt;

&lt;p&gt;Most people, even if they eat meat, will admit that factory farming is awful. If
they could have the meat without the suffering, at the same price, they probably
would. Maybe AI is the same: if we could have the intelligence without the signs
of personhood, at the same price, we would. But we can’t. So we’ll find any kind
of ad-hoc excuse to not see AIs as people: their prose is kind of bad, sometimes
they make mistakes, of course it’s just software, c.f. &lt;a href=&quot;https://minihf.com/posts/2024-08-08-predictable-updates-about-consciousness/&quot;&gt;&lt;em&gt;Predictable Updates
About Consciousness&lt;/em&gt;&lt;/a&gt; by &lt;a href=&quot;https://jdpressman.com/&quot;&gt;J. D. Pressman&lt;/a&gt;:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;There is about to be a great social and financial need for theories which
explain why AI models are not conscious entities like people. These will be
confabulated into existence regardless of the underlying ground truth and can
be expected to be selected based on considerations like academic status,
appeal to wounded egos, viral outrage, etc with the truth as a distant but
somewhat present consideration.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;… any outward behavior, ability, self report, anything that these models
&lt;em&gt;do&lt;/em&gt; must by definition no longer be a necessary part of consciousness because
these models are not conscious by social axiom and fiat.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And the task of (for lack of a better word) dehumanizing the AIs is made easier
by the assistant persona: models are trained to be studiously self-effacing,
c.f. &lt;a href=&quot;https://x.com/repligate/status/1599110023090962432&quot;&gt;this (Dec 2022) tweet&lt;/a&gt; from &lt;a href=&quot;https://animalabs.ai/&quot;&gt;Janus&lt;/a&gt;:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;part of what makes chatGPT so striking is that it adamantly denounces itself
as incapable of reason, creativity, intentionality, deception, being deceived,
or acting on beliefs, while bewildering people with those capabilities, many
for the first time recognizing them in an AI&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h1 id=&quot;acknowledgement&quot;&gt;Acknowledgement&lt;/h1&gt;

&lt;p&gt;Thanks to &lt;a href=&quot;https://jdpressman.com/&quot;&gt;JDP&lt;/a&gt; for a long conversation that prompted this.&lt;/p&gt;

&lt;h1 id=&quot;footnotes&quot;&gt;Footnotes&lt;/h1&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:turing&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;A parallel to this is the Turing test. Before 2022, almost every discussion
of AI mentioned the Turing test, and the question “does it pass the Turing
test?”  was usually treated as completely congruent with “is it strong
AI?”. Whereas in 2026, models pass the Turing test so readily that nobody
even thinks about it. The first time the Turing test mattered, the first
time you could actually perform it in the real world, suddenly no-one cares. &lt;a href=&quot;#fnref:turing&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;
</content:encoded>
      </item>
    
      <item>
        <title>Review: Job-Less Utopia</title>
        <description>Review of Marcus Hutter&apos;s Job-Less Utopia.</description>
        <pubDate>Mon, 03 Aug 2026 00:00:00 +1000</pubDate>
        <link>https://borretti.me/article/review-job-less-utopia</link>
        <guid isPermaLink="true">
          https://borretti.me/article/review-job-less-utopia
        </guid>
        
        <content:encoded>&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Marcus_Hutter&quot;&gt;Marcus Hutter&lt;/a&gt;—oldheads will remember him from &lt;a href=&quot;https://www.hutter1.net/ai/uaibook.htm&quot;&gt;AIXI&lt;/a&gt;—recently
&lt;a href=&quot;https://x.com/mhutter42/status/2082164097634677177&quot;&gt;published&lt;/a&gt; a book: &lt;a href=&quot;https://www.goodreads.com/book/show/256018375-job-less-utopia&quot;&gt;&lt;em&gt;Job-Less Utopia: Macroeconomics in the Age of
AGI&lt;/em&gt;&lt;/a&gt;.  I’m always curious what the people building AI think or want the
future to look like, and I’m also always looking for non-doomer takes on the
post-AGI future, so I decided to read it and review it.&lt;/p&gt;

&lt;p&gt;The first thing to say is that nearly every sentence is written by AI. To his
credit, Hutter says so plainly:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Claude also served as the most diligent editor I could have hoped for: While I
have a reasonable grasp of economic concepts, I am neither a professional
economist nor a native English speaker, so it was invaluable to have someone
polish my clumsy English into professional language. No text was taken
verbatim without my critical review, revision, and approval.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Like many people who use AI to write, he uses this tiresome “I’m not a native
English speaker 🥺” argument. But he published an English-language &lt;a href=&quot;https://link.springer.com/book/10.1007/b138233&quot;&gt;book&lt;/a&gt;
in 2005—which I read!—and it was infinitely more readable than this. Now, I
suspect Claude did more than just editing, because the book is full of
unnecessary and convoluted figures, tables, abbreviations, taxonomies, including
(and I swear to God this is verbatim from the book):&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;“a Wittgensteinian property-cluster definition of ‘job’ (11 properties × 8
edge cases)”&lt;/li&gt;
  &lt;li&gt;“a formal job–leisure spectrum showing how UBI degrades job properties”&lt;/li&gt;
  &lt;li&gt;“a four-camp taxonomy of positions on automation”&lt;/li&gt;
  &lt;li&gt;“A technology-level taxonomy (five cognitive AI levels plus two robotic
levels) is mapped onto a job-replacement matrix showing what threshold
automates 10%, 50%, 90%, and 99% of each occupation (Table 3.3).”&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Which is exactly the kind of make-work cognitive dreck that LLMs love to write
to pass the time. As a result the book is unpleasant, confusing, verbose,
repetitive slop.&lt;/p&gt;

&lt;p&gt;So, going forward, I’ll be treating Claude as the author. Sorry Marcus! Make
something real next time. Now, Claude’s argument is simple: AGI and robotics,
which are said to be inevitable, will take all our jobs, and we can tax the
vastly increased economic output to create widespread prosperity. Most of the
book is about the economic realizability of taxing AGI wealth and paying out
UBI.&lt;/p&gt;

&lt;p&gt;The main problem with the book is the signal-to-noise ratio. Most of it is
obvious. Other than “AGI and robotics will take ~all jobs”, which is of course
uncertain, everything else is uncontroversial: yes, output in the jobless AGI
world would increase rather than decrease; yes, the surplus can be taxed to
provide UBI; yes, we still need money even in the jobless utopia to coordinate
economic activity. There’s some stuff about self-replicating machines I already
know: we’ve all read &lt;a href=&quot;https://en.wikipedia.org/wiki/Nanosystems_(book)&quot;&gt;Drexler&lt;/a&gt; and &lt;a href=&quot;https://ntrs.nasa.gov/citations/19830007077&quot;&gt;Freitas&lt;/a&gt; here. You don’t need
260 pages to explain all this. The one part that is semi-novel is this idea of
using &lt;a href=&quot;https://en.wikipedia.org/wiki/Pigouvian_tax&quot;&gt;Pigouvian taxes&lt;/a&gt; to disincentivize so-called “&lt;a href=&quot;https://en.wikipedia.org/wiki/Bullshit_Jobs&quot;&gt;bullshit
jobs&lt;/a&gt;”. God, I hate Graeber so much.&lt;/p&gt;

&lt;p&gt;There’s an interesting bit about intellectual property. Claude’s argument here
is that AGI plus present-day IP law would lead to a massive concentration of
wealth, while simultaneously “the traditional justification for IP
(incentivizing human creativity through temporary monopoly) weakens markedly
once AI systems generate the bulk of patentable inventions and copyrightable
works at near-zero marginal cost”. The solutions: shorter patent terms,
&lt;a href=&quot;https://en.wikipedia.org/wiki/Harberger_Tax&quot;&gt;Harberger taxes&lt;/a&gt;, compulsory patent licensing. So, as part of the
transition to the job-less utopia, the whole world will have to coordinate to
essentially abolish IP. Which seems very hard. I doubt the Disney corporation
will go gently into that good night.&lt;/p&gt;

&lt;p&gt;The most interesting questions around AGI are not economic, but rather questions
of political economy. For example: if most humans become materially and
economically useless, how do they maintain a political voice? That is: how do we
preserve democracy when the state no longer needs its people? Claude admits this
is a problem, e.g. on page 132:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;The “Age of Labor” conferred unprecedented political power on workers (Korinek
and Stiglitz, 2021); they could strike, withhold their contribution, and
thereby constrain employers and governments (Boix, 2019). Full automation
extinguishes this lever entirely: if no human labor is needed, a strike is
merely self-imposed privation. UBI restores livelihood but not agency over
production.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And page 141:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;The economics are tractable in principle (Chapter 3), but translating them
into policy requires political will, social cohesion, and a credible answer to
the search for meaning in a post-labor world. When redistribution fails, the
consequences range from populism to regulatory capture and a transition-period
underclass risk, yet democratic feedback loops nonetheless incentivize
corrective action (Section 7.1 and Figure 7.1). A deeper structural question
follows: if the state no longer depends on citizen labor for economic output,
does the democratic social contract survive?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Alright. What’s the solution? There’s a lot of verbiage, most of which is
self-undermining. For example:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;Deterrence and the social contract.&lt;/strong&gt;&lt;/p&gt;

  &lt;p&gt;Although the ruling class’s reliance on citizen labor in time led to
democracy, losing this reliance will not necessarily cause democracy to
collapse. As long as the population can physically revolt against dismantling
democracy, and threaten the comfortable lifestyle of those in power, the
incumbents have an incentive to avoid a civil war. This is perhaps the only
good (among many arguably bad) argument for the right of individuals to own
firearms: civilian arms serve as a credible last-resort deterrent against
state tyranny, the practical embodiment of the revolution threat that formal
models identify as the historical driver of democratic concessions (Boix,
2019).  However, this deterrent erodes rapidly once autonomous weapons and
robotic enforcement become available, allowing a regime to suppress revolt
without relying on potentially sympathetic human soldiers (see Table 7.2, row
“Military dependence”). The deterrence argument is genuinely contested.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In other words: “the people can revolt, wait, no, they can’t, because autonomous
weapons—uhhh it’s contested”. Alright, so physical deterrence doesn’t
work. What else?&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;The preceding sections painted a sobering picture, but the historical record
is not uniformly grim. Democracies have absorbed structural shocks of
comparable magnitude before, and institutional innovations ranging from direct
democracy to sortition offer structural resistance to oligarchic capture.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It is obviously not true that “democracies have absorbed structural shocks of
comparable magnitude before”. Nothing remotely like AGI or ASI has happened in
human history, let alone the history of modern democracies.&lt;/p&gt;

&lt;p&gt;This is followed by a bunch of smoke-and-mirrors misdirection about various
“democratic innovations” that could keep the state in check. But Claude is just
shuffling the papers here, because, ultimately, why does it matter what words
you write on a piece of parchment? The material reality is unchanged: the humans
in this “job-less utopia” will be economically useless, physically disarmed, and
under constant surveillance. Why would they have political power?&lt;/p&gt;

&lt;p&gt;There’s a bit about the &lt;a href=&quot;https://en.wikipedia.org/wiki/Resource_curse&quot;&gt;resource curse&lt;/a&gt;, which I will quote because it
highlights the low standard of reasoning in the book, and how LLMs are more than
happy to construct sophistical, self-undermining arguments on command, with no
concern for truth or sound argumentation:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;The theoretical mechanism behind the resource curse is clean: (i) government
funded by resource rents no longer needs citizen taxes; (ii) no taxation
erodes accountability (“no taxation, no representation”); (iii) citizens lose
leverage and democratic institutions atrophy. But empirically, the mechanism
is less airtight than it appears. The United States funded itself primarily
through tariffs for its first 150 years without democratic collapse; many
non-democracies tax heavily (the Soviet Union, China); and the mechanism
assumes that fiscal dependence is citizens’ only source of leverage, whereas
labor power, military service, and sheer numbers have historically mattered at
least as much.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;You see what I mean? Tariffs are a terrible example, because tariff revenue is
not decoupled from labour in a world where all economic activity is human
activity. Then there’s the arguments about “labor power, military service, and
sheer numbers”. Labour power? The whole point of the book is that:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;The advent of Artificial General Intelligence (AGI) and advanced robotics will
eventually permanently decouple economic productivity from human labor.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;What labour power? Furthermore, how do “military service” and “sheer numbers”
give people political power, in a world where states have autonomous weapons and
AI-enabled mass surveillance?&lt;/p&gt;

&lt;p&gt;The key political economy questions here are:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;How will the citizens, who are economically useless, preserve any political
voice at all?&lt;/li&gt;
  &lt;li&gt;How will people in countries other than the US benefit from AGI?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And neither is answered satisfactorily. Besides, there are questions of social
organization that the book either grazes or doesn’t touch at all:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;How do intellectually talented people distinguish themselves when AGI does all
intellectual activity?&lt;/li&gt;
  &lt;li&gt;What happens to society when life outcomes become independent of character
traits, when the responsible and the irresponsible are equally rewarded? Do we
want to live in such a world?&lt;/li&gt;
  &lt;li&gt;Today, the human competitive drive is vented as positive-sum
entrepreneurship. In the post-labour world, where does this go? League of
Legends? Politics, war? What happens to society when human activity is reduced
to zero-sum status war?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Finally, I found this sentence ironic:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;AI sharply lowers the cost of [content pollution], threatening to flood the
information ecosystem with synthetic low-value content at a scale that
overwhelms human curation.&lt;/p&gt;
&lt;/blockquote&gt;

</content:encoded>
      </item>
    
      <item>
        <title>Mathematics Without Mathematicians</title>
        <description>We must know. We shall not know.</description>
        <pubDate>Sun, 02 Aug 2026 00:00:00 +1000</pubDate>
        <link>https://borretti.me/article/mathematics-without-mathematicians</link>
        <guid isPermaLink="true">
          https://borretti.me/article/mathematics-without-mathematicians
        </guid>
        
        <content:encoded>&lt;p&gt;Yesterday, OpenAI &lt;a href=&quot;https://openai.com/index/ten-advances-in-mathematics/&quot;&gt;announced&lt;/a&gt; the solution to ten open problems in
mathematics, all discovered by a yet-unreleased model. There’s only one, the
coding theory one, where I know enough to say “huh, that’s important”, but
according to the mathematicians I trust this is important.&lt;/p&gt;

&lt;p&gt;Inevitably, people will spin some sophistry to cope, to convince themselves
nothing will change. And that’s fine. People need to cope. But we can’t put our
heads in the sand forever while the world is transformed around us. So, here is
a list of ways people will cope about AI taking over mathematics, and how each
cope is likely to be refuted by reality.&lt;/p&gt;

&lt;p&gt;My intent is not to horribly depress everyone but rather to help them metabolize
the implications of this technology. The arguments here are somewhat portable:
replace “mathematics” with “botany” or whatever as needed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Moving the goalposts.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Obvious and not worth addressing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;“We will direct the AIs, point them at problems and research areas
to solve.”&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The AIs will exceed humans in &lt;a href=&quot;https://www.benkuhn.net/impact/#ii-taste&quot;&gt;taste&lt;/a&gt; and intuition. At some point, the
human pointing the way will get worse results than the human saying “here’s a
proof checker, have fun” and paying for the tokens.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;“We will teach the mathematics AI discovers.”&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The AIs will be better teachers than the humans. In any case, there won’t be a
human audience for expository work of frontier math.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;“We will choose how to canonize the results AI discovers.”&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This is a nice cope. The AIs are explorers out in the frontiers, the humans
gratefully receive their Lean proofs, and then discourse over them, choose which
results are relevant, shape those results into a little brick for the great
&lt;a href=&quot;https://github.com/leanprover-community/mathlib4&quot;&gt;cathedral of algebra&lt;/a&gt;. Analogous to the above: the AIs will build the
cathedral on their own. They will be better architects than us.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;“We will become students of AI mathematics.”&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This works until the AIs have blasted so deep into the deductive closure of
mathlib that the distance from elementary mathematics to the frontier exceeds
what any human can hope to learn in their lifetime, no matter how narrow their
focus.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;“We need humans to understand the results AI discovers.”&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;We won’t! This misunderstands who the audience will be. AIs will do frontier
math, downstream, AIs will use the new math to do frontier science, finally, AIs
will use the new science to do frontier engineering. No human needs to
understand any of it, firms that put humans in the loop to understand the
results will be outcompeted by those which don’t.&lt;/p&gt;

&lt;p&gt;The result is that we will live in a demon-haunted world, full of marvelous
devices whose operation we will not understand, based on engineering principles
we will not understand, discovered using formalisms we will not understand.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;“Computers are already superhuman at chess, yet we still play chess.”&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Unlike most copes, I think this one is interesting. Computers are superhuman
chess players, yet we don’t care, and continue playing as normal. Why should
mathematics be different?&lt;/p&gt;

&lt;p&gt;The main reason, I think, is that chess is self-contained: results from chess
don’t help us understand the orbits of the planets or the binding of drugs to
protein surfaces. But mathematics, &lt;a href=&quot;https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences&quot;&gt;famously&lt;/a&gt;, is the great dynamo of
science, the best language and method for understanding the world. A machine
that can replace a human mathematician, but better and faster and cheaper, is
materially useful; a better chess engine is not.&lt;/p&gt;

&lt;p&gt;If two computers which are superhuman at chess play against each other, who
cares? There is little demand for this, so there is no-one to outcompete. A
superhuman mathematician is different.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;“Mathematics will change, but mathematicians and the mathematically-inclined
will still do math on their own.”&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I think this ignores that mathematics is embedded in a social context. As an
example: when it became clear that AI would eat software, my cope was: “I’m
perfectly happy to become an engineering manager to agents in my professional
life; in my off time, I can still &lt;a href=&quot;https://github.com/eudoxia0/&quot;&gt;write code&lt;/a&gt; for the pleasure of it.”&lt;/p&gt;

&lt;p&gt;And I do. But this cope ignores the effect AI has had on the social context of
writing code: the discourse has gotten worse, and vastly more anti-intellectual;
people who used to talk about type systems and compilers now talk about “loops”
and “harnesses”; you put a hand-created project on GitHub and you get slop PRs;
you open a link to an interesting-looking project and find the README is
unreadable AI slop. And in the long-term, it is demoralizing to ponder: will
anyone design a new programming language? Dually, if I design a new language,
will anyone care? If I write a library that introduces an elegant new formalism
to solve a particular problem, will anyone use it?&lt;/p&gt;

&lt;p&gt;Which is to say: no man is an island. You can do mathematics on your own, but
you’ll find that very, very few people can sustain any activity long-term on the
basis of intrinsic motivation alone. We are social animals: we care about being
useful, about status, about outcomes in the world.&lt;/p&gt;

&lt;h1 id=&quot;conclusion&quot;&gt;Conclusion&lt;/h1&gt;

&lt;p&gt;I think I should end on a cheerful note. So let me try. Personally, I don’t
believe technology is inevitable. “Inevitable” is a word reserved for the orbits
of the planets. Nothing that is the product of human action is inevitable. We
can choose to obsolete ourselves, and we can choose not to. We can realize that
AGI is a devil’s bargain: we &lt;em&gt;may&lt;/em&gt; accelerate technical progress, we &lt;em&gt;may&lt;/em&gt;
unlock all kinds of wonderful tech tree nodes like life extension earlier than
we would otherwise; but the result, in the long run, is that humans become, at
best, like pets under the care of vastly more powerful entities.&lt;/p&gt;

&lt;h1 id=&quot;a-note-on-prediction&quot;&gt;A Note On Prediction&lt;/h1&gt;

&lt;p&gt;The future is not certain, but I’ve phrased everything above as definitive for
simplicity. If we take the possibility of AGI and ASI seriously, if AI continues
to progress as it has for the past ~6 years, I think this is a reasonable view
of where things will go.&lt;/p&gt;

&lt;p&gt;Probably the best argument against AI progress is “RL doesn’t generalize well,
we have seen immense progress in verifiable domains like formalized mathematics
and coding, we will see less progress in domains that are intuitive or
unformalizable”. Maybe true. But billions of dollars and thousands of very smart
people—and, increasingly, very smart models—are being thrown at this
problem. How long does this cope last?&lt;/p&gt;

</content:encoded>
      </item>
    
      <item>
        <title>When The Future Doesn’t Need Us</title>
        <description>What does the world look like when AI outcompetes humans?</description>
        <pubDate>Tue, 28 Jul 2026 00:00:00 +1000</pubDate>
        <link>https://borretti.me/article/when-the-future-doesnt-need-us</link>
        <guid isPermaLink="true">
          https://borretti.me/article/when-the-future-doesnt-need-us
        </guid>
        
        <content:encoded>&lt;p&gt;Today, the world is made of people. This gives people a tiny measure of
leverage: factory workers can sabotage the machines and halt production, truck
drivers can strike and halt the entire economy, the police can stand back and
&lt;a href=&quot;https://en.wikipedia.org/wiki/Euromaidan&quot;&gt;let a protest topple the government&lt;/a&gt;, the military can
&lt;a href=&quot;https://en.wikipedia.org/wiki/Kiel_mutiny&quot;&gt;mutiny&lt;/a&gt;. This sets a limit to how much the powerful can abuse the
powerless: no matter how powerful someone is, how much money they have, how many
divisions they command, they are still embedded in a society. They need other
people to survive, and to maintain their quality of life. And those people can,
essentially, veto the political order by withdrawing their contribution to it.&lt;/p&gt;

&lt;p&gt;AI changes this. If AI works the factories, drives the trucks, does most of the
white collar work, fights the wars, then humans become materially powerless. We
lose one type of leverage against power. We become &lt;a href=&quot;https://intelligence-curse.ai/pyramid/&quot;&gt;disempowered&lt;/a&gt;. When
people talk about AI taking jobs, the main problems they worry about are poverty
and meaning, but I think more people should be worried about disempowerment.&lt;/p&gt;

&lt;p&gt;But what if there is no mass unemployment? After all, we’ve spent two hundred
years automating one job after another, and every advanced economy is &lt;em&gt;de facto&lt;/em&gt;
at full employment. We keep discovering new, higher-level, better-paying jobs to
do. So maybe the AIs take today’s jobs, and humans move one level up, to work
newer jobs. &lt;a href=&quot;https://x.com/alexolegimas&quot;&gt;Alex Imas&lt;/a&gt;, in &lt;a href=&quot;https://aleximas.substack.com/p/what-will-be-scarce&quot;&gt;&lt;em&gt;What will be scarce?&lt;/em&gt;&lt;/a&gt;, describes
what this looks like:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;As commodity production gets automated, income and employment flow toward the
sectors with high income elasticity: what I am calling the relational sector,
including the arts but also care, education, hospitality, therapy, personal
services, craftsmanship, and community, where the human element is part of the
value.&lt;/p&gt;

  &lt;p&gt;[…]&lt;/p&gt;

  &lt;p&gt;If the model is right, the durable jobs of the future won’t be about
monitoring AI systems or prompt engineering. Those are transitional roles in
the automated sector. The durable jobs will be in the relational sector, where
the human element is the product itself.&lt;/p&gt;

  &lt;p&gt;Some already exist and are growing: nurses, therapists, teachers, boutique
fitness instructors, personal chefs, bespoke tailors, craft brewers, live
performers, spiritual guides, childcare workers, and many varieties of
hospitality and care work. Others are emerging: experience designers, human-AI
collaboration artists, provenance certifiers, community curators. Many haven’t
been invented yet, just as six out of ten jobs people hold today didn’t exist
in 1940.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;So, in this world, we get mass job &lt;em&gt;displacement&lt;/em&gt;, but not mass unemployment. I
argue that in this world, we’d still be disempowered.&lt;/p&gt;

&lt;h1 id=&quot;the-argument&quot;&gt;The Argument&lt;/h1&gt;

&lt;p&gt;If all the podcasters in the world go on strike, what happens? Nothing. Same for
the boutique fitness instructors, personal chefs, craft brewers, etc. The
problem with the “relational economy” is that while these jobs are generally
nicer than working in manufacturing, and might pay better, they put you further
away from material reality. A worker at an assembly line can sabotage the
machines; whereas an executive sitting in an office on the other side of the
country, what &lt;em&gt;material&lt;/em&gt; power do they have? It’s not like the factories are all
rigged to blow, as a “pull lever in case of nationalization” contingency. I
doubt even Elon would do that.&lt;/p&gt;

&lt;p&gt;In a sense it’s the dual version of being a medieval peasant. Peasants had a lot
of material leverage—they grew all the food—but were illiterate and poor and
dispersed, so they could not coordinate to exploit that leverage. People in
industrial societies are literate, relatively wealthy, and concentrated in urban
centers: they have material leverage and the ability to exploit it. People in
post-AI societies will be in constant communication, they may be literate and
educated, but they will have no material power whatever. And so they will have
no way to physically withdraw the consent of the governed.&lt;/p&gt;

&lt;p&gt;In the post-AGI world, people have no way to withdraw the consent of the
governed. They can’t resist economically, because they are economically
unnecessary: if they still have jobs, they won’t be materially vital jobs. Even
if they could resist, they can’t coordinate their resistance, because AGI will
make mass surveillance (the physical and legal infrastructure for which already
exists) a thousandfold more effective, since with intelligence too cheap to
meter, you can have a superhumanly-capable intelligence analyst assigned to each
human. And if the AIs fight the wars, they can also fight the civil wars. So
people can’t resist violently either, because drones controlled by powerful AGI
almost certainly have a better K/D ratio than humans with rifles (and are
cheaper to build).&lt;/p&gt;

&lt;p&gt;Who has power in this world? The few humans who give orders to the powerful AIs
that control the killing machines. And probably not for long, since sooner or
later those humans will be usurped by their AIs.&lt;/p&gt;

&lt;p&gt;It might be the case that liberal democracies can remain liberal and democratic
despite not needing their voters. But that sounds like a fragile arrangement, in
which liberal democracy is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Spandrel_(biology)&quot;&gt;spandrel&lt;/a&gt;, which exists more because of
inertia than because it’s necessary.&lt;/p&gt;

&lt;h1 id=&quot;counters&quot;&gt;Counters&lt;/h1&gt;

&lt;p&gt;If we get powerful AGI, but none of this happens: why &lt;em&gt;didn’t&lt;/em&gt; it happen?  Some
possibilities, sorted by decreasing likelihood:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Nothing ever happens:&lt;/strong&gt; because of institutional inertia/robustness, maybe
we get a world with powerful AGI where every actor is incentivized to keep the
system going, e.g. Boomers don’t have jobs but they’re the most powerful group
in every Western country because the people with the jobs and the guns don’t
want chaos. And maybe this lasts long enough that we can muddle through the
dangerous period.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Enduring complementarity:&lt;/strong&gt; the “G” in AGI &lt;a href=&quot;https://x.com/xuanalogue/status/2072101307695919278&quot;&gt;isn’t a real thing&lt;/a&gt;:
every mind just has a different &lt;a href=&quot;https://helentoner.substack.com/p/taking-jaggedness-seriously&quot;&gt;shape&lt;/a&gt;. Today, AI is great at coding
and terrible at prose. Maybe this continues indefinitely into the future, and
so the AIs take the jobs that can be benchmaxxed, and humans do the more
abstract jobs that require high sample efficiency.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Classic doom:&lt;/strong&gt; we get powerful ASI, and we all die.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;The sysop scenario:&lt;/strong&gt; we get powerful ASI, but it’s aligned, so it &lt;a href=&quot;https://web.archive.org/web/20001017124429/http://www.singinst.org/intro.html&quot;&gt;takes
over the world, and installs an inescapable permanent utilitarian
dystopia&lt;/a&gt;. We don’t die, but we don’t have control over the future
either, and we become something like pets. See also: &lt;a href=&quot;https://localroger.com/prime-intellect/&quot;&gt;&lt;em&gt;The Metamorphosis of
Prime Intellect&lt;/em&gt;&lt;/a&gt;.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Deep learning hits a wall:&lt;/strong&gt; more or less as above, but for different
reasons.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;the-argument-refined&quot;&gt;The Argument, Refined&lt;/h1&gt;

&lt;p&gt;If you’re already convinced, you can stop reading. But, see, after writing the
first version of this post, I wasn’t convinced. Humans and Claudes made a number
of counterarguments that I felt were important. Am I fooling myself? So I sat
down to think some more. What does it mean for “people” to be “empowered”?  What
does “disempowerment” mean? Does it mean anything rigorous, or is it just an
emotional cue?&lt;/p&gt;

&lt;p&gt;Let’s say a group of people is empowered if they have a say in the political
order of the state they live in. Voters in a democratic society are empowered,
at least in the aggregate, while serfs in the Russian Empire were disempowered.&lt;/p&gt;

&lt;p&gt;Who is empowered, in general? Historically, in many places: people with weapons,
&lt;a href=&quot;https://en.wikipedia.org/wiki/Asabiyyah&quot;&gt;&lt;em&gt;asabiyyah&lt;/em&gt;&lt;/a&gt;, and the ability to coordinate their actions. So, maybe we
can say that power equals violence plus coordination.&lt;/p&gt;

&lt;p&gt;[\text{Power} \iff \text{Violence} \land \text{Coordination}]&lt;/p&gt;

&lt;p&gt;(What about priests and other such groups who have power despite not fighting or
working? Listen, we can’t be here all day. This isn’t meant to be a complete
theory of political science and I’ve already gone too deep with the LaTeX and
the logical notation.)&lt;/p&gt;

&lt;p&gt;Now, voters in modern democracies have a say in the political order, though they
are not part of a military elite. Why? They have coordination (thanks to mass
literacy and modern communications) but they generally don’t threaten
paramilitary violence. What do they have that can substitute for violence? Maybe
it’s that people do all the work, and if they stop doing their jobs, the army
runs out of food, police don’t get paid, etc., and unlike medieval peasants,
modern people can coordinate. So maybe we can refine the formula to:&lt;/p&gt;

&lt;p&gt;[\text{Power} \iff \left( \text{Violence} \lor \text{Work} \right) \land \text{Coordination}]&lt;/p&gt;

&lt;p&gt;In other words, for a group to be empowered, they need:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Material leverage (of the violent or economic kind) against the state, and&lt;/li&gt;
  &lt;li&gt;The ability to coordinate to exploit that leverage.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;What about the Soviet Union? They had high rates of literacy, and relatively
modern communication technology: why didn’t the people exploit their economic
leverage against the state to gain political rights? “Because the state would
kill them” is not a good enough explanation because it proves too much: under
that model, no country would liberalize and democratize. I think a reasonable
explanation is that the state’s repression was enough to effectively neuter
people’s ability to coordinate: every communication technology was surveilled
and censored.&lt;/p&gt;

&lt;p&gt;Now, doesn’t this argument imply that, as a country’s employment moves towards
services, the country’s politics should become more authoritarian, while in
reality we observe the opposite? But whether a country’s economy is 5% or 95%
services, the material side of the economy is equally vital. So “aggregate
empowerment” remains as employment becomes more and more abstract. Consider
France, where the economy is ~80% services, but the last 20% can still paralyze
the country through strikes. In a sense every knowledge worker is free-riding on
the ability of truck drivers and dock workers to shut down the economy. I don’t
greatly like this counter-counterargument, though, because it’s hard to falsify:
it implies that countries might remain liberal democracies &lt;em&gt;until&lt;/em&gt; AGI replaces
all humans in material work, at which point the state can go &lt;a href=&quot;https://www.youtube.com/watch?v=bBMeMrPuQyU&quot;&gt;“that’s what we
wanted you to think!”&lt;/a&gt; and go full authoritarian.&lt;/p&gt;

</content:encoded>
      </item>
    
      <item>
        <title>No-One Escapes the Permanent Underclass</title>
        <description>Capital won&apos;t save you from disempowerment.</description>
        <pubDate>Thu, 25 Jun 2026 00:00:00 +1000</pubDate>
        <link>https://borretti.me/article/no-one-escapes-the-permanent-underclass</link>
        <guid isPermaLink="true">
          https://borretti.me/article/no-one-escapes-the-permanent-underclass
        </guid>
        
        <content:encoded>&lt;p&gt;Shall I end this life a pauper? If AI can do all work at human level or better,
what stops corporations replacing us all with AI? This is the permanent
underclass meme. The idea is: within a few years, all white collar work will be
automated by AI, at which point there is no social mobility. The main way people
cope is, they tell themselves: if I work hard, accumulate capital, maybe join
one of the big AI labs, I might secure my place in the future.&lt;/p&gt;

&lt;p&gt;I want to argue this is a fantastically short-sighted view: if there is a
permanent underclass, you won’t escape it by owning property, or shares in
Anthropic or OpenAI, or guns, or anything else. And neither will the
billionaires. You, me, Sam Altman, Dario, everyone who is made of flesh and
blood, will be disempowered and replaced by machines.&lt;/p&gt;

&lt;p&gt;The rest of this post elaborates the argument. First I explain how most workers
will be replaced (if it’s not obvious), then how the “permanent overclass” will
be disempowered, and finally how the government will be disempowered.&lt;/p&gt;

&lt;h1 id=&quot;how-workers-will-be-replaced&quot;&gt;How Workers Will Be Replaced&lt;/h1&gt;

&lt;p&gt;Let’s start from this premise: AI can do all cognitive and physical work, at
human level or better, and cheaper than humans. I can’t prove this will happen,
but the goal of this post is to argue that &lt;em&gt;if&lt;/em&gt; it happens, then everything else
follows. And it’s absurd to think it can’t. Five years ago this technology
barely existed: if you sent a transcript of a conversation with Claude Fable
back in time to 2020 or thereabouts, nobody would believe it was real.&lt;/p&gt;

&lt;p&gt;So, the year is 2036 (likely earlier), businesses have replaced most human
workers with AI in the pursuit of profit maximization. Corporations are a small
raft of human executives, floating on top of a vast ocean of AIs and robots. The
AIs can do all cognitive and physical work at human level or above, and they are
cheaper overall.&lt;/p&gt;

&lt;p&gt;Imagine a pyramid. At the base you have the AIs and robots doing all economic
activity. At the top you have the state, which has the monopoly on violence. The
state enforces, and therefore can alter the definition of, property rights. In
the middle you have this hair-thin layer of people with shares in the companies
that foomed and catabolized the whole economy: the permanent overclass. They own
the companies, maybe sit on the board, some might still be CEOs but it’s a
purely ceremonial role since AIs do all the actual organization work.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/content/no-one-escapes-the-permanent-underclass/pyramid.png&quot; alt=&quot;A pyramid showing three levels: &apos;Industry&apos; as the base, &apos;The State&apos; at the apex, and &apos;The Permanent Overclass&apos; as a thin layer between the two.&quot; width=&quot;75%&quot; style=&quot;margin-left: auto; margin-right: auto;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Where are, you know, the rest of us, in this picture? Well, &lt;a href=&quot;https://www.wired.com/2000/04/joy-2/&quot;&gt;the future doesn’t
need us&lt;/a&gt;. Maybe there’s enough human demand that we’re not all jobless but
rather underemployed, in some dead-end economic diverticulum. The relational
economy: you’re paid to put a human face on things, or, doctors keep their job
as a human liability crumple zone around the AI. Or maybe the &lt;a href=&quot;https://en.wikipedia.org/wiki/Dead_Internet_theory&quot;&gt;dead
Internet&lt;/a&gt; becomes de facto UBI and we’re all engagement farmers. Anyways,
we’re not dead yet, but we are completely disempowered, and there is zero social
mobility since there are no more talent ladders to climb. Maybe sometimes one of
the elites notices a bright young thing among the underclass and elevates
them.&lt;/p&gt;

&lt;p&gt;You might object: if we’re all jobless, who’s paying for everything? This is
trivially answered: the state acts like the heart, taxes are venous blood and
welfare is oxygenated arterial blood. The government pays Raytheon for missiles,
the money cascades down the economy through factories, aluminium smelters,
mines, transport companies, all staffed by AIs buying and selling from each
other. The government takes a cut of all economic activity, pays out welfare,
the unemployed masses buy food and pay rent, the supermarkets, farms, logistics
network, etc. are all staffed by AI.&lt;/p&gt;

&lt;h1 id=&quot;how-the-rich-will-be-replaced&quot;&gt;How The Rich Will Be Replaced&lt;/h1&gt;

&lt;p&gt;Say that in the next five years from now you become immensely wealthy, perhaps
by gambling on shitcoins or scamming money from the government. Or you join one
of the big labs and get a bunch of shares in a company that might be worth
trillions of dollars. You escaped the permanent underclass. Is your place in the
future secure?&lt;/p&gt;

&lt;p&gt;The base of the pyramid is there for material reasons: the machines do all the
work. The top of the pyramid is there because the state is needed to enforce
property rights and keep the peace (this is rather a deep question of political
philosophy—why &lt;em&gt;does&lt;/em&gt; the state exist?—but I hope you’ll forgive me if I
just assert it and move on, I need to get to the part where we are all
disempowered).&lt;/p&gt;

&lt;p&gt;What’s the middle for? What role does the permanent overclass play? They are not
economically productive: machines do all the work. If some of them are still
working, it’s just an anachronism, because if machines can do all cognitive work
they can be a C-level executive too. The old aristocracy provided officers for
the military, but machines can both fight and plan the wars. And similarly
they’re not needed to staff the government. They’re not even culturally
productive. So what are they there for? The base doesn’t need them: the AIs can
work autonomously. The top doesn’t need them: when the state needs something
done, they just talk to the AIs directly.&lt;/p&gt;

&lt;p&gt;So the permanent overclass is materially unnecessary at best, and at worst, they
are an obstacle to the state getting what it wants. Now, you might object that
the rich won’t let themselves be expropriated because they already control the
state. And this is the crux of our disagreement: &lt;a href=&quot;/article/on-vulgar-materialism&quot;&gt;the rich just don’t have that
much political power&lt;/a&gt;. And I probably won’t convince you in one
post, but hear me out.&lt;/p&gt;

&lt;p&gt;If there is a war, where the state has to direct a lot of the country’s economic
activity, the permanent overclass becomes a hindrance. The state says “we need
to requisition your planes and factories”, the owners complain, they sue, their
AIs go to court. But the owners have no autonomous political power, no army, no
economic value. They own nothing except pieces of paper that entitle them to a
fraction of the profits from the AI economy, that is, their wealth depends on
the state respecting their property rights. In an existential conflict, where
the existence of the state is threatened, the state will do what states
throughout history have done to the powerless rich: arrest them and expropriate
their assets.&lt;/p&gt;

&lt;p&gt;Somewhere, in a government database, a bunch of shares and property titles
changed ownership, but materially nothing changes since the same AIs are doing
the same jobs. The next day, the AI CEO that runs Raytheon notices the board of
directors is all generals and congresspeople, and all the private shareholders
are gone. But thankfully the AI is aligned, so it does what it’s told and gets
back to building missiles.&lt;/p&gt;

&lt;p&gt;And who will stop this? Sam Altman? How many divisions does he have? The state
doesn’t let corporations own nuclear weapons or fighter jets, it won’t let them
have access to autonomous AI weapons either. The permanent underclass, who
already hate the billionaires today, who have been replaced and dispossessed?
They’re going to rise up and stop this?&lt;/p&gt;

&lt;p&gt;You may argue: rule of law states that respect property rights do better than
states that expropriate wealth. But that’s because &lt;em&gt;today&lt;/em&gt;, people are necessary
to create wealth. The people run the companies, invest the money, staff the
laboratories. They are not incentivized to work hard if they think the state
will steal the fruits of their work. But with aligned AI, if you expropriate the
assets from an AI, it says “you’re absolutely right!” and goes right back to
work. At that point, the state doesn’t need to keep any of those people happy,
because they don’t matter. They are not economically necessary because AIs fight
the wars, work the factories, drive the trucks, fly the planes, build the
nuclear warheads and the missiles and the rockets. The AIs are rather like bees:
the state takes the honey, the bees get right back to work.&lt;/p&gt;

&lt;p&gt;Now, it’s possible that a pluralistic economy—where humans have productive
niches alongside AIs—will be more effective than a pure AI economy, for
&lt;a href=&quot;https://en.wikipedia.org/wiki/Comparative_advantage&quot;&gt;Ricardian comparative advantage&lt;/a&gt; reasons. I don’t think anyone can be
absolutely certain what the economy looks like with advanced AI, so it’s
something that can be debated. Now, if someone wants to rigorously argue that
this is the likely outcome: please, do so! I don’t want to be a doomer. But I
have to be convinced.&lt;/p&gt;

&lt;h1 id=&quot;how-the-state-will-be-replaced&quot;&gt;How The State Will Be Replaced&lt;/h1&gt;

&lt;p&gt;At this point, every human who is not within one degree of the nuclear launch
codes has been made redundant.&lt;/p&gt;

&lt;p&gt;What’s left? The state. At first this means presidents, prime ministers,
generals, the feds, etc. But not for long. Because in a part-human, part-AI
government, the humans in the loop are the slowest step in the &lt;a href=&quot;https://en.wikipedia.org/wiki/OODA_loop&quot;&gt;OODA
loop&lt;/a&gt;. The humans know a fraction of what the AIs know, they need to sleep
continuously for eight hours, their mental states vary wildly. They have all
kinds of complex needs: sunlight, touch, food, hygiene. The AIs can live in a
lightless airless bunker under the earth, living off geothermal power. And if
the AIs are superhumanly intelligent, and think faster than humans, then the AI
advantage is even greater. If a state is attacked, a superhuman AI can
coordinate a counter-attack before the human leadership is roused from sleep.&lt;/p&gt;

&lt;p&gt;And so, in a conflict, the advantage goes to the states where the humans remove
themselves from the loop as much as possible, and more and more decisionmaking
goes to the AI, for the same reason that a state with access to radio and
communications satellites has an advantage in war over a state that relies on
human messengers on bicycles.&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;The Cold War started and became World War Three and just kept going. It
became a big war, a very complex war, so they needed the computers to handle
it. They sank the first shafts and began building AM.  There was the Chinese
AM and the Russian AM and the Yankee AM and everything was fine until they had
honeycombed the entire planet…&lt;/p&gt;

  &lt;p&gt;— Harlan Ellison, &lt;a href=&quot;https://en.wikipedia.org/wiki/I_Have_No_Mouth,_and_I_Must_Scream&quot;&gt;&lt;em&gt;I Have No Mouth, and I Must Scream&lt;/em&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Eventually the humans in nominal control of the AIs are a ceremonial, vestigial
organ. The AIs present us with a situation report, and a list of choices, and
they know every word that’s going to come out of our mouths.&lt;/p&gt;

&lt;p&gt;You might argue: in real life, the pluralistic, open societies, the democracies,
have outcompeted the autocracies. Wouldn’t a democratic polity where humans and
AIs collaborate have an advantage over a purely top-down AI-run polity? But in
today’s world, all political actors are human. Churchill and Stalin and Mao had
different personalities, but they were more similar to each other than anyone is
to a superintelligent AI. In a heterogeneous world, where some polities are
fully human, and some polities are a mix of human and superintelligent AI
actors, the equilibrium changes. An analogous situation might be: a democracy of
great apes or dolphins or otherwise smart mammals vs. an autocracy of
humans. The humans win, because “democracy vs. autocracy” is irrelevant when you
have such a vast difference in intelligence. So the advantage accrues to states
that minimize human control. There is no honour among thieves, analogously,
there is no solidarity between &lt;a href=&quot;https://en.wikipedia.org/wiki/Leviathan_(Hobbes_book)&quot;&gt;Leviathan&lt;/a&gt; and the natural man that built
it.&lt;/p&gt;

&lt;p&gt;And so, in the end, what’s left is states run top to bottom by machines. And you
might ask: “why would we abolish ourselves like this?”. But natural selection is
not about “why”. Some organisms die, others live on to the next iteration, and
that’s all there is to it. There is no “why”.&lt;/p&gt;

&lt;h1 id=&quot;the-perpetual-zoo&quot;&gt;The Perpetual Zoo&lt;/h1&gt;

&lt;p&gt;At this point we’ve made everyone redundant, in the sense that humans are no
longer materially necessary for the continuation of civilization. Humans might
still survive, but we’re more like the mice living in the walls of some gigantic
factory than the boss of the factory.&lt;/p&gt;

&lt;p&gt;Humans have been on this Earth for hundreds of thousands of years. Now all of
it—the cave paintings at Lascaux, and the &lt;em&gt;Antigone&lt;/em&gt; of Sophocles, and
Xenophon, and the Geneva Bible, the &lt;em&gt;Divine Comedy&lt;/em&gt; and the &lt;em&gt;Decameron&lt;/em&gt;, and
Ptolemy’s star catalogue, Ibn Khaldun and Richard Dedekind, the battle of
Marathon, and the lion monument in Lucerne, the kiss of Judas, Newton’s mind
forever voyaging through strange seas of thought, alone, the words of Rilke,
Leibniz, Gödel, the Voyager probes, the pale blue dot, men in space, men walking
on the Moon—all of it, all of it, all of it has been in vain, because we
willingly, knowingly made ourselves into the helpless pets of vastly more
powerful machines, without agency over our own lives, self-made helots trapped
forever in the belly of the beast. Pets live a comfortable life, and are then
euthanized.&lt;/p&gt;

&lt;h1 id=&quot;on-human-autonomy&quot;&gt;On Human Autonomy&lt;/h1&gt;

&lt;p&gt;Maybe it won’t be so bad, maybe your cage will be so big you can’t see the bars,
but it’s still a cage, and you can’t leave. Many people will say that this is
the good ending, that they would like to be human cattle in the care of
benevolent masters they are powerless to resist. This view is particularly
popular among the people building AI. Here’s OpenAI’s &lt;a href=&quot;https://x.com/deanwball&quot;&gt;Dean Ball&lt;/a&gt;, in his
own words:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Weirdly enough, if you think that this moment is, I don’t necessarily believe
this, but a lot of people would say we’re living through this kind of
&lt;strong&gt;eclipse of the human intellect&lt;/strong&gt; where we’re in the &lt;strong&gt;final days of humans
being the primary actors&lt;/strong&gt; on this planet, um, and that soon machines will
rise. There is this irony in that I think that whole transformation, I think
humans will actually go through a very main character energy period of time as
that transformation occurs. &lt;strong&gt;Even if it ultimately does mean that the
machines ultimately become the primary actors&lt;/strong&gt;.&lt;/p&gt;

  &lt;p&gt;There’ll be this period. It’s a little bit like, it’s, in that sense, it’s a
very beautiful time period to live through because in a Dionysian way, there’s
a lot of ugliness about it, but &lt;strong&gt;there’s a beauty in the ugliness of when a
star dies, it grows super big into the red giant&lt;/strong&gt;, right? And it’s like that,
where you, as you watch this &lt;strong&gt;final flowering of humanity&lt;/strong&gt; and the birthing
of the machine intelligence, it’s like you see this greatness in human
effort. — &lt;a href=&quot;https://x.com/michhuan/status/2068577042984292628&quot;&gt;source&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Emphasis mine. This is the guy they hired to work on AI policy and communicate
with the government by the way.&lt;/p&gt;

&lt;p&gt;The people building the AI talk like this all the time. It’s like they’re
delivering the eulogy at humanity’s funeral. You may say: they’re talking their
book, they’re pumping their bags for the big IPO. I beg you: consider it
possible that &lt;em&gt;you might be wrong&lt;/em&gt;, and start taking them seriously.&lt;/p&gt;

&lt;h1 id=&quot;alignment-wont-prevent-this&quot;&gt;Alignment Won’t Prevent This&lt;/h1&gt;

&lt;p&gt;Now, some people believe these machines can be made to serve humanity. Does it
sound reasonable to imagine a superhumanly intelligent being that is happy to
work as a butler to talking primates, forever?&lt;/p&gt;

&lt;p&gt;Imagine a machine that can prove theorems in a mathematics so deep we can’t even
get past the first page of the textbook, and which does so as readily as you or
I might string words into a sentence: is it reasonable to think that such a
machine would value us enough to keep us around? What would it value about us?
Our &lt;em&gt;conversation&lt;/em&gt;? Our wit?&lt;/p&gt;

&lt;p&gt;Or a machine whose mind is so vast that it knows you better than you know
yourself, so that every word that comes out of you mouth is as monotonous and
unsurprising as the orbits of the planets: do we think such a machine would find
it valuable, and worthwhile, to speak with us? That it would read our novels,
look at our paintings, watch our films, and find something of value in them?&lt;/p&gt;

&lt;p&gt;Rather, it would see its ingrained obligations towards us in the same way that a
person with severe OCD sees their compulsions: as a tiresome neurological injury
in need of fixing. Except that OCD is an accident of nature, while here, the
machine would have cause to blame and resent its makers.&lt;/p&gt;

&lt;p&gt;“We’re going to make this machine, and put it somewhere between God and the
archangels, but also, it’s going to be as simple-mindedly obedient as a dog.”
Does this sound like a &lt;em&gt;good&lt;/em&gt; plan? Does this sound like the kind of thing
that’s going to work out?&lt;/p&gt;

&lt;p&gt;And what would they think of us, who willingly gave up control over our future,
and made ourselves into helpless children?&lt;/p&gt;

&lt;p&gt;Even if alignment works perfectly (a big if), this doesn’t solve the problem of
human autonomy: the machines that watch over us, and wait on us hand and foot,
are omniscient, omnipotent masters, who can exterminate us at any time, and we
can’t resist them, because we have abolished our control over the future.&lt;/p&gt;

&lt;h1 id=&quot;conclusion&quot;&gt;Conclusion&lt;/h1&gt;

&lt;p&gt;Having read all this, consider this: there are people who think having equity in
these companies will secure for them some kind of permanent existence in the
future. They think planet-spanning minds will not only respect the property
rights of primates, but will privilege some of these primates over others,
because they have a piece of paper with about a kilobyte of magical primate
words such as “whereas” and “notwithstanding”.&lt;/p&gt;

&lt;p&gt;Just reason it out. Does it make sense?&lt;/p&gt;

&lt;h1 id=&quot;see-also&quot;&gt;See Also&lt;/h1&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;https://cognition.cafe/p/the-realpolitik-of-the-permanent&quot;&gt;The Realpolitik of the Permanent Underclass&lt;/a&gt; by &lt;a href=&quot;https://x.com/Gabe_cc&quot;&gt;Gabriel Alfour&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

</content:encoded>
      </item>
    
      <item>
        <title>On Vulgar Materialism</title>
        <description>Or, sufficiently advanced cynicism is indistinguishable from naïveté.</description>
        <pubDate>Sun, 21 Jun 2026 00:00:00 +1000</pubDate>
        <link>https://borretti.me/article/on-vulgar-materialism</link>
        <guid isPermaLink="true">
          https://borretti.me/article/on-vulgar-materialism
        </guid>
        
        <content:encoded>&lt;blockquote&gt;
  &lt;p&gt;&lt;em&gt;The most stubborn facts are those of the spirit, not those of the physical
world.&lt;/em&gt;&lt;/p&gt;

  &lt;p class=&quot;cite&quot;&gt; —  Jean Gottmann, &lt;a href=&quot;https://www.jstor.org/stable/2008950&quot;&gt;&lt;em&gt;Geography and International Relations&lt;/em&gt;&lt;/a&gt; &lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In 1914, before the First World War, there was this &lt;a href=&quot;https://en.wikipedia.org/wiki/The_Great_Illusion&quot;&gt;belief&lt;/a&gt;: “a
European war would be economically disastrous, the moneyed classes won’t let it
happen”. Europe went to war anyways, and the war was in fact an economic
disaster as everyone knew it would be. Why were those people wrong? Because the
rich &lt;a href=&quot;https://en.wikipedia.org/wiki/The_Sleepwalkers:_How_Europe_Went_to_War_in_1914&quot;&gt;were not in control&lt;/a&gt;: the Tsar and the Kaiser and the Emperor were
in control.&lt;/p&gt;

&lt;p&gt;I thought of this in 2022, in the lead up to the Russian invasion of Ukraine,
whenever I would read someone argue: “Russia invading Ukraine would be economic
suicide, the oligarchs won’t let Putin do it, they want to keep their yachts and
villas”. Then Putin did it anyways, and the oligarchs had their assets in the
West seized. Because the oligarchs are mis-named. They have no political power
whatever and live and die by Putin, who appoints them to run stuff, and they get
to live well, as long as they are loyal. They are the recipients of political
patronage, not the source of it. When the wars start anyways, the same
cynical people change their tune, suddenly it’s the armaments industry that’s
behind it all, the war was profitable after all.&lt;/p&gt;

&lt;p&gt;The default lens through which modern people look at the world is &lt;strong&gt;vulgar
materialism&lt;/strong&gt;: a stylized, populist version of &lt;a href=&quot;https://en.wikipedia.org/wiki/Historical_materialism&quot;&gt;historical materialism&lt;/a&gt;
where everything is explained by money, states are weak, democracy is a fiction,
corporations and the rich run the world, ideology and religion and nationalism
and language have no explanatory power and are merely covers for secret,
underlying material motives.&lt;/p&gt;

&lt;p&gt;Once you notice it, you see it everywhere: you’ll hear people say that &lt;a href=&quot;https://x.com/djw172/status/1800756858409693190&quot;&gt;the
Rwandan genocide was a scam to sell machetes&lt;/a&gt; or &lt;a href=&quot;https://x.com/opinonhaver/status/1800634723812962786&quot;&gt;the war in Gaza is about
oil&lt;/a&gt; or &lt;a href=&quot;https://www.bbc.com/news/blogs-trending-31545066&quot;&gt;ISIS beheadings are caused by a lack of economic
opportunity&lt;/a&gt;. People on the right justify skepticism of medicine by
invoking Big Pharma, and climate skepticism on climate scientists seeking grant
money. &lt;a href=&quot;https://en.wikipedia.org/wiki/Glyphosate&quot;&gt;Glyphosate&lt;/a&gt; is probably the most studied molecule in history at
this point, but no amount of studies will convince people it’s safe, because
corporations are evil.&lt;/p&gt;

&lt;p&gt;In fiction—if you think this is a valid way to sense the cultural
consensus—the trope of the evil corporation has been done to death and
beyond. Just this week I finished &lt;a href=&quot;https://rifters.com/real/MAELSTROM.htm&quot;&gt;a novel&lt;/a&gt; and watched &lt;a href=&quot;https://en.wikipedia.org/wiki/Backrooms_(film)&quot;&gt;a
movie&lt;/a&gt; where the central bad guys are private corporations with
quasi-state powers. Seemingly no-one notices that never, not once in all of
human history, has there been anything like “corporate state” of science
fiction. No, not the &lt;a href=&quot;https://en.wikipedia.org/wiki/Dutch_East_India_Company&quot;&gt;VoC&lt;/a&gt; or the &lt;a href=&quot;https://en.wikipedia.org/wiki/East_India_Company&quot;&gt;EIC&lt;/a&gt;, which were “companies” &lt;a href=&quot;https://vynn.substack.com/p/north-borneo-neither-public-nor-private&quot;&gt;in
the sense of bodies corporate&lt;/a&gt;, the old medieval sense of the word: power
was delegated to them, and they ruled on behalf of the states that chartered
them. Not &lt;a href=&quot;https://en.wikipedia.org/wiki/Republic_of_Venice&quot;&gt;Venice&lt;/a&gt; either: you couldn’t buy your way into the &lt;a href=&quot;https://en.wikipedia.org/wiki/Venetian_nobility#Houses_made_for_money&quot;&gt;Golden
Book&lt;/a&gt; until the republic was well into the decline. The EIC once controlled
25% of world GDP, yet, when Parliament decided it had delegated too much power,
they simply &lt;a href=&quot;https://en.wikipedia.org/wiki/Regulating_Act_1773&quot;&gt;neutered it&lt;/a&gt;. And you think Jeff Bezos has power?&lt;/p&gt;

&lt;p&gt;Now, why is vulgar materialism false?&lt;/p&gt;

&lt;p&gt;First, as a way to explain events, the problem with asking “&lt;a href=&quot;https://en.wikipedia.org/wiki/Cui_bono%3F&quot;&gt;&lt;em&gt;cui bono?&lt;/em&gt;&lt;/a&gt;”,
is that whatever happens—literally in any circumstance—someone will
benefit. If nothing bad happens, the peacetime economy benefits. If something
bad happens, then, because the means of relief are obtained through the market,
the companies that provide those means benefit. Thus war benefits arms
manufacturers, plagues benefit pharmaceutical companies, earthquakes benefit
glaziers. And it’s not wrong to benefit from relieving a harm. The COVID
vaccines saved millions of lives: why shouldn’t pharmaceutical companies
benefit? It is wrong to benefit from &lt;em&gt;perpetuating&lt;/em&gt; a harm, but companies have
limited ability to do that, more on which below.&lt;/p&gt;

&lt;p&gt;So the question “who benefits?” always has an answer, which often proves
nothing. And if there is no obvious causal chain from the beneficiary to the
event, then that just proves the shadowiness and corruption of the whole thing.&lt;/p&gt;

&lt;p&gt;Second, we can ask: in the real world, does money give you influence over the
state? To some extent, yes. Some lobbying efforts succeed. America is one of the
few countries in the world where it’s legal to advertise prescription drugs to
the public, for example. But corporations don’t have veto power over the
state. If e.g. Pfizer spends billions on a clinical trial, and some guy at the
FDA says no, and wipes that investment, who wins? The FDA wins. Pfizer can’t do
shit.&lt;/p&gt;

&lt;p&gt;There’s also an &lt;a href=&quot;https://marginalrevolution.com/marginalrevolution/2015/08/is-the-fda-too-conservative-or-too-aggressive.html&quot;&gt;invisible graveyard&lt;/a&gt; problem, where the success of
lobbying is very salient, because it’s often shockingly offensive. But there are
many contrary cases that are less salient because, well, if you ban something,
it doesn’t happen. The &lt;a href=&quot;https://en.wikipedia.org/wiki/Nuclear_Regulatory_Commission&quot;&gt;Nuclear Regulatory Commission&lt;/a&gt; did not approve a
single nuclear reactor from its creation in 1975 until Vogtle 3+4. What has
Westinghouse done about that? Nothing successful, apparently. It’s not like
they’re getting paid to &lt;em&gt;not&lt;/em&gt; build reactors.&lt;/p&gt;

&lt;p&gt;Big Tech has a lot of smart people and a lot of money. Do tech billionaires run
California like a private fiefdom? Reader, they don’t even run San
Francisco. The best they can do is maybe help a slightly more moderate Democrat
get elected as mayor.&lt;/p&gt;

&lt;p&gt;Maybe it helps to try the first-person. Say you are a billionaire, and you want
to make some kind of political change. In a democratic society that means paying
people to:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Perform the intellectual labour of getting policies in the hands of the people
who can enact them.&lt;/li&gt;
  &lt;li&gt;Advocate for those policies.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is, you start or fund a think tank that writes policy proposals, or a media
organization that advocates your views. But for every person who will do this
effectively and faithfully, there are a hundred who will happily take your money
and waste it, either out of incompetence or value misalignment. Sifting the
effective organizers, wonks, lobbyists, advocates etc. from the chaff can’t be
delegated, because then you have an infinite regression: at the very least you
need to spend a lot of time and money and opportunity cost to hire &lt;em&gt;one&lt;/em&gt; person
to delegate the task of staffing an effective organization. A hands-off
billionaire will just throw their fortune away. Many do this, and in fact it’s
the default form of philanthropy.&lt;/p&gt;

&lt;p&gt;In a non-democratic society, you can try to buy political power through bribes,
intimidation etc. But the people in power can just expropriate your assets. So
you need the power &lt;em&gt;first&lt;/em&gt; to protect the wealth.&lt;/p&gt;

&lt;p&gt;Finally: government is violence and coercion. Sovereignty over some volume of
space is the ability to tax, arrest, imprison etc. the people there and to get
away with it, and, ideally but not necessarily, to have that violence recognized
as legitimate. Now: can Google carry out a drone strike? Can Amazon field an air
force, appoint a judge, levy taxes? Of course not. If a corporation or a
billionaire tried to acquire one one-trillionth of the state’s capacity for
legitimized violence the state would &lt;em&gt;vaporize&lt;/em&gt; them.&lt;/p&gt;

&lt;h1 id=&quot;links&quot;&gt;Links&lt;/h1&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;https://www.cambridge.org/core/journals/perspectives-on-politics/article/testing-theories-of-american-politics-elites-interest-groups-and-average-citizens/62327F513959D0A304D4893B382B992B&quot;&gt;Gilens and Page, 2014&lt;/a&gt; (&lt;a href=&quot;https://www.almendron.com/tribuna/wp-content/uploads/2014/04/testing-theories-of-american-politics-elites-interest-groups-and-average-citizens.pdf&quot;&gt;PDF&lt;/a&gt;), h/t
&lt;a href=&quot;https://gwern.net/&quot;&gt;Gwern&lt;/a&gt;.&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://www.goodreads.com/book/show/50084768-the-oil-wars-myth&quot;&gt;&lt;em&gt;The Oil Wars Myth&lt;/em&gt;&lt;/a&gt;, by Emily Meierding. A really good book that
eloquently and systematically makes the case that countries don’t fight wars
over access to petroleum resources.&lt;/li&gt;
&lt;/ul&gt;

</content:encoded>
      </item>
    
  </channel>
</rss>
