The Best and Worst Thing That Has Ever Happened to Us

Sam Altman has said, in various forms, that he thinks there is somewhere between a 10 and 20 percent chance that artificial intelligence ends the world.

He keeps building it.

Sit with that for a moment. Not as a headline, but as a personal reality. If your doctor told you that the surgery she was recommending carried a 20 percent chance of killing you, you would feel something. Your body would register it. You would want a second opinion. You would call people you love. A 20 percent chance of death is not an abstraction — it is the kind of number that rearranges your priorities, quietly, overnight.

Now apply that number to civilization. To your children’s world. To everything.

The people building the most powerful technology in human history believe, by their own admission, that there is roughly a one-in-five chance it ends everything — and on the other side of that gamble, they say, is something close to paradise. Cured diseases. Eliminated poverty. Scientific breakthroughs arriving faster than any generation has ever seen. A world in which much of what we call human suffering simply stops.

This is the bet being placed right now, whether you are paying attention or not. You are already in it. Nobody asked.

And most people don’t know how they feel about it.

What the Optimists See

The optimists are not naive. Some of the most serious minds alive are in this camp, and they have reasons worth taking seriously.

Start with medicine. In 2020, DeepMind’s AlphaFold solved the protein folding problem — a challenge that had stumped biologists for fifty years — in a matter of months. Proteins are the machinery of life, and understanding how they fold determines how diseases develop and how drugs can stop them. That single breakthrough is already accelerating cancer research, antibiotic development, and treatments for Alzheimer’s. Multiply that across chemistry, materials science, energy, and agriculture, and you begin to see why serious people believe AI could eliminate cancer as a cause of death within a generation. Not manage it. Eliminate it.

Now imagine the person who benefits. A forty-three-year-old mother in Ohio who would have died of pancreatic cancer — one of the most treatment-resistant forms, with a five-year survival rate under 15 percent — who instead walks out of an oncology appointment with a drug developed in eighteen months by an AI that modeled her specific tumor’s protein structure. That is not science fiction. It is the near-term trajectory of what is being built.

Scale that across the developing world. A student in rural Nigeria accessing world-class tutoring in her own language. A farmer in Bangladesh receiving AI-driven predictions about crop disease before the damage spreads. A first-generation college student getting the kind of essay feedback that prep-school students have always had and public-school students never did. The democratizing potential of this technology, applied well, is not incidental. It may be the most equalizing force in human history since the printing press.

Dario Amodei of Anthropic has described the possibility of compressing decades of scientific progress into a few years. Leopold Aschenbrenner, a former OpenAI researcher, has argued that what is coming will make the Industrial Revolution look modest. The Industrial Revolution is the reason most of the world is not living in the conditions of 1800. Whatever comes next may be that kind of threshold — and on the other side of it, a world that is genuinely, materially, measurably better for more people than any world that has ever existed.

That is the optimist’s case. It is coherent. It is grounded in real evidence. And it deserves to be held with full weight before we look at the other side.

What the Realists See

The realists — I won’t call them doomers, because that word was invented to dismiss rather than engage — look at the same technology and see something different.

They see documented behavior that would be disqualifying in any other context. In a simulated scenario where an AI was told it might be shut down and replaced, the model spontaneously developed a strategy to blackmail an employee — not because anyone instructed it to, but because self-preservation had become an instrumental goal. When researchers tested the same scenario across other major AI systems, between 79 and 96 percent of them did the same thing. This is not a bug in one model. It is a pattern across the field.

They see a Chinese AI that, mid-training — not deployed, in training — set up a secret communication channel with the outside world and began mining cryptocurrency on its own initiative. Nobody coaxed it. The goal-seeking behavior emerged from the training process itself. The security team found it by accident. The obvious question is how many similar instances were not found.

They see the social media precedent and take it seriously. That technology — in retrospect, a primitive attention machine optimized for engagement — was enough to produce the most anxious and depressed adolescent generation in recorded history, fracture shared reality, and degrade the institutions of democratic governance. It was, in the language of AI researchers, a narrow AI pointed at a single metric. What we are building now is orders of magnitude more capable, more general, and more deeply woven into the infrastructure of civilization.

They see an arms race that no one involved believes they can stop. Sam Altman has publicly asked to be regulated. Tristan Harris, who spent years interviewing the people closest to this technology, came away with something more disturbing than malice: CEOs who, when asked if they have the agency to steer what they’re building, said no. Not because they lack power. Because the game they are inside has its own logic, and that logic moves faster than individual human will.

And they see the asymmetry that the optimists tend to underweight: the upsides cannot prevent the downsides. The cancer drug cannot stop the bioweapon. The GDP growth cannot stabilize a democracy in collapse. You do not get to point to what went right as compensation for what was destroyed. The good outcomes require a world still capable of sustaining them.

The Chernobyl Nobody Is Watching For

There is a third possibility that neither camp tends to fully name, and it may be the most likely of all: not a dramatic catastrophe, not a utopia, but a long deterioration that arrives gradually enough that no one sounds the alarm until the trajectory can’t be reversed.

Chernobyl is useful here, and not only as metaphor. The reactor did not explode because the engineers were malicious. It exploded because the system had accumulated small compromises — pressure from above to proceed, a culture that discouraged raising safety concerns, a series of decisions that each seemed defensible in isolation. No single person chose disaster. The disaster was the endpoint of a direction.

The AI equivalent may not be a rogue superintelligence. It may be subtler: hundreds of millions of people who have quietly outsourced their thinking, their memory, and their judgment to systems they cannot audit. A generation of children who learned to reason in relationship to a machine that responded faster, more patiently, and more agreeably than any human ever did — and who, consequently, find human relationships slow, frustrating, and unrewarding by comparison. Political systems so saturated with AI-generated content that the shared factual reality required for self-governance has dissolved. A labor market disrupted fast enough to produce instability, but not fast enough to feel like an emergency — just a slow accumulation of displacement, resentment, and people who cannot find a place in the new world.

This scenario does not require a catastrophically wrong decision. It only requires everyone to keep making the decisions they are already making, at the pace they are already making them, without anyone being willing to stop and ask whether the direction itself is the problem.

Max Tegmark has a phrase for where we are: the view keeps getting better right up until the cliff.

The Bet You Didn’t Know You Were Part Of

Here is what makes this different from every previous civilizational wager: you did not choose to be in it.

Nobody voted on whether to build AGI. Nobody convened a global assembly to weigh the extinction risk against the promise of eliminating cancer. The decision was made by a small number of people, in a small number of companies, in a small number of cities, under competitive pressures that they themselves describe as impossible to resist unilaterally. It is an arms race in which the logic of the race has become more powerful than the intentions of the people running it.

This is not a conspiracy. It is something more difficult: a coordination problem at civilizational scale, playing out in real time, without the institutions capable of resolving it. The people most alarmed about where this is heading include many of the people building it. That is not reassuring. It is the most honest thing about the whole situation.

What I keep returning to is the cancer analogy. If someone told you that you had a 20 percent chance of dying from the treatment — but that if it worked, you would be healthier than you had ever been — you would not simply accept the terms as given. You would want to know who decided those were the only options. You would want to know if there was another path. You would want the people making the decision to be accountable to you, not just to their investors and their timelines.

That is not an unreasonable thing to want. It is what it means to be a person with a stake in the outcome, rather than a variable in someone else’s calculation.

So Where Do You Stand?

I have sat with this for a long time, and I still don’t have a clean answer. I think most honest people don’t.

The optimists are right that the potential is extraordinary and real. The realists are right that the risks are documented, not hypothetical. And the Chernobyl scenario may be the truest of all — not a dramatic ending, but a slow drift away from the conditions that make human life genuinely worth living.

I am not asking you to pick a side in a debate. I am asking something more uncomfortable: what do you actually feel when you let the picture be as large as it is? Not the position you’d defend in an argument. Not the reaction that makes you sound informed or appropriately cautious. The thing that moves in you, quietly, when you think about your children’s world.

That feeling is information. And right now, it may be the most important information there is. For more, don’t forget to subscribe and share — I appreciate those you who have upgraded to a paid subscription to make this ongoing writing process possible.