There’s a documentary called The Most Unknown. Its structure is the best thing about it: nine scientists, nine different fields — a microbiologist, a physicist, an astrobiologist, a neuroscientist, a cognitive psychologist, and on — arranged as a relay. The first scientist visits the second in her lab, learns what she’s working on, and then she leaves to visit the third, who leaves to visit the fourth. A daisy-chain of experts, each one dropped without warning into a field that isn’t theirs.
You’d expect a film like that to be about how much we know. It’s the opposite. What the scientists keep circling, over and over, is how much they don’t — what one reviewer called their shared “awareness of the boundlessness that confronts them.” These are people at the literal frontier of human knowledge, and the thing they have in common isn’t mastery. It’s a vivid, almost vertiginous sense of how much is still dark.
I’ve been chewing on that feeling for years, and I’ve started calling it the abyss.
What it is
Here’s the idea. In any sufficiently complex field, the space of things you could explore is so vast that it can’t actually be navigated — not by anyone, not in a lifetime. And here’s the strange part, the part that makes it worth a whole essay: you have to become genuinely competent in the field before you can even see that the abyss is there. To a beginner, a field looks finite, even small — a stack of textbooks, a set of rules, a thing you could imagine finishing. The deeper you go, the bigger it gets. By the time you’re an expert — by the time most people would point at you and say “she’s an expert” — your own view is mostly of the edge: the enormous, growing rim of things you now understand just well enough to know you will never get to.
Come up for air with something ordinary. A beginner cook learns thirty recipes and feels, reasonably, like cooking is a finite skill — learn a few hundred dishes and you’d be “good at cooking.” A chef sees something else entirely: an unbounded space of techniques, ingredients, chemistries, cuisines, where every answer opens ten new questions and no one alive has tasted a fraction of it. Same kitchen, two completely different rooms. The beginner thinks the room has walls. The chef has walked far enough to know there aren’t any.
That gap — between the room the beginner thinks they’re in and the one the expert knows they’re in — is the abyss. And almost everything interesting about it comes from the fact that you can’t see it from the door.
Why only experts can see it
Why should you have to be good at something to see how impossibly big it is? It sounds backwards. It isn’t.
Two things grow as you learn, and they grow at different rates. Your knowledge grows — slowly, with effort, with diminishing returns. But your perception of how much there is to know grows faster, because every new thing you understand reveals adjacent things you previously couldn’t even have asked about. The physicist John Wheeler put it the way I’ve never been able to improve on: “We live on an island surrounded by a sea of ignorance. As our island of knowledge grows, so does the shore of our ignorance.”
Sit with the geometry of that for a second, because it’s the whole thing. A bigger island has a longer coastline. The beginner stands in the middle of a tiny island and sees only land in every direction — the field looks like solid, finite ground. The expert has walked all the way to the shore, many shores, and spends most of their time staring at the sea. Marcelo Gleiser built an entire book, The Island of Knowledge, on exactly this image, and his one-line version is worth keeping: “Learning more about the world doesn’t lead to a point closer to a final destination but to more questions and mysteries.”
So there are really three positions, not two. There’s the novice, on the tiny island, who feels appropriately small — they know they know almost nothing. There’s the expert, at the shore, who also feels small, but accurately, because they’ve seen the sea. And in between there’s the most dangerous spot on the whole curve: the person who has learned enough to feel like the room is conquerable but hasn’t yet walked to a single shore. They feel big. They’re the one who’ll tell you the field is basically figured out.
This looks like the Dunning-Kruger effect, and it’s worth being precise about how it’s both like it and not. The real finding behind that famous name is narrower than the meme: people who are bad at something tend to overrate themselves, because the very skills you’d need to do the thing are the skills you’d need to know you’re bad at it. (The “Mount Stupid” curve everyone shares isn’t actually in Dunning and Kruger’s paper — it’s an internet embellishment.) Dunning himself stated the underlying mechanism in a way that is the abyss seen from the other side: “to know that you don’t know something, you need to know something.” The abyss is that sentence run all the way up the ladder. The more you know, the more accurately you can perceive the shape of what you don’t. Expertise isn’t the cure for feeling ignorant — it’s the prerequisite for feeling ignorant accurately. It’s the machinery that converts Rumsfeld’s “unknown unknowns” into known ones: the expert can finally see the questions they couldn’t even have formed before.
It isn’t just a feeling
I want to head off an objection, because it’s the one I’d raise. Maybe this is just a mood — humility, impostor syndrome, the romance of science dressed up as a thesis. It isn’t only that. For a lot of fields the abyss is a measurable, structural fact, and the people who’ve measured it most precisely are complexity theorists and computer scientists.
Take chess. Not a mystical field — a board, sixteen pieces a side, fixed rules a child can learn in an afternoon. In 1950 Claude Shannon estimated the number of possible chess games at around 10^120 — a 1 with a hundred and twenty zeros after it. For scale: there are something like 10^80 atoms in the observable universe. The game of chess contains, by a margin so vast the comparison stops meaning anything, more distinct games than the universe has atoms. Go is worse — commonly cited estimates put its game tree somewhere around 10^360. These aren’t infinities. They’re finite, fully specified by a few simple rules, completely knowable in principle. And they are still, in any practical sense, unnavigable. No player and no computer “solves” chess by walking the tree. They can’t. The space is too big, and the rules that generate it are trivial. Simplicity at the bottom is completely consistent with an abyss on top. That’s the part people miss.
Borges saw this before the complexity theorists had the numbers for it. His “Library of Babel” is the total library — every possible book, every combination of letters, which means it contains the true history of the universe and also every subtly false version of it, and the catalog of the library and every false catalog too. Everything true is in there somewhere. And, Borges notes, the chance of any person actually finding their one vindicating book “can be computed as zero.” The answers exist. The space is just too large for the answers to be findable. That’s the abyss, written as literature forty years early.
Herbert Simon — who more or less founded the study of how real minds cope with all this — put the human side of it bluntly back in 1957: “The capacity of the human mind for formulating and solving complex problems is very small compared with the size of the problems whose solution is required for objectively rational behavior in the real world.” His word for what we do instead of solving was satisficing: we don’t find the best move, we find one that’s good enough, because the space of all moves cannot be searched. Bounded rationality, in his framing, isn’t a bug in human reasoning. It’s the only possible response to an abyss.
And the abyss doesn’t even hold still. Stuart Kauffman has a lovely term — the adjacent possible — for the set of things reachable in a single step from where you are now. The catch is that every time you actually step into it, the adjacent possible gets bigger: each new thing you understand opens doors to things you couldn’t previously have reached or even named. The frontier recedes as you advance on it. That’s Wheeler’s shoreline lengthening, stated as a mechanism rather than a metaphor.
There’s even a hard floor under this, for the cases where you might hope a clever shortcut saves you. Stephen Wolfram calls it computational irreducibility: for sufficiently rich systems there is no formula that lets you skip to the answer — the only way to find out what the system does is to run it, step by step, with no compression available. Where a system is computationally irreducible, the abyss isn’t a temporary gap that smarter people or faster machines will close. It’s permanent. You cannot fold the territory down into a map small enough to skip walking it.
Put all of that together and you get something I find genuinely useful: a clean test for what “sufficiently complex” actually means, a phrase I’d been leaning on without defining. A field has an abyss exactly when its possibility space outgrows any traverser’s budget — when the space of moves grows faster than anyone, human or machine, could ever search it. Tic-tac-toe has no abyss; its whole game tree fits in a child’s head and the game is genuinely, finishably solved. Chess, Go, medicine, law, mathematics, every real science, Brazilian Jiu Jitsu — these are abyssal, in the precise sense that the search can never catch the space. The mathematician Marcus du Sautoy, mapping the edges of his own field in What We Cannot Know, even separates the known unknowns from a stranger category he calls the known unknowables — things we can prove we will never know. The most precise discipline we have turns out to contain an abyss it can prove is permanent.
How experts actually cope: compression
So how does anyone function inside an abyss? Not by mapping it — that’s impossible by construction. By compressing it.
This is the part that ties the abyss to almost everything else I think about. An expert is not someone who has searched more of the space than you have. An expert is someone who has built a radically better compression of it — a private library of heuristics, patterns, instincts, and tastes that lets them skip almost the entire space and land near the good parts without searching. Cognitive scientists call the building block chunking. In a piece I wrote a while back I used chess for it: to a beginner, a board is thirty-two pieces under simple rules; to a grandmaster it’s “an intricate landscape of strategic tensions, positional leverage, and potential futures that unfold 15-20 moves ahead.” The grandmaster isn’t computing more positions than the beginner. She’s seeing the board already compressed into meaningful units, which is exactly what lets her ignore 99.999% of that 10^120 tree and feel her way to a strong move in seconds.
That reframes what expertise even is. Expertise is a compression of the abyss. Taste, intuition, “that smells wrong,” the senior engineer who can’t tell you why the architecture is bad but turns out to be right — none of this is mystical. It’s a lossy map of a territory too large to traverse, built up over years of walking small parts of it on foot. And because these maps are compressions of an un-navigable space, they come with two properties that reliably frustrate everyone involved: they’re hard to put into words (the map was never made of words), and they’re hard to hand to someone else (your compression is built on the specific ground you walked). The map is not the territory — and in an abyssal field, the map is the only thing anyone has. There is no territory-sized map. There can’t be.
The abyss and humility
Here’s the part I think actually matters, and why this isn’t a counsel of despair. The people who have truly seen the abyss are, in my experience, the most intellectually trustworthy people you’ll meet — and it’s because they’ve seen it.
There’s a failure mode of expertise that is the exact opposite. Deep knowledge, run without ever walking to the shore, curdles into something dangerous: the specialist who mistakes the edge of their own map for the edge of the world. I’ve written about this as the curse of expertise — paradigm lock-in, overconfidence the moment you reach into a neighboring field, an inability to even see evidence that doesn’t fit the chunks you already have. That’s depth without the abyss. The very same depth, in someone who has seen the abyss, produces the reverse: humility, curiosity, a standing assumption that the next field over is at least as deep as your own and that you are a rank beginner standing in it.
This is an ancient idea wearing modern clothes. Nicholas of Cusa, in 1440, called the highest form of knowledge docta ignorantia — “learned ignorance,” the wisdom of knowing precisely the extent of what you cannot know. Socrates, in Plato’s telling, claimed no special knowledge at all — only that, unlike the confident men around him, “what I do not know I do not think I know.” James Clerk Maxwell, one of the greatest physicists who ever lived, said that “thoroughly conscious ignorance is the prelude to every real advance in science.” The neuroscientist Stuart Firestein built a whole book, and a popular course, on the argument that science doesn’t actually run on knowledge — it runs on high-quality ignorance, on knowing the precise shape of the next thing you don’t yet know. Every one of these is a description of the same person: someone standing at the shore, looking at the sea, and reporting back accurately about its size.
That’s the posture the documentary captures, and it’s why nine frontier scientists can spend ninety minutes mostly talking about what they don’t know and come off as exhilarated rather than defeated. To see the abyss is not to be beaten by it. It’s the thing that keeps you honest.
Where I land
Two larger things this connects to, which is most of why I wanted to write it down.
The first is the book I’m working on, Lossy, which is about how ideas shed information as they travel from where they’re made out to the rest of us. One of its chapters works through a trade-off I call optionality vs. access: the more options — precision, power, expressive range — a tool carries, the harder it is for any given person to approach, and vice versa. matplotlib gives you everything and is brutal to learn; seaborn makes most of the choices for you and quietly takes the steering wheel away. It would be easy to read that curve as the whole story I’m telling here, so let me be clear that it isn’t. The thing that sits on that curve is an artifact — a finite, designed, masterable thing. matplotlib is enormous, but it has a top, and there are people who have genuinely reached it and command the entire library. A software library is not an abyss. The abyss is the domain the tool points into — the unbounded space of everything you could ever do with data and a picture — and nobody commands that. The two are easy to run together, because both involve “a lot of options.” But they differ in the way that matters: a tool’s optionality is a learning curve with a summit, while the abyss is a horizon with none. The cleaner way to hold them is the one this essay has been circling all along — the artifacts are maps, the abyss is the territory, and optionality-vs-access is the trade-off you’re making when you draw a map. The abyss is the reason you have to draw one at all.
The second is AI, which makes the whole question urgent rather than merely interesting. An abyss is by definition a space too large for any human budget to search, and we have now built machines that can hold and search vastly more of these spaces than we can. The obvious hope is that they are the first thing in history actually capable of navigating an abyss on our behalf. Maybe. But there’s a darker reading, and it falls straight out of the three-positions picture above. The dangerous spot on the curve was never the novice, who knows they know nothing. It was the confident middle — fluent enough to produce convincing answers, not deep enough to have seen any shore. A system that compresses an abyss it cannot perceive the edges of, and hands you that compression with total fluency and no sense of what it dropped, is the confident middle built at planetary scale. Whether AI becomes the first real navigator of the abyss or the most persuasive thing that ever mistook its map for the territory turns, I think, on a single question: whether these systems can be built to see the shore — to know the extent of their own ignorance — which so far they plainly cannot.
The film is built entirely out of hand-offs: one scientist, then the next, each walking into a field they don’t command, to be amazed again. That’s the posture the abyss asks for. Not mastery, which was never on offer. Just the willingness to keep walking toward the shore, and to report back honestly about the size of the sea.
Where I’m still uncertain
- Maybe the abyss is just the mirror image of the curse of expertise and doesn’t earn its own name. I don’t think so — the curse is about depth failing without flexibility, while the abyss is about the structure of the space that the flexibility is responding to — but I hold the distinction loosely.
- The Nietzsche line everyone reaches for here — “when you gaze long into an abyss, the abyss gazes also into you” — is about a different abyss: the moral and existential one, not the epistemic one. I’ve borrowed his word, not his meaning, and I’d rather say so plainly than let the borrowed gravity do unearned work.
- “Possibility space outgrows the traverser” is crisp for chess and Go, where I can actually count. It’s hand-wavier for history, or ethics, or design — fields whose space isn’t formally enumerable. I think the abyss is just as real there, but I can’t put a Shannon number on it, and I don’t want to smuggle in false precision by pretending I can.
- The hopeful branch of the AI argument leans hard on the word “see.” I don’t actually know what it would mean, in technical terms, for a model to perceive the shore of its own ignorance — and the whole optimistic case depends on that being a buildable thing.
Written with the assistance of Claude.