Writing
The Knowledge Boundary Illusion: Why Deltas Get Harder to See
When a model is worse than me at something I know well, I can see it instantly. The mistake jumps out, I correct it, and I walk away feeling like I have a good read on what these systems can and can't do.
The trouble starts when the model gets close to the edge of what I know. Past that line, its errors stop looking like errors. They look like nuance. Sometimes they look like expertise I don't have. I can't tell the difference between "this answer is subtly wrong" and "this answer is more sophisticated than my understanding," because telling those apart requires exactly the knowledge I'm missing.
Fluency is not evidence
I think of this as the knowledge boundary illusion: the feeling that I can still grade the model because I can understand its words. Reading comprehension and evaluation ability feel like the same skill, but they aren't. The prose is equally fluent everywhere, including the places where the claims are wrong. So my confidence in the output stays roughly constant while my actual ability to check it falls off a cliff.
There's a perverse shape to this. The better models get, the more of their output sits beyond my boundary, which means the deltas I most need to see are precisely the ones I'm least equipped to see. My intuition becomes least reliable at exactly the point where the answers become most persuasive.
I run into this at work, where part of my job is deciding how much to trust model judgments at scale. When a model classifies something I understand deeply, I can audit it directly. When it operates somewhere I'm shallow — a language I don't speak, a community's slang I've never encountered — I have to admit that my spot-checks are closer to vibes than verification.
Grade the process, not the answer
The practical move I've landed on is to shift from evaluating answers to evaluating process. Near my boundary I can't reliably judge whether a conclusion is right, but I can still ask: what is this based on? What assumptions were made? What would falsify it? Which claims would an actual expert need to check? The model can help generate that map. I still have to decide where the map is trustworthy enough to act on.
One tell I've learned to watch for: when my critique of an output gets vague — "something feels off" rather than "this specific claim is wrong" — that often means the model has moved past my easy evaluation range, not that the output is bad. Vague unease near the boundary is information about me, not about the model.
For anything with real consequences — money, safety, legal exposure, production systems — I treat my own judgment as one weak signal and go find stronger ones: primary sources, domain experts, actual tests. That sounds obvious written down. The illusion is that it rarely feels necessary in the moment, because the answer reads so well.
The hardest gap to measure is the one sitting just past your own knowledge boundary. Which is inconvenient, because as models improve, that's where more and more of the interesting gaps will live.