The Black Hole

There's a story that everyone in AI has heard, and most people have started to believe.

It goes like this. Intelligence emerges from scale. Train a large enough neural network on a large enough dataset, and capabilities you didn't explicitly program for start showing up on their own. Reasoning. Planning. Translation. Code. The pattern of the last seven years has been remarkably consistent: bigger model, more data, better behaviour. The question isn't whether scaling works. It's how far it goes. And the dominant answer in the field right now is: very far, possibly all the way.

I've spent the last few years building production AI systems for a living. I work at the frontier daily. I have watched, up close, the things scaling has produced — and they are genuinely extraordinary. I am not writing this to argue against scaling. The history of the last decade has been the history of bets against scaling losing money.

I am writing this because there is a class of problems where scaling, by itself, cannot reach the answer. Not because the models aren't big enough yet. Not because the datasets aren't large enough yet. Because the information required to solve the problem is not in the corpus at all, and no amount of additional corpus will put it there.

The class of problems is large. It includes most of the domains where AI is currently being promised as a transformative force in the next decade — commercial decisions, legal strategy, medical diagnosis, therapeutic intervention, executive coaching, complex recruitment. Anywhere the system is supposed to understand and influence adversarial human behaviour in real time.

This is the black hole at the centre of contemporary AI. It is the gap between what the corpus contains and what intelligence in these domains actually requires. And until you see it clearly, the architectural shift that needs to happen next stays invisible.

Let me try to make it visible.

Start with what's true.

For a class of tasks — translation, summarisation, code generation, image recognition, large-scale pattern matching across structured domains — scaling has been the closest thing to a solved problem in computer science. You take more data, you train a bigger model, capabilities improve. Often dramatically. Sometimes in ways nobody predicted.

The intuition behind why scaling works is itself elegant. The world generates a vast and continuous record of itself. Books, articles, code, images, conversations, transactions — all of it gets written down somewhere, indexed somewhere, scrapeable somewhere. A sufficiently large neural network trained on a sufficiently large slice of that record learns the statistical structure of the world's recorded behaviour. The patterns are in the data, the model is the apparatus that surfaces them, and the result is a system that can answer questions about almost anything humans have written about.

This works because, for most of the tasks people care about, the relevant information genuinely has been written down. Translation works because every concept needed to translate between languages has been expressed in both languages somewhere in the corpus. Code generation works because every programming pattern that's ever been useful has been published in some repository. Even reasoning, to the extent that contemporary models do it, works because the structure of human reasoning has been captured in millions of articles, textbooks, and explanatory writings that are now part of training data.

The seductive idea is that if this works for those domains, it works everywhere. That given enough data and enough compute, neural networks become general-purpose intelligences capable of operating in any domain humans operate in.

The seductive idea is wrong about a specific class of domains. And the wrongness isn't a quantitative limitation that more compute will solve. It's a categorical one.

Imagine you've been asked to predict whether a buyer will commit to a £400,000 software contract on a particular Tuesday afternoon at 3pm. You have access to every sales transcript ever recorded. Every conversation between every salesperson and every prospect, perfectly transcribed, perfectly indexed, perfectly available. A trillion examples.

What's actually predictive of the outcome at 3pm on that Tuesday?

Some of it is in the transcript. The words spoken. The questions asked. The responses given. The pacing. These are real signals and a sufficiently capable model can learn them.

But the dominant signal — the thing that actually determines whether the buyer commits — isn't in the transcript at all.

The buyer slept badly the night before. He had an argument with his partner over breakfast. His CFO told him at 11am that the budget is tighter than he thought. His mother is in hospital. He is, despite all of this, the kind of person who makes decisions when fatigued rather than postponing them, and on a Tuesday afternoon his decision style tilts toward avoidance rather than commitment. None of this is written down anywhere. None of it ever will be.

The salesperson, sitting opposite him, is processing all of this in real time without consciously knowing she's doing it. She reads his face. She notices the slight delay before he answers questions about timeline. She picks up that his usual confidence is calibrated lower than the last meeting. She adjusts her approach without articulating why. If she gets it right, the deal closes. If she gets it wrong, it stalls. Either way, the thing she got right or wrong — the contextual reading that determined the outcome — never makes it into any record.

This is the black hole. The enormous contextual envelope that surrounds every commercial decision is generated in the moment, processed in the moment, and consumed in the moment. It leaves almost no trace. A trillion transcripts contain a trillion shadows of decisions whose actual mechanics are invisible. The corpus preserves the surface; the structure underneath is gone before anyone could write it down.

This is not a quirk of sales conversations. It's a structural feature of any domain where humans make adversarial decisions in real time.

The therapist sitting with a patient at 4pm on a Wednesday is reading hundreds of variables that don't appear in any case note. The lawyer choosing whether to push or yield in a negotiation is processing signals about the other side's resolve that won't be captured in the deal memo. The doctor noticing something off about a patient's gait is reading a contextual envelope that no medical record will ever contain. In every one of these domains, the practitioner's intelligence is operating on information that exists only in the moment, and the corpus of recorded outputs from these domains contains the decisions without the underneath.

You can train the largest neural network you want on every legal document, every therapy note, every medical record ever produced. You will not learn what the practitioner was perceiving when they made the decision. Because that perception — the thing that actually mattered — was never written down.

This is what I mean when I say the information isn't in the corpus.

The standard response, when someone points out a limitation in current AI systems, is to say: yes, but the next generation of models will be bigger. The argument is that as models scale, capabilities emerge that nobody anticipated. So perhaps a sufficiently large model will learn the contextual envelope by inference, even if no individual training example contains it.

This is worth taking seriously, because it's been right before. Capabilities have emerged from scaling that researchers didn't predict. Reasoning over multiple steps. Translating between language pairs the model wasn't explicitly trained on. Writing code in styles that weren't in any training example. The history of "this won't emerge from scaling" claims is largely a history of claims that turned out to be wrong.

But the limitation I'm describing is structurally different from the limitations scaling has overcome.

When scaling produces an unexpected capability, it's because the raw material for that capability was in the training data, distributed across many examples, in a form the model could learn to integrate. Reasoning emerges because reasoning is implicit in the structure of millions of arguments humans have written down. Translation emerges because the parallel structure of meaning across languages is implicit in the existence of bilingual content. The model isn't conjuring these capabilities from nothing. It's surfacing structure that was latently present in the corpus all along.

The contextual envelope of human behaviour isn't latently present in the corpus. It's categorically absent. The buyer's Tuesday morning mood is not implicitly distributed across millions of sales transcripts. It's not in any of them. No degree of pattern aggregation across the available record will reconstruct it, because reconstruction requires some signal about the underlying variable, and there is no such signal anywhere in the data.

You cannot triangulate a coordinate that has never been observed.

This is the difference between problems where scaling helps and problems where it doesn't. For the former, the data contains the structure and the question is whether you have enough compute to find it. For the latter, the data does not contain the structure, and you can throw infinite compute at it without recovering what was never captured.

The black hole is the second kind of problem. It is not solved by larger models. It is solved by acquiring the data the corpus was missing. And there is only one place to acquire that data: the moment of the interaction itself, instrumented in real time, captured before the contextual envelope evaporates.

There's a tempting alternative path that I should address before going further, because a thoughtful reader will already have raised it: if the contextual envelope is what experts perceive, can't we just learn from experts?

This is the imitation-learning idea. Watch enough expert salespeople, lawyers, therapists, doctors. Capture their decisions in their fullest context. Train a system to imitate them. The system inherits whatever the experts were perceiving, even if neither the experts nor the system can articulate it.

The path looks promising. It is also a dead end, for two reasons.

The first is that experts cannot reliably introspect on their own performance. The salesperson who closes high-value deals usually cannot explain why her judgment is good. She'll give post-hoc rationalisations — she'll tell you about reading the room, about gut feeling, about experience. The actual variables she's processing in the moment are not available to her conscious reflection. This is well-established in cognitive science: expert performance in adversarial domains is largely tacit, and experts' explanations of their own behaviour are reconstructions rather than recordings. So the data you'd need from experts — what were you actually perceiving when you made that choice? — they themselves do not have.

The second reason is more fundamental. Even if you could perfectly capture what experts do, you'd be capturing their behaviour without their understanding. The system would learn to act like an expert without learning why the expert's actions worked. This is sufficient for stable domains where the patterns don't change much. It is insufficient for adversarial domains, where the other side adapts and the rules shift. A system that imitates expert behaviour without understanding the underlying dynamics will fail the moment the dynamics change. And in commercial, legal, therapeutic, medical interactions, the dynamics are always changing — that's the whole point.

The same logic applies to the related approach of preference learning. If you can't observe the underlying variables, you might at least ask humans to rate which outputs are better, and use those ratings as a training signal. RLHF, in current AI systems, is exactly this. It works for many tasks. It does not work for the black hole, because preference ratings reduce a multidimensional dynamic to a scalar judgement, and the scalar judgement loses precisely the structure that mattered. You learn what humans prefer without learning what produces good outcomes. These are different things. A buyer might prefer a smoother conversation while ultimately committing more reliably to a salesperson who pushed back uncomfortably. Preference ratings reward smoothness; outcome data rewards what actually worked. They diverge constantly in adversarial domains.

The corpus doesn't contain the underneath. Expert imitation can't recover it. Preference learning collapses it. We have systematically eliminated the available approaches for solving this class of problem.

Which leaves only one place left to look.

If the information required to predict and steer adversarial human behaviour isn't in the historical record, isn't recoverable from expert imitation, and isn't capturable through preference ratings, then the only remaining source is the moment itself.

The buyer's contextual envelope on that Tuesday afternoon doesn't survive into any corpus. But it exists, fully and observably, while the conversation is happening. The salesperson is reading it in real time. The system, if it is going to perceive what the salesperson perceives, has to be inside the loop while the loop is running. There is no other position from which the relevant data is visible.

This is the conceptual move that, once made, is hard to unmake.

Applications — the surfaces through which humans interact with software in these domains — are not delivery mechanisms with data as exhaust. They are the only sensory apparatus that can capture the contextual envelope while it exists. Without the application, the information is gone the moment the interaction ends. With the application, instrumented at the right resolution, around the right measurement framework, the system can finally observe what the practitioner is observing — not after the fact, not through a transcript, but in the moment, at the granularity at which the actual dynamics operate.

Product design under this paradigm becomes, in part, a question of what does the model need to perceive? The application's job isn't only to serve the user. The application's job is also to be the perceptual surface through which a learning system finally has access to the dynamics it could never extract from any corpus.

This is the inversion. It is also the only path I can see by which AI systems become genuinely capable in the black-hole domains. You don't get there by scaling models trained on static records of human behaviour. You get there by building applications that turn the live moment of human behaviour into structured observation, and training systems on the closed loop between observation, prediction, and outcome.

What I have just described is, in compressed form, the architectural premise of a training paradigm I am calling Reinforcement Learning from Observed Behaviour — RLOB — and the type of system it produces, which I call a living repository. The full technical paper is at FBED Labs, with permanent DOI, and is the first in a planned sequence. The position paper does the formal work of positioning RLOB against adjacent training regimes, defining its architectural requirements, and presenting early empirical evidence from a commercial deployment.

What I want to do here is the philosophical work that the position paper deliberately omits. The position paper introduces the paradigm. This piece argues that the paradigm is necessary — that the alternative, a future where AI systems in adversarial domains are produced by scaling neural networks against static corpora, leads to a ceiling that no amount of compute will pierce.

I think the field is going to discover this ceiling within the next three to five years, and that the architectural shift toward instrumented sensing in real-time domains is going to become obvious in retrospect, the way most foundational shifts do. I think the people building serious applications today should already be thinking about their products as sensory apparatus, not delivery mechanisms. The data their applications generate is not a byproduct. It is the primary asset. The model that learns from it will be the asset that compounds.

Let me state the implication plainly, because it matters beyond AI research.

If the black hole is real — if there is a class of human-behavioural domains where corpus learning categorically cannot reach the underlying dynamics — then a substantial portion of the AI deployment that's been promised over the next decade cannot happen through scaling alone. Not because the technology is too immature. Because the technology is being pointed at a class of problems where the technology, in its current form, is structurally insufficient.

This affects how investment in AI should be allocated. It affects which companies will produce durable advantages. It affects which application categories will generate genuine new capability versus which will produce sophisticated chatbots that approximate the surface of expert behaviour without ever reaching the underneath.

The companies that will own the high-value AI capabilities of the next decade, in my view, will not be the ones with the largest models. They will be the ones with the best instrumented application surfaces in adversarial human domains. The ones whose products are designed, deliberately, around what the model needs to perceive. The ones whose deployment is itself their training infrastructure.

This is the architectural shift I think is coming. RLOB is one specific technical formulation of how to operate inside it. The broader claim — that AI in human-behavioural domains is going to be transformed by treating applications as perceptual systems — is one I expect will be widely held within a few years, even if the people who eventually hold it arrive at it through different framings than mine.

The black hole has been there the whole time. The data we needed has always existed. The mistake was thinking it would show up in the corpus. It won't, and it can't. The path forward is to build the instruments that finally see it.

The work continues at FBED.