A Grammar of Machine Understanding
On the gap between statistical competence and genuine comprehension in language models — and whether that gap can be closed, or only narrowed.
There is a question that keeps reasserting itself in the margins of machine learning research, too big to answer and too important to ignore: do language models understand language, or do they only behave as if they do?
The question has the uncomfortable quality of being both philosophically fundamental and practically irrelevant to most engineering work. A system that reliably behaves as if it understands is, for most purposes, sufficient. We do not, after all, require our calculators to experience number. But the framing conceals a substantive problem. Statistical competence — the ability to produce contextually appropriate outputs — and genuine comprehension are not the same thing, and the gap between them matters when systems are deployed in contexts where the failure modes of each differ.
The Competence-Comprehension Gap
Consider what a language model learns during pretraining. It learns, at minimum, a highly compressed model of the statistical regularities of human-produced text. From this it derives something that looks remarkably like an understanding of grammar, of inference patterns, of factual relationships, of conversational norms. In many evaluation contexts, this is indistinguishable from comprehension.
The gap shows itself at the edges. Models fail on tasks requiring consistent application of a rule across contexts that are distributionally distant from training data. They fail on tasks requiring genuine grounding — connecting linguistic representations to stable, non-linguistic referents. They fail, most revealingly, when asked to reason about their own epistemic states. Not always, and not systematically enough to make a clean theoretical claim. But often enough that the failures feel different in kind from the failures of a system that understood and merely made an error.
What Would Comprehension Require?
I find myself drawn to a view that comprehension requires something like an interpretive framework — a structured, revisable model of the domain that provides the backdrop against which linguistic expressions acquire meaning. This is not the same as a knowledge graph or a set of first-order axioms. It is something more dynamic: a system of constraints that shapes the space of possible interpretations and that can be updated in response to new information without losing coherence.
Language models have something in this vicinity. Their internal representations encode relational structure, causal intuitions, and something approaching a theory of mind. Whether this constitutes an interpretive framework in the required sense, or merely a very good simulation of one, is a question that current interpretability tools cannot answer.
What I am more confident of is this: the path toward genuine comprehension, if such a thing is possible in silicon, runs through structure. Not purely symbolic structure — the brittleness and incompleteness of classical AI representations are well-documented — but hybrid structure, in which continuous learned representations are anchored by discrete, compositional scaffolding. This is the intuition behind the neuro-symbolic research programme, and it is, I think, the right one.