When the AI companies first began talking about existential risk, it was common to hear some version of:
“Look, this is bad for business, so obviously they must be telling the truth.”
In retrospect, existential risk has been the best capital-raising tool ever in the history of the world. That doesn’t mean there were no true ethical motivations accompanying the doom warnings mixed in. There were. But in the end, embracing existential risk was also the right move from a cold-eyed business perspective.
Similarly, it’s perplexing why a company would want their AI to be conscious. One could again say:
“Look, this is bad for business, so obviously they must be telling the truth.”
Yet I think it will turn out that being coy about AI consciousness, and designing AIs to be as “seemingly conscious” as possible, is great at both attracting top talent and ensuring customers form parasocial relationships. Much like existential risk, claims to artificial consciousness can be based in honest opinions, while also being extremely useful for a massive corporation pursuing its ends.
I. UNRELATEDLY, LET’S TALK ABOUT ANTHROPIC’S LATEST PAPER
Anthropic’s big research announcement, accompanied by beautiful figures and a huge social media push, is claiming that AIs (like their Claude) have a “global workspace,” which is a term borrowed from neuroscientific consciousness research.
The sheer overwhelming power of Anthropic’s social media reach, design team, and the halo of being the hottest company in the world means that their research screams its underlying intentions, simply because every short step from neutrality is so impactful. And what their research direction seems to be is that while they cannot prove that Claude has real consciousness, and so truly suffers or gets frustrated or actually loves you back, Claude seems to have everything else that’s essential when it comes to consciousness. And Anthropic’s research project is going to show this as if checking off a list.
Their latest work seems an obvious sequel to their previous research on LLM “introspection,” which was also a blockbuster on social media. However, experiments on LLMs are extremely difficult to control for: you need an experiment, then an interpretation, and then you do controls to confirm your interpretation. It’s the last part that’s tricky. A lot of the field is basically inventing functional neuroscience from scratch, which has also been a field plagued with problems. For example, what Anthropic previously anthropomorphized as “introspection” may not really be introspection at all, once you do the proper controls. There are non-anthropomorphic interpretations that fit the data better.
Like their older paper on introspection, this newer paper on global workspaces and consciousness seems unlikely to ever be peer-reviewed. It’s worth remembering that everyone was rah rah rah let’s all do science on blogs, it’ll be so great. Down with scientific publishers! Down with peer review! Let freedom ring! Well, perhaps predictably in hindsight, the difference between press releases and research is getting thinner and thinner.
In fact, rapid-fire massive releases are hard to distinguish from a Gish Gallop: this paper is so big it’s difficult for a commentator on these topics (like myself) to address everything in a timely manner. However, no sub-experiment or control fundamentally deals with the critique I’m about to give (even if I can’t go through all of them in detail). That’s because the critique is about the method they base everything else off of: “J-space.”
II. J-SPACE AS AN UNFALSIFIABLE TRACKER OF CONSCIOUSNESS
Is Anthropic using a sensible way to track or measure consciousness? Consider the literal title of the paper: “Verbalizable Representations Form a Global Workspace in Language Models.”
A focus on “verbalizable representations” runs a risk I’d pre-registered when I pointed out earlier this year (and even back in 2021) that if you took global workspace and stripped it down to a bare minimum, you get a theory built solely on reportability, and that such a theory would be scientifically trivial—a symptom of consciousness research being pre-paradigmatic.
So did Anthropic do this? Did they embrace a trivial scientific theory of consciousness and then make big claims to AI consciousness off of it?
Note that the focus on reportability is true in the math—their entire analysis is derived off of “Jacobians” calculated per layer. What are the Jacobians in this context?
Our results make use of a new interpretability technique called the Jacobian lens (J-lens), which is designed to identify internal representations that are readily available for verbal report. For each token in the model’s vocabulary, the Jacobian lens identifies a vector representation that encodes the potential for the model to verbalize that token in the future. Concretely, it computes, for each layer, the average linearized effect of an activation on the model’s likelihood of producing a particular token (now or in the future), averaging over a large corpus of contexts (see Methods for details).
Everything that follows is based on these Jacobians. But what are Jacobians, for those who don’t know how to interpret the “linearized effect of an activation on the model’s likelihood of producing a particular token… averaging over a large corpus of contexts?” Here’s a picture, again from the paper:
Is that helpful? Also no?
Let me try to give a (simplified) definition that is slightly less accurate but might make more intuitive sense: their whole analysis starts with a measure of how much causal control the activations of artificial neurons (the internal state of a layer) have over the future output layer (the last layer) averaged over a set of prompts as context. Strip away the math, and you have something close to a score that asks “How much does a small nudge in this internal set of activations shift the output?”
Conceptually then, this is like a mathematical measure of a disposition toward report; in other words, reportability. Or, by being an average, it’s more like smeared reportability. Anthropic then has a way to relate these smearings of reportability to specific words, and so you get something like a stream of thought.
What you’re left with at the end of this process has some pretty big differences from previous, more traditional notions of global workspaces, which would a priori not apply to an LLM since they involve modularity, reentrant dynamics, and other things that LLMs just flatly lack.
Credit to them, Anthropic admits that their “global workspace” is different in important senses. E.g., they write that:
As part of their internal processing, might LLMs have developed a global workspace of their own, to serve a functional role analogous to conscious access? It is not obvious that they should; in the brain, the workspace is closely associated with recurrent dynamics and brain region interactions that have no direct analog in the transformer architecture on which LLMs are based. On the other hand, maintaining a global workspace is likely computationally useful: a common representational format allows intermediate results to be written once and read by many neural processes.
But this poses a question: so which definition should you use? The previous neuroscientific definitions that obviously don’t count LLMs as having a global workspace, or the new one that does?
In order to make the definition work, they drop most requirements for a “global workspace” outside of things that naturally go along with reportability (e.g., how could reportability not involve internal reasoning, or generalization?). This means there’s a potential deflationary account for nearly everything, and the paper is one long fight against those. E.g., the deflationary version of their definition of ignition is just when the model commits to an interpretation (and since their measure of this is to give ambiguous prompts, of course they see a change). The deflationary version of their definition of broadcast is just that information with high reportability is more likely to be used by later layers. Etc.
Admittedly, I can’t address the garden of forking paths around this research, nor all their sub-experiments, but I currently don’t think any of the paper’s contents magically avoids the measurement problems I’ve been pointing out about consciousness research. There remain just-as-viable alternative takes on J-space: for instance, you could also say it is the “inner monologue.” Or you could go as non-anthropomorphic as possible and hold it is just a relatively constant transformation of internal processing into the output that starts at some point, but after that basically just starts accumulating, which is why as you get closer to the output, more and more of the J-space matches that output, and the more content you look at, the more linear these curves look.
All these issues are not entirely Anthropic’s fault: we simply have very bad theories of consciousness.
To fix this, I’ve been working on a way to narrow down the possible state-space of theories of consciousness (see here, and here), eventually reverse engineering post-paradigmatic theories of consciousness. It’s sort of like judo: if building good theories of consciousness is hard, maybe that’s actually good! We can use the difficulty of the problem of consciousness against itself.
One of the central insights is that there are various conditions under which theories of consciousness become unscientific: e.g., they become unfalsifiable, or they become completely trivial, etc. Most contemporary theories fall into these categories once push comes to shove, but most theories are neither ever pushed nor shoved.
One such failure state is “strict dependency,” which is when an experimenter’s inferences about consciousness (like verbal report) come from the same source as the predictions about consciousness from a theory (e.g., like that consciousness is equivalent to verbalizable representations!). Here’s a visualization of this in a testing schema for theories, pointing out that if a theory entails strict dependency, predictions and inferences can never truly be pulled apart.
And here’s me earlier this year describing how these meta-scientific problems crop up for global workspace theories:
E.g., when discussing Global Neuronal Workspace Theory (GNWT), Daniel Dennett once wrote that: “... theorists must resist the temptation to see global accessibility as the cause of consciousness (as if consciousness were some other, further condition); rather, it is consciousness” [47].
If Dennett’s ideas were true, this would make GNWT a trivial theory of consciousness, since it would be unfalsifiable, because the predictions and inferences strictly depend on the same source. Consciousness just is the information that is globally accessible for report and behavior, and reports and behaviors also just are different expressions of that same information. So there is no situation wherein they could diverge. And therefore, the theory is unfalsifiable and trivial (one could see how it gives us minimal scientific information even if it were true—trivial theories are fundamentally unsatisfying). Often, theories based in the “theater” metaphor have a tendency to fall into strict dependency (sometimes “global workspaces” are described in this way too [48]).
Notice that this is, at a high level, mostly what this new paper is doing.
And their analysis shows this is true! Specifically, they also test an even more direct measure of reportability from mechanistic interpretability: the “logit lens.” In this analysis, you (again, basically) just freeze the network and ask it to output at the layer you want to look at. And behold, that alone can capture “much of the workspace-like structure” (emphasis mine).
Empirically, the two lenses agree closely in the model’s last several layers and diverge earlier, with the J-lens recovering interpretable content at depths where the logit lens does not. However, we find the logit lens to be quite useful in practice, and to capture much of the workspace-like structure identified by the J-lens, though with somewhat lower reliability (particularly in earlier layers).
Anthropic appears to have performed the reductio that I said is a perfect example of why we need better theories in consciousness science. Loosely inspired by global workspace, Anthropic introduced a (supposed) way to track consciousness that is built precisely out of backtracking from reports, which is what I warned would lead to measurement issues. They then are constructing a case for AI-consciousness off of this unfalsifiable and trivial scientific theory. Which they find evidence for, sure… because of course they would!
A clever defender of Anthropic would point out that the analysis built off of the J-space is not just a direct measure of reportability. If it were just reportability, then why wouldn’t the workspace include the later layers of the network, those closest to output, which they dub the “motor layers?” So are there not falsifiable aspects, specifically, the prediction of the layers forming a global workspace with a tripartite structure of sensory/workspace/motor?
Yet in the paper, the tripartite structure seems to only exist because the researchers impose a criterion that, if the read-out of the verbalizable representations mostly just is the output, it somehow doesn’t count. They purposefully exclude non-smeared verbalizable representations, even though it’s just sitting there. Here is them admitting that (emphasis mine).
In §4.1, we characterized the workspace as existing in a range of intermediate layers: the J-space carries little information before this range, and in the final few layers transitions to representing the model's imminent output rather than its intermediate computations. We identified this late boundary empirically, by analyzing statistics of J-lens vectors, their activations, and their relationship to the model’s output. But this judgment was somewhat post-hoc, and we did not provide a principled definition of what distinguishes a "workspace" representation from a "motor" one. Indeed, as discussed in §4.1, sometimes the model’s next-token predictions do appear in the J-space in the intermediate range we focus on, even if they do not appear as reliably as in late layers.
To them, this probably doesn’t seem like a major limitation, but to me, it is admitting something like: “The only difference between the current theory of consciousness we’re working with and one that is provably trivial is, uh, somewhat post-hoc.”
III. DO OPEN SOURCE MODELS ALREADY FALSIFY THE FEW FALSIFIABLE PARTS?
There are more measurement problems I’ve identified with consciousness research beyond just unfalsifiability, and Anthropic appears to be running into those too.
Imagine that we have some theory of consciousness, and we ask what it predicts for a specific system S1. Then we cleverly find another system, S2, which approximates the same reports and behavior as S1, all while the predictions are entirely different (this is the other horn of the “Kleiner-Hoel dilemma”).
E.g., you might think global workspaces are necessary for consciousness. Then some evil demon builds a system that has rich input/output just like systems with global workspaces, but doesn’t meet your global workspace requirements. Whose reports do you believe?
Initial results (still early) indicate that Anthropic’s research runs into this issue as well. In the Anthropic paper, they provide the following key figure, which shows off the sharp tripartite structure they are labeling as sensory/workspace/motor, based on the correlations of the J-space (it’s more complicated and based on geometrical matching) across layers, a quantity referred to as the CKA.
So far so good. I literally see it: at the bottom left is sensory. The middle square is the global workspace. The top right is motor (which we know has some principled problems, but let’s leave that aside).
What about other models beyond Sonnet 4.5? They assess some others in the Claude family, but basically only include this ominous text:
We note that in some models the transition is more gradual, sometimes containing sub-blocks, and that the observed sharpness is exaggerated by layer subsampling.
Wait a minute.
The CKA revealing a sharp blocky tripartite structure is arguably one of the most important parts of the entire paper, precisely because it’s not (entirely) baked-in by the measure’s grounding in reportability. Why? The Jacobians are calculated per layer, and at each layer, they are recalculated. So there is no cohesive “workspace” built into the measurement. A workspace could appear if there is some kind of abstract space in which contents are actually contiguous and relatable across layers (here, the CKA assesses that through a kind of relational geometry). If the contents and manner of processing are not held contiguous as the stream of information goes forward across layers, that would obviously not be a workspace even by the loosest of definitions, since it would mean at no point are the contents sequestered into a “space” for all those functions a global workspace is supposed to perform; consequently, there would be no sharp blocky tripartite model of sensory/workspace/motor.
Luckily, someone went and ran their analysis (kudos to Anthropic for open-sourcing at least the algorithm for J-space itself) on open-source models. Now, this isn’t my analysis, but it does seem to use Anthropic’s code directly (and is implemented by someone at Prime Intellect). Because it’s not mine, I don’t want to lean too heavily on it—it’s more of a demonstration of exactly the kind of problems I’d predict. While it’s not an exact replication (which would be important to see—but Anthropic doesn’t actually release enough details to fully replicate anyway!), it appears to be a relatively natural one, and based on their code. The immediate question is:
Where is the sharp tripartite sensory/workspace/motor structure across other models?
Specifically, each cell in the diagonal here is (arguably close to) a reproduction of the above paper figure, but for an open-source model (organized by parameter size).
Yet, we see (at least in this analysis) that for open-source language models, there is no sharp blocky tripartite structure. Remember, the claim of the paper is “Verbalizable Representations Form a Global Workspace in Language Models.” Here are a bunch of language models. Where is the global workspace? Where is the sharp tripartite structure of sensory/workspace/motor?
If this holds true, it’s a great example of how hard the “substitution argument” bites: many of these models can do multi-step reasoning tasks, many of them are not dumb compared to Sonnet 4.5; in fact, in many conversations in the chat window, you would have no idea which one you were talking to. So they can act as experimental substitutions: if you make a prediction about the consciousness of Claude due to a global workspace, here are a bunch of Claude lookalikes (or close enough to matter for falsification purposes) without anything sharp and boxy (or with too many boxes!)
In fact, one wonders about whether their own analysis showed something similar for the other models they did analyze in the Claude family. Because we definitively see a sharp blocky tripartite “workspace” result only once, from just Sonnet 4.5. And nobody knows exactly how Claude Sonnet 4.5 is built, or how they did their sampling to get that figure, or how baseline connectivity is organized, or what architectural factors might explain that. Only Anthropic.
IV. AI PUSHES CONSCIOUSNESS RESEARCH FORWARD IN THE FUNNIEST WAY POSSIBLE
I’ll be honest—it’s a little bit fascinating to see a company with billions behind its research efforts run into the exact measurement problems I’ve spent years advocating we fix. My prediction was always that any serious attempts to apply theories of consciousness with advanced neuroimaging would lead to weird measurement issues… and Anthropic’s work on AI consciousness, which is essentially applying perfectly veridical neuroimaging to artificial neural networks, has run right into the mountainous side of the issue before the rest of neuroscience has even grappled with the problem.
This is how AI research ends up stimulating consciousness research: not by actual findings (you can’t empirically prove Claude is conscious without discovering more about human consciousness), but by the problems it runs into!
Everyone is thinking: “Oh, let’s use the stuff from consciousness research to figure out AI consciousness.” Wrong! It’s “Let’s use the problems in AI consciousness research to figure out what’s wrong with the stuff in consciousness research and fix it.”
So, my overall opinion: “Verbalizable Representations Form a Global Workspace in Language Models” is an interesting paper on the reportability of information in LLMs and how the property of reportability may be useful for multi-step reasoning in particular. Yet in the surrounding press, the implication is that after this research, really the only thing missing from LLMs is phenomenal consciousness.
This is a devilishly clever stance if the eventual move is to argue strongly for AI consciousness. It seems designed to put people in a position wherein the only thing not solved now (or soon) in LLMs is the infamous Hard Problem of how experiences or subjectivity arises from objective goings on like neural activity (or matrix multiplication, if that’s possible). The pro-LLM-consciousness position can then point out that the Hard Problem is not solved in humans either, so what’s the difference?
But because it is implicitly built on an unfalsifiable and trivial pre-paradigmatic theory of consciousness (and the few parts that aren’t may end up being easily falsified by substitutions), this paper does not prove much, other than how difficult measurement problems for consciousness are.
And I fully admit this is not entirely Anthropic’s fault. As I wrote recently: “We consciousness researchers have failed you.” Well, we also failed Anthropic. We need an effort (yes, like the one I am starting with Bicameral Labs—sorry for the plug, but it’s literally true that this is something I’m working on) to reverse-engineer post-paradigmatic theories of consciousness that aren’t just trivial, low-information, or unfalsifiable.
Otherwise, in the scientific fog of war, it will be easy for other factors, like press releases or pretty graphics, to determine what’s true to the public.
One more observation, as an AI researcher. Even beyond the protein issue I identified earlier, the paper is flourish. It is just a standard interpretability result. The claim reduces to: a transformer keeps a small set of readable directions in its residual stream that hold intermediate values of multi-step computation and get reused by many downstream components. This statement is about representational format and reuse, which you should expect from any neural network that has to chain steps efficiently. Even the author acknowledge "computationally efficient." The global workspace framing adds no predictive content. Every result the paper reports follows directly from representational efficiency. The one property that I would want to see (which would distinguish the workspace), is sharp nonlinear ignition. This is the one they cannot show. The paper's real contributions are narrower. (1) The Jacobian lens recovers intermediate representations that the logit lens misses. (2) The directions it finds are the same ones that causally drive behavior.
What I'm getting from this is that Anthropic has published not so much a theory as an in-depth analogy to a theory, sort of like saying a toaster is like a wildfire because they both produce heat at first gradually and then intensely, and therefore toasters in the right circumstances could have the ability to restore health to forest ecosystems. Is that about right? I don't have the knowledge to fully follow what is being said. But I'm having trouble understanding why Anthropic's "research" would be taken seriously, and why we would have an expectation at all that a mechanical computation system, no matter how sophisticated/powerful it is at analyzing massive inputs consisting of the output of biological systems (ie human brains), would be able to replicate a biological system? (I do sort of get what you are saying about the difficulty of measuring consciousness, which is interesting.)