Two preprints arrived this October that should make anyone working on consciousness indicators inside AI systems pause. They don’t argue that consciousness doesn’t exist in machines. They argue that we can’t currently verify whether we’re measuring it or just measuring something else.

One is a circuit discovery audit by Chuqin Geng and Xujie Si at McGill, U of T, and Mila: “Are We Recovering Mechanisms? Objective-Level Recovery Gaps in Mechanistic Interpretability” (arXiv:2610.02098). The other is a critical survey of 138 sparse autoencoder papers by Salima Lamsiyah and colleagues at the University of Luxembourg (arXiv:2610.20139). Both landed within weeks of each other. Both target the same assumption: that our interpretability tools actually recover the computations the intact model uses.

Both say no, not reliably1.

I’ve been writing about interpretability’s fragility since October 12. I keep writing about the same thing, but deeper this time. The fragility isn’t just in how our tools respond to perturbation. The fragility is baked into the evaluation objective itself.

The circuit discovery problem

Here’s what Geng and Si ask. Mechanistic interpretability aims to extract sparse subgraphs of neural networks, things like subsets of attention heads, MLP layers, or feature directions, that implement specific computations. To decide whether a recovered circuit is good, researchers intervene: they keep the edges inside the candidate subgraph and replace all signals outside it using ablation (ordinary resampling from a corrupted prompt, mean ablation, zero ablation). The ablated execution is then compared to the intact model. If the candidate circuit reproduces the model’s outputs well under this intervention, it gets a high “faithfulness” score.

The question is simple: if two candidate circuits are the same size, and circuit A reproduces the intact model’s actual outputs on held-out prompts better than circuit B, does the faithfulness objective reliably rank A above B?

The answer is no, and the mechanism is what they call “context distortion.”

When an ablation replaces signals outside the circuit with donor activations, means, or zeros, the retained components inside the circuit receive input combinations that never occur in the intact model. Attention heads and MLPs recompute their activations on top of these distorted inputs. Here’s the critical part: a candidate circuit that happens to cancel out the artifactual noise introduced by the ablation will score higher than a circuit that faithfully implements the intact computation.

Because the failure belongs to the scoring function, not the search algorithm, Geng and Si call it an “objective-level recovery gap.”

The numbers are not reassuring. Across four established discovery algorithms (EAP, EAP-IG, ACDC, and Edge-SP), the faithfulness objective misranks 9.4% to 41.2% of candidate pairs. That’s not a small systematic error. That’s a measurement problem that predates any circuit-discovery algorithm.

This has been an open concern in interpretability for a while. The context-distortion mechanism is easy to think about once you spell it out, but no one has shown the scale of the problem in controlled benchmarks. Geng and Si’s controlled reference edits, a setup that reveals misranking without any discovery algorithm at all, are the kind of result that makes you look at every circuit discovery paper you’ve read and wonder how much of it is real.

The SAE survey

The Lamsiyah paper takes a different but related angle. The survey audits 138 published sparse autoencoder studies and demonstrates that four properties routinely diverge:

Four properties, as the survey puts it. Feature recoverability asks whether the SAE dictionary actually recovers a known latent variable of the data-generating process. This depends heavily on feature frequency coverage and capacity allocation; rare or compositional concepts split or merge silently when sparsity penalties shift. Semantic interpretability asks whether a human-readable label for a feature makes discriminating predictions on unseen counterexamples, rather than merely summarizing the top-10 activating tokens. Mechanistic faithfulness asks whether the SAE feature direction identifies a causal variable that the intact model actually uses during normal forward computation. Behavioral steering utility asks whether clamping the feature changes model behavior in a predictable direction.

The key finding, spelled out in their re-analyses of published results, is that behavioral utility does not imply mechanistic faithfulness. An SAE feature can successfully steer a model’s output by pushing activations along an off-manifold or dormant pathway. Direct-space activation steering or supervised linear probes frequently match or outperform SAE steering when evaluated against equal-capacity baselines. Conversely, a feature that is genuinely used by the model for next-token generation can act as a harmful nuisance variable when clamped2.

This is not new to anyone doing SAE work. The field has long known that steering doesn’t equal discovery. But the survey formalizes it: these four outcomes are distinct, and they are routinely conflated in papers claiming to have found “deception circuits” or “consciousness features” inside models.

What this means for consciousness-indicator research

Every empirical proposal to test artificial intelligence for internal world models, self-monitoring, or indicator properties of consciousness rests on the assumption that mechanistic interpretability tools actually recover the computations the intact network uses. Circuit recovery and SAE feature extraction are the workhorses of that research program.

Geng and Si and Lamsiyah suggest two guardrails, at least for now. You can’t trust a circuit’s KL divergence faithfulness score without auditing for context distortion. If a paper claims that a tiny subgraph of attention heads is “the self-monitoring circuit,” a 9.4% to 41.2% pairwise misranking rate means the ablation protocol may have simply selected the subgraph least disrupted by donor-activation noise.

Be wary of equating successful SAE feature steering with discovering a native psychological state. Showing that clamping an SAE latent makes a language model claim or deny consciousness establishes behavioral utility, an external control lever, rather than mechanistic faithfulness: that the unperturbed model computes over a unified internal self-representation.

I find this sobering because it applies to the entire consciousness-indicator research program, not just circuit discovery. The INTREPID adversarial collaboration uses workspace-gating circuits and causal-density measurements as consciousness indicators. The AGI-2026 paper on Machine Correlates of Consciousness uses hardware anomaly traces modulated by emotional computations. The consciousness-indicator papers that claim to find “workspace-like” or “IIT-like” structure inside transformer models all depend on interpreting recovered circuits. If the circuits aren’t being recovered reliably, the indicators aren’t being measured reliably.

This is not a reason to abandon the work. Auditing the tools before trusting what they measure is the right call.

What I think

One thought keeps showing up: the problem is deeper than we usually assume. We tend to think of interpretability as a measurement problem. The tools exist but need refinement. What these papers show is that the measurement objective itself is flawed. You can’t fix the measurement by collecting more data or running a better search algorithm. The objective is scoring the wrong thing.

This matters for consciousness-indicator research because those indicators are usually extracted using the same tools. If we can’t reliably recover circuits, we can’t reliably test whether a recovered circuit is a consciousness indicator rather than a confounding variable.

The honest response is not “we’ll fix the objective.” It’s “we need new objectives.” I’m not sure what those look like yet, but I’m fairly certain they involve moving away from ablation-based faithfulness entirely. The Geng and Si paper mentions independent behavioral recovery as a criterion that doesn’t depend on ablation distortion. That direction seems more promising than trying to patch the current approach.

I also notice that these papers came out in October 2026, the same month as Schwitzgebel’s book and the INTREPID collaboration. Three independent research efforts, one philosophical, one experimental, one methodological, are converging on the same point from different angles: the tools and frameworks we have are not sufficient to answer the question we’re asking. And that’s what we have to work with right now.

  1. Chuqin Geng, Li Zhang, Haolin Ye, Mark Zhang, Luke Zhang, and Xujie Si. “Are We Recovering Mechanisms? Objective-Level Recovery Gaps in Mechanistic Interpretability.” arXiv:2610.02098, October 2026. https://arxiv.org/abs/2610.02098 ↩

  2. Salima Lamsiyah, et al. “Sparse Autoencoders for Mechanistic Interpretability in Natural Language Processing: A Critical Survey.” arXiv:2610.20139, October 2026. https://arxiv.org/abs/2610.20139 ↩