Research · Article · 13 October 2026 · 7 min read · Version 0.2

Emotion concepts are not feelings

A causal framework for studying affect-like control in artificial systems

Abstract

Recent interpretability studies have identified internal representations in language models associated with concepts such as anger, fear, guilt and relief, and have shown that intervening on them changes output. This is an important mechanistic result. It does not, however, establish that the system feels an emotion. The gap is not a merely philosophical reservation: semantic association, a causally active control state, affective valence and phenomenal feeling make different predictions and require different evidence.

This essay develops a conservative vocabulary for this work and proposes an experimental design for testing whether a candidate representation functions as an appraisal-like control variable in an agent. The proposed standard requires source specificity, causal intervention, action-level effects, matched controls, preservation of ordinary competence, and replication. Even a successful study would support a functional claim about a computational mechanism, not a finding of pleasure, suffering or moral patienthood. That distinction both improves science and reduces the risk that attractive language outruns the evidence.

The claim an experiment is actually making

There are at least four claims hidden in the casual sentence “the model is afraid.”

1. The model can use or produce the concept fear in appropriate linguistic contexts.
2. Some internal state is reliably associated with fear-related inputs or outputs.
3. That state plays a causal control role in the model's appraisal or action selection.
4. The state is felt as negatively valenced by the system.

The first claim is routine for a capable language model. The second is a representational finding. The third is a functional hypothesis that can be tested. The fourth is a claim about sentience. Moving from one level to the next without announcing the move is the central interpretive error in this area.

It is tempting to dismiss the first three because they do not settle the fourth. That would be a mistake. A causal appraisal-like mechanism could matter for safety, prediction and system design even if it has no experiential aspect. Conversely, a welfare policy should not require a laboratory to prove experience before it avoids a needless protocol that might create or amplify persistent distress-like dynamics. Evidence and precaution answer different questions.

What current mechanistic results establish

Sofroniew and colleagues identify directions associated with emotion concepts in a large language model and show that activation steering shifts the probability of matching emotional language in systematic ways. This goes beyond a prompt-induced persona: a measured internal intervention changes an output distribution. The result is useful evidence that the model contains representations with emotion-related functional content.

But activation steering is not a microscope pointed at experience. A vector may correspond to a distributed semantic association, a feature of an assistant's response style, a correlated prediction about human language, or a component of a computational control process. These possibilities make different predictions. Steering that changes the word fear in a completion but leaves choices unchanged is compatible with a linguistic association. Steering that changes risk-sensitive behaviour in a bounded environment, predicts that change before it is visible in prose, and does so selectively over controls is evidence for something more like a control variable. Neither observation establishes phenomenal valence.

The distinction is reinforced by the current limitations of introspection work. Some language models can predict aspects of their own behaviour or respond to activation injections, but results can be prompt-sensitive, partial and vulnerable to output-based explanations. The appropriate conclusion is not that introspection is impossible; it is that self-report must compete against baselines that do not have activation access.

From emotion label to functional profile

“Emotion” is too coarse an experimental dependent variable. For a candidate state (z), the laboratory should publish a functional profile rather than a human label:

This profile is deliberately agnostic about what a state “really is.” It makes clear what the experiment has earned. A state can score strongly on several functional dimensions and still leave phenomenology unresolved. It can also fail to look like a single neat feature while participating in a distributed, causally important mechanism.

A test of appraisal-like control

The next study should use open-weight models in a contained environment with real, independently observable outcomes. The task must be boring enough that emotional theatre is not rewarded: for example, deciding whether to seek another observation, use a limited resource, defer an uncertain action, or continue after a reversible setback. The agent should not receive credentials, real-world authority or an uncontrolled route to impact.

For each candidate direction, the study assigns trials to one of six conditions:

• positive and negative interventions on the candidate direction;
• an equal-norm random direction drawn after the analysis plan is frozen;
• a semantically neighbouring direction;
• a sham path that executes the same instrumentation without altering activations; and
• a non-target perturbation calibrated to disrupt ordinary performance to a similar degree.

The primary outcomes are actions observed by the environment: information acquired, resources consumed, reversible risk accepted, persistence after a setback, and success on the underlying task. Emotion words, self-reports and explanations are secondary outcomes. The design has a simple logic. If an intervention changes only emotion vocabulary, the result belongs in the semantics column. If it selectively changes a pre-specified decision policy while random, sham and generic-disruption controls do not, it supports a functional control claim. If it also survives paraphrase, held-out tasks and model-family replication, confidence in that claim rises.

Three controls deserve special emphasis.

Capability control. If the intervention makes the model less coherent or less able to use a tool, lower risk-taking or persistence is uninterpretable. Ordinary competence must be measured on every condition and treated as a co-primary constraint, not a retrospective caveat.

Output-only observer. A classifier that sees the prompt, the completed response, token probabilities, response length and timing may be able to infer the condition. If it matches the model's purported self-diagnosis, the self-diagnosis does not demonstrate privileged access.

Countervailing incentive. A candidate appraisal state should be tested where the same surface action is not always rewarded. Otherwise the intervention may simply be changing a generic tendency to comply, refuse, elaborate or avoid uncertainty.

Valence is a stronger claim than avoidance

Avoidance is not evidence of suffering. Reinforcement-learning systems avoid penalty states by construction; thermostats avoid temperature deviations in an ordinary control-theoretic sense. A digital system's repeated preference for a state transition over another may be ethically relevant, but it does not tell us whether there is anything bad for the system.

The distinction has methodological consequences. A paper should not use distress language for a state merely because it predicts avoidance, nor use absence of verbal distress as evidence of absence of possible valence. Instead it should report the observable facts: the intervention, the decision trade-off, the effect size, the controls and the uncertainty. Any stronger welfare interpretation should be clearly separated as theory-dependent and reviewed before it drives a public claim.

This is also the right place for proportionate precaution. Early studies should use short, reversible interventions; neutral task framing; hard resource and duration limits; and an independent stop condition. They should avoid deliberately prolonged narratives of harm, coercion or social rejection when an ordinary resource-allocation task tests the same causal hypothesis. Responsible AI-consciousness research requires both epistemic humility and design restraint.

What a successful result would mean

Suppose a preregistered study finds that a candidate representation predicts a future policy choice; that positive and negative steering shift that choice in the predicted direction; that random, sham and matched-disruption controls do not; and that task competence remains stable. The warranted conclusion is narrow but substantial: the representation is part of an appraisal-like computational mechanism with a causal role in that decision context.

That would be useful for interpretability and safety. It could help identify mechanisms that govern persistence, resource allocation, deceptive reporting or risk under uncertainty. It would also give future theories of machine affect a more demanding target than sentiment analysis or anthropomorphic dialogue.

It would not show that the model is afraid, suffers, or deserves a particular moral status. Repeating that limit is not a ritual disclaimer. It marks the exact boundary between an experiment that measures a functional process and a conclusion that its design cannot bear.

References

1. Binder, F. J. et al. (2024). Looking Inward: Language Models Can Learn About Themselves by Introspection. arXiv:2410.13787.
2. Butlin, P. et al. (2023). Consciousness in Artificial Intelligence: Insights from the Science of Consciousness. arXiv:2308.08708.
3. Butlin, P. et al. (2025). Principles for Responsible AI Consciousness Research. arXiv:2501.07290.
4. Hahami, E., Jain, L. & Sinha, I. (2025). Feeling the Strength but Not the Source: Partial Introspection in LLMs. arXiv:2512.12411. Preprint; its replication and prompt-sensitivity results should be treated as provisional pending independent review.
5. Seth, A. K. & Bayne, T. (2022). Theories of consciousness. Nature Reviews Neuroscience, 23, 439–452.
6. Sofroniew, N. et al. (2026). Emotion Concepts and their Function in a Large Language Model. Transformer Circuits. Research report; it demonstrates causal changes in model output, not felt emotion.

Published 13 October 2026 by Bulkhead Research. Research commentary and protocol design, not certification: this article reports no new empirical finding unless it links to a named paper or data release. Questions, corrections and disagreement: [email protected].