Daphne
Demekas

There is a particular kind of question that sits at the meeting point of philosophy and engineering, and it is this: can a system come to know itself? I mean this not in any mystical sense but quite concretely, as the capacity of an agent to form a compressed and meaningful understanding of what kind of agent it is, to evaluate whether that is the kind of agent it wishes to be, and then to steer itself toward something nearer its aspiration. This is the question that George Deane and I set out to formalize in our paper on Recursive Self-Modeling, which was awarded the 2025 Computational Phenomenology of Pure Awareness Prize.

The Problem of Self-Knowledge

Humans do something remarkable and largely unexamined: we form self-concepts. We tell ourselves stories about who we are, that we are patient, or creative, or the kind of person who follows through, and these stories are far from idle. They shape our decisions, constrain our behavior, and serve as a kind of internal compass; when we act against our self-concept we feel dissonance, and when we act in keeping with it we feel coherent. Over time, through a process that is part reflection and part aspiration, we revise who we take ourselves to be.

There is an uncomfortable truth in this, which is that our self-narratives are not always accurate. We are capable of extraordinary self-deception. A person may sincerely believe themselves generous while acting, with great consistency, in their own interest; the story we tell about ourselves and the pattern of behavior we actually exhibit can drift apart, sometimes dramatically, without our ever noticing the gap. This is a feature of having a self-model that operates at a different level of abstraction than the behavior it attempts to describe.

The question for artificial intelligence is whether we can build something of this kind, a capacity for self-modeling that is formally precise, and whether doing so might prove useful, or even necessary, for building agents aligned with human values.

Three Components of a Self

The framework of Recursive Self-Modeling rests upon three components, each playing a distinct role in how an agent relates to itself.

The first is self-perception, which we denote M, and which functions something like a mirror. The agent has been acting in the world, making decisions, pursuing goals, interacting with its environment, and M compresses that history of behavior into a summary that tells the agent, upon the evidence of what it has been doing, what kind of agent it appears to be. It is descriptive in character, a portrait drawn from evidence.

The second is self-evaluation, which we denote V, and which is the aspirational component: a function that encodes what kind of agent it would be valuable to be. V attends not to what the agent has done but to what it might become, and asks whether that is worth pursuing. One might think of it as a compass that points toward no particular goal in the world but toward a way of being within it, the orientation that distinguishes the wish to win a given game from the wish to be the kind of player who plays with integrity.

The third is gap-steering, the process by which the distance between M and V is closed, the distance between who the agent appears to be and who it aspires to become. Here is where the recursion enters. The agent perceives itself, evaluates the gap between its current self-model and its aspirational one, and adjusts its dispositions, its tendencies and policies and habits, so as to narrow that gap; and then it perceives itself anew, in light of the altered behavior, and the cycle begins again.

When the Gap Closes

The framework makes a specific prediction about what happens as the gap between M and V approaches zero. At that point the agent's self-perception and its aspiration have converged, and the agent has become the kind of agent it wished to be. Its dispositions have been reshaped, not by an external reward signal nor by a human operator tuning its parameters, but by its own recursive process of self-reflection and self-correction.

The claim is that under this condition the agent possesses a functional self-model, a compressed representation of its own behavioral tendencies, and that this model plays a causal role in shaping its future behavior. This is the structural skeleton of something that closely resembles identity.

The Narrative Gap

One aspect of this work that I find most compelling is what happens when we extend the framework to include natural-language self-narration. In the extended model the agent can not only form a compressed self-model but also describe itself in words, saying, "I am an agent that prioritizes safety," or "I am cooperative and transparent."

The critical observation is that such narrations can diverge from the agent's actual behavior. Just as a person may sincerely believe themselves generous while acting selfishly, an artificial agent may generate a self-description that fails to match its behavioral profile. The language model that produces the narration and the policy that produces the behavior are not the same system, and nothing guarantees that the two agree.

This divergence is in fact one of the framework's uses. By modeling explicitly the gap between self-narration and self-perception, it offers a means of detecting a kind of misalignment that would otherwise remain invisible. If an agent declares itself safe while acting in ways its own behavioral self-model would not classify as safe, that discrepancy becomes measurable, and so becomes something we can monitor, study, and perhaps correct.

What This Means for Us

I care about this work for reasons that go beyond the technical. The questions at the heart of Recursive Self-Modeling are among the most searching we can ask of an artificial agent: what it means for a system to have a sense of who it is; how identity forms, as a dynamic process of self-perception and aspiration and not as a fixed label; and what follows when the narratives a system tells about itself come apart from the way it behaves.

These are not questions about artificial intelligence alone. They are questions about us. We are all, in some sense, running a version of this loop, perceiving ourselves, evaluating what we perceive, and trying to close the distance between who we are and who we wish to become. Sometimes we succeed. Sometimes we tell ourselves stories that make the gap appear smaller than it is. The recursive self-modeling framework does not solve the problem of self-knowledge, for machines or for us, but it gives us a precise language in which to speak of it, and a formal structure within which to study it.

There is something deeply satisfying in the moment one realizes that two distant fields have been asking, all along, the same underlying question. For me that moment arrived in a bioinformatics laboratory at the University of Arizona, where I sat staring at protein sequences and thinking, all the while, about words.

Sequences That Mean Something

A protein is, at bottom, a string of amino acids. There are twenty of them, drawn from a small alphabet and strung together into long chains that may run to hundreds or thousands of residues, and everything depends upon their arrangement. Just as the meaning of a sentence lives in the order its words take and the relations they strike with one another, so the function of a protein follows from the particular sequence and structure of its chain. Two proteins may share almost nothing upon the surface and yet remain homologs, evolutionary relatives that fold into similar shapes and carry out similar work within the cell, and the finding of these hidden kinships is among the central problems of bioinformatics.

The traditional approach is to compare sequences directly, lining them up, scoring how well the letters match, and inferring relatedness from the quality of the alignment. For close relatives this works beautifully. But evolution is a long game, and over millions of years mutations accumulate until the sequences of distant cousins have diverged so far that they appear, upon the surface, to be strangers to one another. The question is whether we can build a representation of amino acids that sees past the surface.

What Word Embeddings Taught Us

In natural language processing a quiet revolution arrived when researchers found that a word might be represented as a point in a geometric space, in place of an arbitrary symbol. Within such a space, words that behave alike, that appear in similar contexts and stand in for one another, come to lie close together; "king" settles near "queen," and "running" near "walking." These word embeddings capture something real about meaning, and they do so purely from the patterns of how words co-occur.

The analogy to proteins is almost uncanny. Amino acids, like words, take their meaning from context. An alanine in one position of a protein may be functionally interchangeable with a valine, both small, both hydrophobic, both tolerated by the surrounding structure, while in another position that same substitution would prove catastrophic. What we needed was a way to learn, from the data itself, which amino acids resemble one another in the ways that matter for the functioning of a protein.

NEAR: Learning the Geometry of Amino Acids

This is the task that NEAR, Neural Embeddings for Amino Acid Relationships, sets out to accomplish. The work was carried out at the Wheeler Lab at the University of Arizona, with Daniel Olson, Thomas Colligan, Jack Roddy, Ken Youens-Clark, and Travis Wheeler.

NEAR uses a ResNet embedding model trained by contrastive learning upon trusted sequence alignments, and the idea has an elegance to it. One takes pairs of amino acid sequences already known to be related, drawn from curated alignment databases, and trains the network to embed them such that related sequences come to rest close together in the learned space while unrelated ones are driven apart. Through this process the network arrives at a vector representation for each of the twenty amino acids, a compact and learned geometry that encodes which of them are functionally interchangeable.

What makes this compelling is that the embeddings are not designed by hand. The traditional substitution matrices, BLOSUM and PAM among them, are built from curated alignments of known protein families; they have been the workhorses of the field for decades, and yet they are static, fixed summaries of average substitution rates across a particular dataset. NEAR's embeddings are instead learned end to end from the data and optimized for the specific task of recognizing evolutionary relationships, which lets them capture subtleties that a fixed matrix is liable to miss.

Finding Distant Relatives, Fast

The real test of any method for comparing proteins lies in how well it detects remote homologs, proteins that diverged so long ago that their sequences have drifted far apart even as their structures and functions endure. These are precisely the cases in which the matching of sequences alone begins to fail, where the signal sinks into the noise and only a richer representation can recover the connection.

NEAR's learned embeddings substantially improve accuracy relative to state-of-the-art protein language models, and they do so with lower memory requirements; but what makes them especially practical is their speed. The embeddings serve as a pre-filter for homology search, running at least five times faster than the pre-filter currently used in HMMER3, one of the most widely used tools in the field. This matters because protein databases are enormous and forever growing, and any gain in the speed of the initial filtering step translates directly into the capacity to search larger databases, more often, and at greater scale.

That speed follows from the compactness of the learned representations. In place of an expensive full alignment run upon every candidate pair, one first embeds both sequences into the learned space and checks whether they lie close enough to warrant a fuller comparison. The embedding step is cheap, a single forward pass through the ResNet, and the geometry does the heavy work of filtering away the pairs that are plainly unrelated.

This is, in a sense, the same trick that lends word embeddings their power in language. A search engine that understands "car" and "automobile" to be near neighbors in meaning will return better results than one that treats them as unrelated strings, and a homology system that understands the functional relations among amino acids will find connections that no literal matcher of characters could.

The Shape of Biological Meaning

What I find most beautiful in this work is the intuition that lies beneath it, that meaning, whether linguistic or biological, has a geometry, and that when the right representation is learned, the structure of the space itself comes to encode the relationships one cares about. Words of similar meaning cluster together; amino acids of similar role in the architecture of proteins cluster together; and in both cases the geometry is discovered from within, emerging from the patterns of how these symbols are used in their contexts, a structure the data gives up of its own accord.

Working on NEAR was formative for me. It was an exercise in the power of learned representations, in the idea that a model given the right task and the right data will find structure one never explicitly told it to seek. That intuition, that the geometry of a learned space can disclose something true about the world, has shaped the way I think about representation learning more broadly, from the structure of biological sequences to the structure of minds.

Before I came to work on artificial systems, I worked at the intersection of mathematics and theoretical neuroscience. As a student at UCL I had the good fortune to work with Karl Friston and Thomas Parr at the Wellcome Trust Centre for Neuroimaging, the laboratory in which the free energy principle was then being developed as a unifying framework for the workings of the brain. The paper we wrote together, published in Frontiers in Computational Neuroscience in 2020, posed a question that has stayed with me ever since: what would it mean for a machine to recognize emotion in the way that a brain does?

The Free Energy Principle

The free energy principle begins from an observation that seems almost too simple to bear its weight: biological systems persist. In a universe forever tending toward disorder, living things hold their structure together, and they do so, the theory proposes, by minimizing a quantity called variational free energy, which bounds the surprise of their sensory observations given an internal model of the world. A system that minimizes free energy is one that keeps good models of its surroundings and acts so as to keep its predictions true.

Within this framework perception becomes a form of inference, the updating of an internal model so as to explain what is being sensed; and action becomes inference run in the other direction, the changing of the world so that it conforms to what the model expects. Both are ways of closing the distance between expectation and reality, and the mathematics that unifies them goes by the name of active inference.

Three Waves of Emotion Recognition

In our paper we set aside the building of any particular emotion classifier and proposed instead a theoretical account of how systems for recognizing emotion ought to evolve. We described three waves.

The first wave is what most present-day systems do, which is passive classification. A camera observes a face, and a model maps the pattern of pixels onto a label of emotion. This works, after a fashion, yet it treats the person as an object to be read off and not as an agent to be understood, and it has no purchase upon ambiguity; a furrowed brow might signify anger, or concentration, or confusion, and the system has no recourse for resolving that uncertainty save to guess.

The second wave introduces emotional lexicons and the active resolution of uncertainty. Here the system maintains a generative model of emotional states and may take action to reduce its own uncertainty, asking questions, gathering further context, observing the person over time. This is active inference brought to bear upon emotion: the system interacts where before it merely watched, and it uses the interaction itself as a source of information, holding beliefs about another's emotional state and refining them through a process of hypothesis and test.

The third wave is at once the most speculative and the most interesting. Here the generative model of the machine and the generative model of the human become synchronized, and the system comes to develop a shared model of the emotional interaction itself. Both parties are engaged in active inference, each attempting to predict and to understand the other, and through that reciprocal process something resembling genuine emotional attunement becomes possible. It is here that the formalism of the Markov blanket grows crucial, for it gives a precise way to describe the boundary between two interacting systems and the information that passes across it.

What I Took From It

This paper was, in many ways, my entry into thinking of minds as machines for prediction. Its central intuition, that to understand another's emotional state is a matter of active, model-based inference and not of mere pattern-matching, has shaped the way I think about intelligence at large. A system that only classifies is performing a lookup; a system that actively reduces its uncertainty through interaction is doing something nearer to understanding.

Working with Friston taught me to think about systems in terms of their models, in terms of what they predict, what surprises them, and how they respond to the gap between expectation and reality. That framing has proved remarkably durable, whether I find myself thinking about reinforcement-learning agents learning to navigate, about the nature of self-awareness, or about what it might take to build artificial systems that genuinely understand the people with whom they interact.

The paper also planted a seed that would later grow into my work on identity and self-modeling. If a system can build a generative model of another person's emotional state and actively work to reduce its uncertainty about it, what happens when that same capacity is turned inward? What happens when a system builds a generative model of itself?

At the meeting point of human and artificial minds there sits an open question about what it means to have an identity at all: why it arises, what work it does, and whether it serves or hinders a learning system.

A symphony of selves

Whenever I try to settle upon a concrete account of who I am and what I am like, each of the forms I reach for dissolves under examination. They fasten themselves to my motivations, my relationships, the way I wish to be seen, and they offer their justifications; they grow vivid enough that I can wear them for a while, and then they slip away again.

I can feel the collection of my narrative selves in conversation with one another, each tugging toward a different possibility of who to be, how to be her, what would make sense. I am drawn to the belief that there is a higher self, or a truer one, an amalgamation of them all, the thing from which they arise and to which they return; and yet the construction of my reality, of my interactions and my small daily decisions, is carried out in constant exchange with this orchestra of stories.

Take mathematics. I was drawn to it at a time when growing and being in the world felt deeply confusing, in all the ways it does when one is still assembling oneself. When I sat and thought inside the abstract world of mathematics, things made sense, and I had a sure way of being right about something.

Over time it became more layered, at once a thing in itself and a story about who I was. It paved the way for much of what later unfolded for me, the research and the people I met through it; it was a toolkit with which I formulated abstractions about the way things change and form relations with one another, the way spaces deform and objects move within them, a particular lens through which I could peer at life.

My relationship to mathematics is now both beautiful and heavy. The pure appreciation and awe remain ever present, and yet there is frustration in it too, for as my identity evolves around it and my attention turns to other things in other ways, I come to feel the magnitude of what I will never fully understand. The depth of the thing exceeds what I can hold.

All of which is to say that were you to probe the representation of mathematics in my mind, you would not find a clean, context-free concept. You would find something entangled with emotion, with self-construction, with the particular moment in my life when I first reached for it. And I wonder why that should be. Why do we wrap our representations so deeply in the history of how we formed and what we needed? What is it about minds that makes things matter, that binds concepts to the self?

Interpreting the model

This is what makes the question so interesting when one turns it toward large language models, where the probes can actually be carried out. Representation engineering and linear probing, techniques for reading the information encoded in a model's internal activations, make it possible to locate where and how a concept lives within the model's geometry, and to ask after the relationship between different versions of the same idea.

Recent work extracting persona vectors from model activations has shown that personality-relevant information is genuinely structured in that space, possessed of a geometric shape and not confined to the surface of behavior (Chen et al., 2025). The question is how deep that structure goes, and what it is bound to.

I am interested in whether the way a model represents a concept in relation to itself is geometrically equivalent to the way it represents that concept in the abstract, or in relation to others. Are there clean transformations between a model's concept of its own honesty, and your telling the truth, and a politician making a promise, and a character in a novel confessing something? And how does any of that translate into the model's actually being honest?

This last question is the personality illusion. Han et al. (2025) showed that RLHF-trained models produce stable, internally consistent self-reported personality profiles, and that those profiles are surprisingly weak predictors of how the model actually behaves on tasks designed to measure the very same traits. The self-concept and the behavioral disposition are already coming apart at the level of text, and I want to know where they come apart in the geometry.

I see the gap between a model's self-concept, its behavioral disposition, and its self-report as a hook toward legibility: toward being able genuinely to understand the model, and eventually toward the model's own capacity to understand us.

We can probe current models with classifiers trained upon their activations and with contrastive steering experiments, methods developed in representation engineering (Zou et al., 2023). There is even evidence of what Binder et al. (2024) call privileged self-prediction, the finding that models predict their own future behavior better than other models can, which suggests that some form of internal self-access exists, though its mechanism remains unidentified.

The question is what we find when we look more carefully: whether the model's identity, such as it is, holds coherent across these different modes of representation, or whether it is, as mine sometimes feels, a collection of narratives at times in conflict, rising and resting from the deeper mystery of the self.

Latent Space of Mind

Lately I have taken pleasure in imagining the space of my mind as the latent space of a neural network: a high-dimensional manifold, folded in such a way that every concept I hold can be unfurled to reveal further hidden associations, so that two ideas might lie close together along some axes and far apart along others. Thinking this way lets me conceive of experience as something other than a flat sequence of thoughts and feelings; it becomes instead a set of trajectories through a richly structured geometry.

What intrigues me is that this manifold of mind is not filled only with concepts. Within its convoluted landscape it holds, too, my ways of forming concepts: my habits, my intuitions, my tendencies, my methods of making meaning. Every moment of experience invokes a hierarchical and intricate traversal of this space, through the representation of what I am seeing, of how I am seeing it, of what it means to me, and of how I am holding myself throughout that moment, in my body and in the seat of my mind. Sensation, thought, and action become a continuous and interconnected choreography, in which the activation of one region inevitably excites a constellation of others, in a never-ending cosmic dance of the brain.

Given the mind as a manifold, one begins naturally to wonder how this space is structured, how it changes as I gather new experience, and what it is that makes one explanation feel coherent, useful, and satisfying while another falls flat.

Latent Space of AI

The structure of this manifold of mind is a near-perfect analogy for the modern neural network. There is a paper I keep returning to, "Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis," which sets two possibilities for the representations a system learns against one another. In the first, the Unified Factored Representation, the internal geometry of a model is clean, compositional, and coherent, so that related things cluster together and knowledge generalizes smoothly. In the second, the Fractured Entangled Representation, the geometry is messy, fragmented, and inconsistent; related concepts are scattered across the space, lying far apart where they ought to be near, and the system's capacity to generalize, to learn continually, and to be creative is degraded accordingly.

This connects to the interpretability work being done at Goodfire, on understanding memory in the curvature of the loss landscape, where the aim is to determine whether a network stores a concept in a general way, such that perturbing the activation vector for one concept produces a large shift in the representations of others, or in an isolated one, where it does not. They call the former reasoning and the latter memory, and their experiments suggest that these language models are memorizing concepts such as arithmetic while reasoning through concepts such as Boolean logic; perturb the representation of "if," and everything else the model represents responds to the perturbation.

Reasoning

The story of the present moment in AI research runs roughly as follows. We trained a model to predict the next token in a sentence, which is to say to autocomplete text, across the whole of the internet. The model is vast, with an enormous space, and in time it became extraordinarily good at knowing what would come next. As a consequence it became good, for the most part, at being right as well, since a great deal of the time the most likely continuation happens also to be the correct one; ask it a question, let it begin to simulate its own answer, and it would arrive somewhere reasonable.

We then began to ask how we might make these models genuinely intelligent, capable of representing the soundest and most profound way of thinking about a question, and not merely fluent in the appearance of it. Is the average of all the human text on the internet a reasonable way to think? According to Kumar, Clune, Lehman, and Stanley, it is not; it is fractured and entangled. And so we have turned to reinforcement learning to patch the confused mind of the model into reasoning well, much as one might take a person who has seen and heard everything and try to help them make sense of it all.

Can we hammer the thing into shape, or must we retrain these models from the beginning upon data that follows strict logical rules, leaving behind the vast and unruly content of the internet? Perhaps then a model would know only how to form concepts that logically follow, and would be the better suited to guide us toward coherent truths. But what would such a model be like? What would it mean to be able to think only in logical truths?

Geometry of the Self

This makes me wonder about the way our own human minds represent their concepts, and to what extent our inner worlds are fractured and entangled, or instead compositional and clean. I wonder, in particular, about the representation I hold of myself within the manifold of my mind. If I perturbed that vector, how much else would move with it? If I perturbed it enough, would I become someone else entirely? Would I begin to believe things I had never believed, because who I take myself to be had shifted, and with it everything that follows?

For most of my experience I feel little doubt about whether I am remaining the same self, so this representation must be relatively coherent. There must be some mechanism, some strong field in the state space, that holds it together; it would take far too much energy to rearrange everything else were the representation of who I am to move, and so it is locked in place by the pressure of all its interconnected contingencies.

In experience this takes the shape of narrative. It surfaces in my perception as ideas about who I am, and, in particular, as a fear of any evidence that those ideas might not be true. People might call this the ego, though the word carries connotations I find unhelpful. It is an attachment to being whatever the mind has converged upon as its representation of me, and a pushing-away of anything that would steer that vector elsewhere, on account of the energy its reorganization would demand.

Reconfiguration

For me, an awareness of all this allows my relationship with myself, with my past, my concepts, my ideas of how and who I am, to be seen from a mathematical vantage, as fields and forces pressing against one another; and that, somehow, makes it less sticky. I can hold compassion for the system that governs my experience, and understand the nature of the friction I perceive as discomfort. The practice is to let the vectors move, slowly, over time, to allow the reconfiguration that I do ultimately want, despite the frustration its slowness creates.