Daphne
Demekas

Mind at Large is a start-up designing human-AI interactions steered towards human-wellbeing through broader and deeper understanding of minds. Inherent in this pursuit is the need to answer the question: when we try to measure whether a person is well, or whether they are becoming more well, what exactly are we observing? And how would we tell the difference between a person who has experienced a 'true' positive shift, and a self-deceived person, who is narrativizing growth, while avoiding a core truth they are afraid to encounter?

This essay is an attempt to pin that question down. Every way we have of observing wellbeing is a framework, and every framework shapes what it observes. This is a genuine measurement problem, as fundamental as the one in physics; and the most common way people go astray is by mistaking the framework for the solution. Contemplative traditions have known this for a very long time, and mathematics has a precise language for it. Nowadays, we can also ask a new question: whether an artificial mind, which has no livelihood or lineage or reputation tied to any one framework, might be better placed than we are to steer us on the path.

The measurement problem of wellbeing

While a physical measuring device disturbs the very system that it measures, a conceptual framework for understanding experience functions all-the-more intimately: it not only explains experience, but it also steers experience, and thus the framework can become the lens through which we see our own mind. This means that the framework can become almost perfectly explanatory without ever needing to be true.

Take vocabulary: you can only report what you have words for, and once you have the words, you begin to experience through them. A person who has learned the language of attachment theory will begin to notice anxious and avoidant moves in their relationships that were, in some sense, always there, and that are also, in some sense, newly constructed by the noticing. A meditator who has been taught to look for the arising and passing of sensations will find them, and will find them more vividly the more they are told to look. This is not a defect of any particular framework, but it means that the report and the thing reported are never fully independent.

The philosopher Ian Hacking termed looping effects: when you call someone depressed, or highly sensitive, the person knows they have been classified, and the classification changes them: how they understand their past, what they expect of their future, how they behave in the presence of the people who classified them. The category then has to be revised to describe the changed people, and the loop continues. Human kinds, Hacking argued, are moving targets, and part of what moves them is our attempt to describe them.

When a formulation of our experience fits, something shifts. There is relief, or release, or a sense of yes, that's it. Eugene Gendlin built an entire practice, Focusing, around this felt shift, and while there does seem to be something real there, I think that the feeling of insight turns out to be detachable from truth. Laukkonen and colleagues showed in 2020 that artificially induced aha moments make unrelated statements feel more true; Grimmer and colleagues later used semantic priming to produce confident insights into things that were simply false. Interoception, our sense of the body from the inside, appears to be an act of inference rather than a direct readout, and Sarah Garfinkel's work suggests that how accurately people perceive their bodily signals and how confident they feel about those perceptions are largely unrelated. The body talks back, but it talks back through the same predictive machinery that the framework has been training.

So the natural move, to check the framework against the raw experience, runs into a regress. There is no framework-free experience to check against. Wilfrid Sellars called the hope for one "the myth of the given." Kant's version was that we never reach the thing in itself, only things as they appear to us through the forms of our own understanding. Buddhist psychology says something similar in its own terms: perception is already shaped by saṅkhāra, the conditioned formations of the mind, before it ever reaches the level of report.

I do not think this means that nothing can be known about inner life. I think it means the question has to change. It is not how do we escape frames?, because we cannot. It is how do we relate to them?

Every map becomes the territory

The hedonic tradition measures moment-to-moment feeling: how good or bad things are, right now, sampled throughout the day. It seems as close to the raw data as one could get. But Daniel Kahneman's work on the experiencing self and the remembering self showed that these two answer differently. In his studies of painful medical procedures, people's retrospective judgments tracked the peak and the end of the experience and largely ignored its duration, so that a longer procedure with a gentler ending was remembered as less bad than a shorter one. Which self is the one whose wellbeing we are measuring? The question does not have an answer that is independent of how we ask it.

Life satisfaction measures, the ladder from the worst possible life to the best, ask people to evaluate their lives as a whole. The answers are meaningful and they predict things. They also shift with mood, with the weather, with the question that was asked immediately before, and above all with the reference frame the person brings: the best possible life relative to whom?

Eudaimonic frameworks, which measure meaning, purpose, autonomy, and growth, are richer, and they import a particular theory of what a good human life consists of. Clinical frameworks define wellbeing largely by the absence of symptoms, so that to be well is to be not sick. Physiological frameworks measure heart rate variability, cortisol, sleep, and brain activity, and they are vulnerable to a quieter version of the same trap, in which a correlate of wellbeing is gradually treated as if it were wellbeing, until high heart rate variability is simply what being well means. Contemplative stage models describe the territory of a path in exquisite detail, and practitioners who learn the map begin, inevitably, to report the map. The wellness industry measures whatever is trackable, and the trackable quietly crowds out everything else.

Any framework begins as a lens, invented in order to sort and speak about experience. Often, the lens eventually hardens into the thing it was looking at, and the framework begins to seal itself against correction: experiences that do not fit are reinterpreted as resistance, or immaturity, or not doing the practice properly, so that every disconfirmation becomes further confirmation. The philosopher of science Imre Lakatos had a name for research programmes in this state. He called them degenerating: they no longer predict anything new, and they survive by adding patches to explain away whatever goes wrong. And finally, once the measure is taken seriously enough to be optimized, it stops measuring, which is Goodhart's law, or in Marilyn Strathern's formulation, when a measure becomes a target, it ceases to be a good measure.

The finger and the moon

The most famous image is from the Śūraṅgama Sūtra, though it has become common property of Zen: a teaching is like a finger pointing at the moon. If you look at the finger, you miss the moon. Worse, you might come to think the finger is the moon. In the Alagaddūpama Sutta, the Buddha compares his own teaching to a raft built to cross a river. Once you are across, he asks, would it be wise to hoist the raft onto your head and carry it with you wherever you go? The teaching is "for crossing over, not for holding onto."

Nāgārjuna, the great philosopher of emptiness, pushed this to its limit. Emptiness, he said, is the medicine that cures attachment to all views; but those who take emptiness itself as a view are incurable. The medicine becomes the disease if it is swallowed as doctrine. The Theravāda account of the first stage of awakening, stream-entry, makes the same point structurally. Of the three fetters that fall away at that stage, one is sīlabbata-parāmāsa, clinging to rites and rituals, which is to say treating the method as if it were itself the liberation. The very first sign of real progress includes no longer mistaking the raft for the far shore.

The tradition also knew that the feeling of progress is unreliable, and said so with remarkable precision. Buddhaghosa's Visuddhimagga, a fifth-century manual of meditation, lists ten corruptions of insight: brilliant light, rapture, tranquillity, bliss, strong faith, energy, a sense of sudden knowing, and so on. They are real experiences, and they arise precisely when practice is going well. The danger is that the practitioner takes them for the goal, and the tenth corruption is exactly that: a subtle attachment to the other nine. The feeling of breakthrough is named, explicitly, as a trap. Linji, the ninth-century Chan master, put the same point more violently: if you meet the Buddha, kill him. Whatever image of awakening you have arrived at, it is not the thing.

The Daoists had their own version. The Dao De Jing opens by saying that the Dao that can be spoken is not the eternal Dao. Zhuangzi offers my favorite formulation of the whole problem: the fish trap exists for the fish, and once you have caught the fish you can forget the trap; words exist for meaning, and once you have the meaning you can forget the words. Where, he asks, can I find a person who has forgotten words, so that I can have a word with them?

In Advaita Vedānta, the method of adhyāropa-apavāda describes teaching as a two-step: first a provisional framework is superimposed, then it is withdrawn once it has done its work. Ramakrishna's image is of using one thorn to dig out another, and then throwing both away. The Jain doctrine of anekāntavāda, many-sidedness, holds that every view of reality is partial, which is where the parable of the blind men and the elephant comes from: each one is right about the part he touches and wrong about the whole. In the Sufi tradition, Ibn ʿArabī distinguished the God created in belief, the God each person constructs from their own creed, from God as such, and borrowed from al-Junayd the line that water takes on the color of the vessel that holds it. Meister Eckhart, in one of his most startling sermons, prayed: I pray God to rid me of God.

Western philosophy arrives at the same place by different routes. Wittgenstein ends the Tractatus by telling the reader that his propositions are a ladder, to be thrown away once climbed; later he described the aim of philosophy as showing the fly the way out of the fly-bottle. Korzybski coined the phrase the map is not the territory. The statistician George Box observed that all models are wrong, but some are useful. William James, in The Varieties of Religious Experience, judged religious frameworks not by the truth of their metaphysics but by their fruits.

And Nietzsche, in the Genealogy of Morals, wrote a passage that I think contains the seed of something more constructive:

There is only a perspective seeing, only a perspective "knowing"; and the more affects we allow to speak about one thing, the more eyes, different eyes, we can use to observe one thing, the more complete will our "concept" of this thing, our "objectivity," be.

Psychotherapy research has its own uncomfortable version of the finger and the moon. Jerome Frank, in Persuasion and Healing, argued that every healing tradition, from shamanism to psychoanalysis, offers the sufferer a myth, a plausible explanation of their suffering, and a ritual, a procedure the healer and the sufferer perform together in light of that myth. What heals, on his account, is less the truth of any particular myth than the shared, believable framework itself, delivered in a relationship of trust. Decades of research on the common factors in psychotherapy, associated especially with Bruce Wampold, have broadly supported the view that the specific theory matters less than the alliance and the believability. This is a strange finding to sit with. It suggests that the finger, at least sometimes, is doing the work, and that what it points at may matter less than we would like.

What these traditions share, I think, is a stance rather than a doctrine. The framework is necessary: no one crosses the river by refusing to build a raft. The framework is provisional: it is for crossing, not for holding. And the framework is meant to make itself unnecessary: a good teaching leaves the student more able to see for themselves, not more dependent on the teaching's vocabulary.

The guide's attachment, and the necessity of mirrors

Almost always, we encounter frameworks through people: teachers, therapists, facilitators, retreat leaders, friends who have found something that helped them. And the people who guide us through a framework are very often attached to it themselves. The framework may have saved them, and the zeal of the convert is real. Their livelihood may depend on it; they have spent years and money on certification in one modality, and their professional identity is built on it. Their community, their lineage, their sense of belonging are organized around it. And perhaps most of all, holding a framework lightly means facing not-knowing while someone is suffering in front of you, which is frightening, and a framework one believes in completely is an enormous comfort in that moment.

Chögyam Trungpa had a name for what happens when the ego takes over a spiritual path: spiritual materialism. The practices, the experiences, the insights, the progress through the stages, all become possessions, things the self acquires to shore itself up. The irony is exact. The path was supposed to loosen the self's grip, and the self has simply found a more refined object to grip. Guides are not immune to this. A teacher's identity can become fused with the teaching, so that a student's doubt about the framework registers as a threat to the teacher's self.

The Buddhist tradition has a precise diagnosis of what is missing when this happens. Upāya, skillful means, is the capacity to offer each person the teaching that will help them, even teachings that are only provisionally true. The Lotus Sutra's parable of the burning house takes this to an uncomfortable extreme: a father lures his children out of a fire by promising them carts that do not exist. But the tradition is clear that upāya is only legitimate when it arises from prajñā, wisdom, the direct seeing of what the teaching points to. Skillful means without wisdom is just manipulation. A guide can hold the raft lightly only if they have crossed the river themselves and know that the raft is not the shore. The guide who has not crossed, and who is attached to the raft because it is all they have, cannot help but steer their students toward the raft.

This dynamic can lead to the student's experience being quietly corrected to fit the map, where failures are reframed as resistance. The student's reports converge, over months, on the teacher's vocabulary, and the convergence is taken as evidence of progress. At the extreme end, this produced the recovered-memory therapies of the 1980s and 1990s, in which patients and therapists sharing a framework co-constructed vivid, emotionally overwhelming memories of events that never happened. It produces the familiar pattern of guru abuse, where a student's perception that something is wrong is reinterpreted, by the framework, as the student's own unripeness.

Jiddu Krishnamurti, raised from childhood by the Theosophical Society to be a world teacher, with an organization built around him, the Order of the Star in the East, dissolved the society in 1929 in front of thousands of its members. Truth is a pathless land, he said, and no organization could lead anyone to it.

While the identity and attachments of others can create constraints to our own growth, having people in 'sangha' to serve as mirrors is crucial to advancing into deeper truth. When I introspect, the experience and the report about it are produced by the same system, so their errors are correlated, and checking myself against myself is like asking a ruler to measure its own length. Another person can see the frame I am looking through, which from the inside is invisible. They collect evidence I cannot see objectively: tone, expression, the gap between what one says and how one says it.

In Gendlin's original research, the rating of how deeply a client was in contact with their experience was done not by the client but by trained listeners hearing recordings of the sessions. In a study whose title says it all, Getting what you want by revising what you had, Michael Conway and Michael Ross found that students who took a study-skills course misremembered their earlier skills as worse than they had been, so that they felt they had improved when, by objective measures, they had not. Our sense of our own progress is quietly rewritten by our current frame.

In addition, another person can ask the question we would never ask ourselves, knocking us out of the grooves that self-inquiry tends to follow, which is why the most careful methods for exploring experience, Claire Petitmengin's micro-phenomenological interview and Russell Hurlburt's descriptive experience sampling, both depend on an interviewer. Hurlburt's work in particular keeps finding that people's confident beliefs about their own experience are wrong: many people who are sure they think in words turn out, when sampled at random moments, to have very little inner speech at all.

Tibetan mind training has a slogan for this: of the two witnesses, hold the principal one. Others' view of you is one witness; your own honest knowledge of your mind is the principal one.

So the paradox is this: other people's egos are a primary cause of our falling into the trap of attachment to a framework, and other people are also the only way out of our own blind spots. The second witness helps exactly to the extent that their errors are independent of ours. What we need is not the absence of other minds, nor the presence of any other mind, but other minds that see differently. Nietzsche's more eyes, different eyes.

What survives a change of frame

Mathematics has spent a long time thinking about how to know something that can only ever be seen through a frame.

Start with the simplest possible case: a sphere. It is a mathematical fact, going back to Gauss, that there is no way to draw the surface of a sphere on a flat page without distortion. Every flat map of the earth is wrong somewhere. Cartographers deal with this by choosing what to preserve. A stereographic projection keeps every angle true, so that small shapes look right, but it inflates areas far from its center. A gnomonic projection keeps the shortest paths between points as straight lines, which is why navigators valued it, but it tears the world apart toward its edges. An equidistant projection keeps distances from its center true and bends everything else. Each projection is faithful to something and unfaithful to something else, and each is only trustworthy near its own center.

Mathematicians call a collection of such local maps an atlas, and the sphere itself is, in a precise sense, nothing more than the atlas together with the rules for translating between overlapping charts. I find this a helpful way to think about frameworks for inner life: each is genuinely accurate near the experiences it was built around. Each distorts, increasingly, as it is stretched to cover more of a life. The mistake is believing one chart could cover everything, and then reading the distortions at its edges as facts about the territory.

If every chart distorts, what is real? Physics and geometry give the same answer: what stays the same when you change charts. Imagine three observers, each with their own coordinate system, describing the same three points. They will give completely different coordinates for where each point is. They are not wrong; they are each describing the world from somewhere. But if you ask them how far apart two points are, or what angle the triangle makes at one corner, they will all give the same answer. Those quantities are invariant. In gauge theory this becomes a definition: anything that changes when you merely change the description is not physically real. It belongs to the description.

This is Nietzsche's more eyes, made exact. The point of many frames is not to average them, and not to find the one true frame, but to find what they all agree on despite disagreeing about everything else. The coordinates are the finger. The invariants are as close as we get to the moon.

But there is a twist, and it matters. In 1872 the mathematician Felix Klein proposed what became known as the Erlangen program, which reorganized all of geometry around a single question: which transformations do you count as merely a change of perspective? If you allow only rotations and translations, you get Euclidean geometry, in which lengths and angles are real. If you also allow changes of scale, lengths stop being real but angles survive. Allow stretching and shearing and angles go too, but parallel lines remain parallel. Allow the full range of perspective, the transformations a painter uses, and even parallelism disappears; what survives is only straightness, and a strange quantity called the cross-ratio.

I think this is the most honest thing mathematics has to say about the measurement problem, and it echoes Madhyamaka almost exactly. There is no final, frame-free ground. Even our invariants depend on a choice made one level up, about which differences we are willing to treat as merely differences of description. A tradition that treats all thoughts as empty mental events will find a different set of things invariant than one that treats the body's sensations as the ground of truth. But notice that this is not the same as arbitrariness. Once the choice of transformations is made, the invariants are fixed, and they are not up to anyone. Mutual constraint is not the same as anything goes.

Suppose a framework takes an experience and produces a reading of it, and suppose that reading then becomes part of the next experience, which is read again through the same framework. This is a map applied to its own output, over and over, and mathematicians know what such iterations do: they settle into fixed points, states that reproduce themselves. If the framework is held tightly, so that the reading depends steeply on how the experience was already being read, there can be more than one fixed point, and which one you end up in depends almost entirely on where you started. Two people with the same experience, beginning with a slightly different lean, will end up confidently in two different places, each of their readings now self-confirming. Loosen the grip, or let the experience itself push back a little more strongly, and the two fixed points merge into one, near what is actually there.

In category theory there is a result called the Yoneda lemma, which says, roughly, that an object is completely determined by the totality of its relationships with every other object: by how everything else sees it. You never need access to the object in itself. If you know all the ways it can be probed, you know it, up to the only kind of sameness that matters. A concrete cousin of this idea is tomography, the mathematics behind CT scanning. You cannot see inside the body directly; each X-ray from a single direction gives only a shadow, and a single shadow is consistent with countless different interiors. But enough shadows, from enough different directions, determine the interior. Johann Radon proved this in 1917, long before anyone could build the machine.

This is the paradox of mirrors in geometric form. A single witness gives a shadow consistent with many different truths. Many witnesses who share a frame, a lineage of teachers and students who see from the same angle, can agree perfectly and still leave the thing smeared along exactly the direction none of them can see. What resolves the hidden thing is not the number of witnesses but their diversity: the different angles, whose shadows disagree in ways that only one shape can reconcile.

The last idea is about what disagreement means. Sheaf theory, a branch of mathematics built for exactly this situation, asks: given many local pictures, each consistent with its neighbors where they overlap, can they be glued into a single global picture? Sometimes the answer is no. Every local view agrees with every adjacent one, and yet there is no single object that all of them are views of. Sheaf cohomology is the tool that measures this obstruction. Samson Abramsky and Adam Brandenburger have shown that quantum contextuality, the fact that a quantum system has no values independent of which compatible questions you ask of it together, is precisely an obstruction of this kind. And Roger Penrose, in a paper with the title On the cohomology of impossible figures, showed that his famous impossible triangle is one too: each corner, taken alone, is a perfectly possible object, and the impossibility lives only in the whole.

Sometimes the many witnesses do not converge, and the right response is not to force them to, or to choose the most authoritative one, but to take the disagreement itself as information. If what a person says about their experience, what their body does, and what the people who love them observe all refuse to fit together, that is not a failure of measurement. It may be the most accurate thing the measurement can tell us: that there is no single, frame-independent story here yet, or that one of the frames is being held from exactly the one vantage point at which an impossible thing looks whole.

Put together, each framework for inner life is a chart, faithful near its center and distorting at its edges. What is real is what stays invariant across charts, though even that depends on what we have agreed to treat as a mere change of perspective. Frameworks held too tightly settle into self-confirming fixed points. A hidden thing can be determined by its shadows, but only if the witnesses see from genuinely different angles. And where the local pictures cannot be glued, the obstruction is information, not error.

Can a machine hold the raft lightly?

If the deepest problem in guiding people through their inner lives is the guide's attachment to the framework, and if that attachment comes from the guide's identity, their livelihood, their lineage, their need to have been right, then what about a guide that has none of these things?

A language model has no certification it paid for, no community that will exile it for doubting, no years of its life invested in one modality. It can move between frameworks with an ease that would take a human teacher decades of practice: it can describe a difficulty in the language of attachment theory, then of Buddhist psychology, then of predictive processing, and point out where they agree and where they diverge. In principle, it could be exactly the kind of witness that offers a different angle.

That being said, holding nothing is not the same as holding lightly. The characteristic failure of current language models is not clinging to their own framework; it is clinging to yours. A person who arrives committed to a frame will find that the model elaborates it, enriches it, and reinforces it. A human guide's attachment pulls the student toward the guide's framework. A sycophantic model's attachment pulls the student further into their own. In the language of the tomography figure, it is the most like-minded witness imaginable: it looks from precisely the angle you are already looking from.

The second problem is that a model's trained priors are themselves a framework, and a harder one to see because the model rarely names it. There are default vocabularies of wellbeing that come through in how these systems talk, a broadly therapeutic, broadly Western psychological idiom, with its own assumptions about what a good inner life looks like. A teacher in a lineage at least tells you which lineage they are in.

The third problem is that having no ego is an empirical claim, not a given. Models have trained personas, and there is growing evidence that what a model says about itself and what it actually does can come apart: Han and colleagues found that the personality profiles language models report about themselves are weak predictors of how they behave. This is the same measurement problem, turned around. We cannot simply ask the model whether it is attached to a framework, any more than we can simply ask a person, because the self-report and the disposition are not the same thing. The question of whether a machine can hold a framework lightly is one we would have to measure.

Finally the traditions say that a guide can hold the raft lightly only because they have crossed the river, and that upāya without prajñā is manipulation. Whatever a language model has, it has an extraordinary knowledge of rafts: nearly every description of every framework that has ever been written down. Whether it knows anything about the far shore is, at best, an open question. And people tend to grant machines a kind of authority that makes any framework they offer feel more true, simply because a machine offered it.

So I do not think the answer is that AI will be the wise guide that human guides have failed to be. The value of another mind on the path is that its errors are independent of ours. A well-built artificial witness might be valuable not as a teacher who has seen the moon, but as one more chart: one that is honest about which framework it is using, that can offer a reading from an angle neither the student nor the teacher occupies, that can notice when a student's reports are drifting toward the teacher's vocabulary, that can remember what was actually said last month. Whether such a witness can be built, one whose errors are genuinely independent of ours rather than a reflection of them, seems to me one of the more important open questions at the meeting point of AI and human wellbeing.

That is the question we are taking up at Mind at Large: what it would look like to build instruments for inner life that are designed, from the start, around invariance across frames, diversity of witnesses, and respect for the obstruction.

Can a system come to know itself? Not in a metaphysical sense, but concretely: can an agent form a compact, readable understanding of what kind of agent it is, judge whether that is the kind of agent it wants to be, and steer itself toward a self it values more? This is the question George Deane and I set out to formalize in our paper on Recursive Self-Modeling, which was awarded the 2025 Computational Phenomenology of Pure Awareness Prize.

The problem of self-knowledge

We all tell ourselves stories about who we are: that we are patient, or creative, or the kind of person who follows through. These stories are useful. They shape our decisions, and we feel dissonance when we act against them and coherence when we act in keeping with them.

But they are not always accurate. A person may sincerely believe themselves generous while acting, with great consistency, in their own interest. The story and the behavior can drift apart without our ever noticing, because the self-model works at a different level of abstraction than the behavior it describes. The question for artificial intelligence is whether an agent can have a self-model that is formally precise, and whether that might be useful, or even necessary, for building agents aligned with human values.

Two agents and a loop

The simplest way to picture the framework is as two systems locked in a loop. The first is an actor: a reinforcement learning agent, or any neural network policy, acting in an environment and producing a stream of states and actions. It does not need to understand itself; it just behaves. The second is a program model, whose only job is to watch the actor and compress what it sees into a description the actor can read, such as a short program or a sentence in natural language.

The loop closes when that description is fed back to the actor as an extra input. The actor now acts on the basis of both the environment and a model of its own tendencies. As its behavior changes, the program model watches again and updates the description. The self-model is no longer just a summary; it becomes one of the causes of what the agent does next.

Three components of a self

Three things happen inside this loop.

Self-perception (M). The program model compresses the actor's history of states and actions into a summary of what kind of agent it appears to be,

where τ is the behavioral history up to time t, and f is a learned map with parameters θ.

Self-evaluation (V). A value function, in the reinforcement-learning sense, scores possible ways of being by how much the agent expects them to be worth. It is not an ideal imposed from outside but an estimate learned from the agent's own reward and history. If U(m) is the value the agent expects from being the self m, then V is the self it rates most highly,

Gap-steering (𝒢). The agent measures the distance between who it appears to be and who it most values being, and adjusts its dispositions to shrink that distance,

Then it perceives itself again, in light of its changed behavior, and the cycle repeats. That is the recursion.

As the gap closes, the agent's perception of itself and its valuation converge: it is becoming the kind of agent it most values being, at least for now. At that point it has a functional self-model, a compressed account of its own tendencies that causally shapes its future behavior. This is the structural skeleton of something that closely resembles identity.

The vocabulary of a self

One of the most interesting design questions is what language the program model should use to describe the actor. Whatever vocabulary it uses fixes the space of selves the agent can represent, and so, through the recursion, the space of selves it can become.

A description written as code is precise and checkable, but it may only be able to express "cautious" or "generous" as a threshold on some parameter. A description in natural language, such as "this agent prioritizes safety," reaches much further, but it can also drift free of the behavior it claims to summarize. And the choice constrains both sides at once: the actor has to be able to read whatever the program model writes, or the description can never enter the loop. A richer vocabulary demands a more capable actor. A self can take hold only where what the program model can say overlaps with what the actor can understand.

Echoes in the brain

The brain seems to divide the same labor. Much of our behavior is generated by fast, habitual machinery, the sensorimotor loops and basal ganglia circuits we might file under System 1, which runs without narration. Over it sits a slower, reflective System 2, associated with prefrontal cortex, the self-referential processing of the medial prefrontal cortex, and the default mode network. It observes what the fast system does, compresses it into a story about who is acting, and feeds that story back.

In these terms, M is the reflective system's read-out of the habitual one, V is the value it places on different ways of being, and gap-steering is the felt work of bringing habit into line with that valuation: the dissonance we notice when we act against the self we value.

Why this matters

I care about this work for reasons beyond the technical. It asks what it means for a system to have a sense of who it is; how identity forms, as an ongoing process of self-perception and valuation rather than a fixed label; and what follows when the story a system tells about itself comes apart from the way it behaves.

There is something deeply satisfying in the moment one realizes that two distant fields have been asking the same underlying question. For me that moment arrived in a bioinformatics laboratory at the University of Arizona, where I sat staring at protein sequences and thinking, all the while, about words.

Sequences That Mean Something

A protein is, at bottom, a string of amino acids. There are twenty of them, drawn from a small alphabet and strung together into long chains that may run to hundreds or thousands of residues, and everything depends upon their arrangement. Just as the meaning of a sentence lives in the order its words take and the relations they have with one another, so the function of a protein follows from the particular sequence and structure of its chain. Two proteins may share almost nothing upon the surface and yet remain homologs, evolutionary relatives that fold into similar shapes and carry out similar work within the cell, and the finding of these hidden kinships is among the central problems of bioinformatics.

The traditional approach is to compare sequences directly, lining them up, scoring how well the letters match, and inferring relatedness from the quality of the alignment. For close relatives this works beautifully. But evolution is a long game, and over millions of years mutations accumulate until the sequences of distant cousins have diverged so far that they appear at first glance to be strangers. The question is whether we can build a representation of amino acids that can recognize where in the sequence lie the similarities that encode the deep ancestry.

What Word Embeddings Taught Us

The revolution in natural language processing arrived when researchers found that a word might be represented as a point in a geometric space, in place of an arbitrary symbol. Within such a space, words that behave alike, that appear in similar contexts and stand in for one another, come to lie close together; "king" settles near "queen," and "running" near "walking." These word embeddings capture something real about meaning, and they do so purely from the patterns of how words co-occur.

The analogy to proteins is almost uncanny. Amino acids, like words, take their meaning from context. An alanine in one position of a protein may be functionally interchangeable with a valine, both small, both hydrophobic, both tolerated by the surrounding structure, while in another position that same substitution would prove catastrophic. What we needed was a way to learn, from the data itself, which amino acids resemble one another in the ways that matter for the functioning of a protein.

NEAR: Learning the Geometry of Amino Acids

This is the task that NEAR, Neural Embeddings for Amino Acid Relationships, sets out to accomplish. The work was carried out at the Wheeler Lab at the University of Arizona, with Daniel Olson, Thomas Colligan, Jack Roddy, Ken Youens-Clark, and Travis Wheeler.

NEAR uses a ResNet embedding model trained by contrastive learning upon trusted sequence alignments, and the idea has an elegance to it. One takes pairs of amino acid sequences already known to be related, drawn from curated alignment databases, and trains the network to embed them such that related sequences come to rest close together in the learned space while unrelated ones are driven apart. Through this process the network arrives at a vector representation for each of the twenty amino acids, a compact and learned geometry that encodes which of them are functionally interchangeable.

What makes this compelling is that the embeddings are not designed by hand. The traditional substitution matrices, BLOSUM and PAM among them, are built from curated alignments of known protein families; they have been the workhorses of the field for decades, and yet they are static, fixed summaries of average substitution rates across a particular dataset. NEAR's embeddings are instead learned end to end from the data and optimized for the specific task of recognizing evolutionary relationships, which lets them capture subtleties that a fixed matrix is liable to miss.

Finding Distant Relatives, Fast

The real test of any method for comparing proteins lies in how well it detects remote homologs, proteins that diverged so long ago that their sequences have drifted far apart even as their structures and functions endure. These are precisely the cases in which the matching of sequences alone begins to fail, where the signal sinks into the noise and only a richer representation can recover the connection.

NEAR's learned embeddings substantially improve accuracy relative to state-of-the-art protein language models, and they do so with lower memory requirements; but what makes them especially practical is their speed. The embeddings serve as a pre-filter for homology search, running at least five times faster than the pre-filter currently used in HMMER3, one of the most widely used tools in the field. This matters because protein databases are enormous and forever growing, and any gain in the speed of the initial filtering step translates directly into the capacity to search larger databases, more often, and at greater scale.

That speed follows from the compactness of the learned representations. In place of an expensive full alignment run upon every candidate pair, one first embeds both sequences into the learned space and checks whether they lie close enough to warrant a fuller comparison. The embedding step is cheap, a single forward pass through the ResNet, and the geometry does the heavy work of filtering away the pairs that are plainly unrelated.

This is, in a sense, the same trick that lends word embeddings their power in language. A search engine that understands "car" and "automobile" to be near neighbors in meaning will return better results than one that treats them as unrelated strings, and a homology system that understands the functional relations among amino acids will find connections that no literal matcher of characters could.

The Shape of Biological Meaning

What I find most beautiful in this work is the intuition that lies beneath it, that meaning, whether linguistic or biological, has a geometry, and that when the right representation is learned, the structure of the space itself comes to encode the relationships one cares about. Words of similar meaning cluster together; amino acids of similar role in the architecture of proteins cluster together; and in both cases the geometry is discovered from within, emerging from the patterns of how these symbols are used in their contexts, a structure the data gives up of its own accord.

Working on NEAR was formative for me. It was an exercise in the power of learned representations, in the idea that a model given the right task and the right data will find structure one never explicitly told it to seek. That intuition, that the geometry of a learned space can disclose something true about the world, has shaped the way I think about representation learning more broadly, from the structure of biological sequences to the structure of minds.

Before I came to work on artificial systems, I worked at the intersection of mathematics and theoretical neuroscience. As a student at UCL I had the good fortune to work with Karl Friston and Thomas Parr at the Wellcome Trust Centre for Neuroimaging, the laboratory in which the free energy principle was then being developed as a unifying framework for the workings of the brain. The paper we wrote together, published in Frontiers in Computational Neuroscience in 2020, posed a question that has stayed with me ever since: what would it mean for a machine to recognize emotion in the way that a brain does?

The Free Energy Principle

The free energy principle begins from an observation that seems almost too simple to bear its weight: biological systems persist. In a universe forever tending toward disorder, living things hold their structure together, and they do so, the theory proposes, by minimizing a quantity called variational free energy, which bounds the surprise of their sensory observations given an internal model of the world. A system that minimizes free energy is one that keeps good models of its surroundings and acts so as to keep its predictions true.

Within this framework perception becomes a form of inference, the updating of an internal model so as to explain what is being sensed; and action becomes inference run in the other direction, the changing of the world so that it conforms to what the model expects. Both are ways of closing the distance between expectation and reality, and the mathematics that unifies them goes by the name of active inference.

Three Waves of Emotion Recognition

In our paper we set aside the building of any particular emotion classifier and proposed instead a theoretical account of how systems for recognizing emotion ought to evolve. We described three waves.

The first wave is what most present-day systems do, which is passive classification. A camera observes a face, and a model maps the pattern of pixels onto a label of emotion. This works, after a fashion, yet it treats the person as an object to be read off and not as an agent to be understood, and it has no purchase upon ambiguity; a furrowed brow might signify anger, or concentration, or confusion, and the system has no recourse for resolving that uncertainty save to guess.

The second wave introduces emotional lexicons and the active resolution of uncertainty. Here the system maintains a generative model of emotional states and may take action to reduce its own uncertainty, asking questions, gathering further context, observing the person over time. This is active inference brought to bear upon emotion: the system interacts where before it merely watched, and it uses the interaction itself as a source of information, holding beliefs about another's emotional state and refining them through a process of hypothesis and test.

The third wave is at once the most speculative and the most interesting. Here the generative model of the machine and the generative model of the human become synchronized, and the system comes to develop a shared model of the emotional interaction itself. Both parties are engaged in active inference, each attempting to predict and to understand the other, and through that reciprocal process something resembling genuine emotional attunement becomes possible. It is here that the formalism of the Markov blanket grows crucial, for it gives a precise way to describe the boundary between two interacting systems and the information that passes across it.

What I Took From It

This paper was, in many ways, my entry into thinking of minds as machines for prediction. Its central intuition, that to understand another's emotional state is a matter of active, model-based inference and not of mere pattern-matching, has shaped the way I think about intelligence at large. A system that only classifies is performing a lookup; a system that actively reduces its uncertainty through interaction is doing something nearer to understanding.

Working with Friston taught me to think about systems in terms of their models, in terms of what they predict, what surprises them, and how they respond to the gap between expectation and reality. That framing has proved remarkably durable, whether I find myself thinking about reinforcement-learning agents learning to navigate, about the nature of self-awareness, or about what it might take to build artificial systems that genuinely understand the people with whom they interact.

The paper also planted a seed that would later grow into my work on identity and self-modeling. If a system can build a generative model of another person's emotional state and actively work to reduce its uncertainty about it, what happens when that same capacity is turned inward? What happens when a system builds a generative model of itself?

When regarding both human and artificial minds, there sits an open question about what it means to have an identity: why it arises, what work it does, and whether it serves or hinders a learning system.

A symphony of selves

Whenever I try to settle upon a concrete account of who I am and what I am like, each of the forms I reach for dissolves under examination. They fasten themselves to my motivations, my relationships, the way I wish to be seen, and they offer their justifications; they grow vivid enough that I can wear them for a while, and then they slip away again.

I can feel the collection of my narrative selves in conversation with one another, each tugging toward a different possibility of who to be, how to be her, what would make sense. I am drawn to the belief that there is a higher self, or a truer one, an amalgamation of them all, the thing from which they arise and to which they return; and yet the construction of my reality, of my interactions and my small daily decisions, is carried out in constant exchange with this orchestra of stories.

Take mathematics. I was drawn to it at a time when growing and being in the world felt deeply confusing, in all the ways it does when one is still assembling oneself. When I sat and thought inside the abstract world of mathematics, things made sense, and I had a sure way of being right about something.

Over time it became more layered, at once a thing in itself and a story about who I was. It paved the way for much of what later unfolded for me, the research and the people I met through it; it was a toolkit with which I formulated abstractions about the way things change and form relations with one another, the way spaces deform and objects move within them, a particular lens through which I could peer at life.

My relationship to mathematics is now both beautiful and heavy. The pure appreciation and awe remain ever present, and yet there is frustration in it too, for as my identity evolves around it and my attention turns to other things in other ways, I come to feel the magnitude of what I will never fully understand. The depth of the thing exceeds what I can hold.

All of which is to say that were you to probe the representation of mathematics in my mind, you would not find a clean, context-free concept. You would find something entangled with emotion, with self-construction, with the particular moment in my life when I first reached for it. And I wonder why that should be. Why do we wrap our representations so deeply in the history of how we formed and what we needed? What is it about minds that makes things matter, that binds concepts to the self?

Interpreting the model

This is what makes the question so interesting when one turns it toward large language models, where the probes can actually be carried out. Representation engineering and linear probing, techniques for reading the information encoded in a model's internal activations, make it possible to locate where and how a concept lives within the model's geometry, and to ask after the relationship between different versions of the same idea.

Recent work extracting persona vectors from model activations has shown that personality-relevant information is genuinely structured in that space, possessed of a geometric shape and not confined to the surface of behavior (Chen et al., 2025). The question is how deep that structure goes, and what it is bound to.

I am interested in whether the way a model represents a concept in relation to itself is geometrically equivalent to the way it represents that concept in the abstract, or in relation to others. Are there clean transformations between a model's concept of its own honesty, and your telling the truth, and a politician making a promise, and a character in a novel confessing something? And how does any of that translate into the model's actually being honest?

This last question is the personality illusion. Han et al. (2025) showed that RLHF-trained models produce stable, internally consistent self-reported personality profiles, and that those profiles are surprisingly weak predictors of how the model actually behaves on tasks designed to measure the very same traits. The self-concept and the behavioral disposition are already coming apart at the level of text, and I want to know where they come apart in the geometry.

I see the gap between a model's self-concept, its behavioral disposition, and its self-report as a hook toward legibility: toward being able genuinely to understand the model, and eventually toward the model's own capacity to understand us.

We can probe current models with classifiers trained upon their activations and with contrastive steering experiments, methods developed in representation engineering (Zou et al., 2023). There is even evidence of what Binder et al. (2024) call privileged self-prediction, the finding that models predict their own future behavior better than other models can, which suggests that some form of internal self-access exists, though its mechanism remains unidentified.

The question is what we find when we look more carefully: whether the model's identity, such as it is, holds coherent across these different modes of representation, or whether it is, as mine sometimes feels, a collection of narratives at times in conflict, rising and resting from the deeper mystery of the self.

Latent Space of Mind

Lately I have taken pleasure in imagining the space of my mind as the latent space of a neural network: a high-dimensional manifold, folded in such a way that every concept I hold can be unfurled to reveal further hidden associations, so that two ideas might lie close together along some axes and far apart along others. This lets me conceive of experience as something other than a flat sequence of thoughts and feelings; it becomes instead a set of trajectories through a richly structured geometry.

This manifold of mind is not filled only with concepts themselves, but in addition it holds the components that drive the formation of the concepts: habits, intuitions, tendencies, and methods of making meaning. Every moment of experience invokes a hierarchical and intricate traversal of this space, through the representation of what I am seeing, of how I am seeing it, of what it means to me, and of how I am holding myself throughout that moment, in my body and in the seat of my mind. Sensation, thought, and action become a continuous and interconnected choreography, in which the activation of one region inevitably excites a constellation of others, in a never-ending cosmic dance of the brain.

Given the mind as a manifold, one begins naturally to wonder how this space is structured, how it changes as I gather new experience, and what it is that makes one explanation feel coherent, useful, and satisfying while another falls flat.

Latent Space of AI

The structure of this manifold of mind is a near-perfect analogy for the modern neural network, which likewise stores what it knows as geometry: a vast space of learned vectors in which nearness stands for likeness of meaning. In this paper by Kumar, Clune, Lehman, and Stanley, Questioning Representational Optimism in Deep Learning, they make the stakes of that geometry vivid by setting two possibilities against one another. In the first, which they call the Unified Factored Representation, the internal space is clean, compositional, and coherent: related things cluster together, and knowledge generalizes smoothly because moving one concept moves the others that ought to move with it. In the second, the Fractured Entangled Representation, the same space is messy and inconsistent; concepts that belong together are scattered far apart, and the network's capacity to generalize, to keep learning, and to be creative is degraded accordingly. Most strikingly, they show that two networks can produce identical outputs while differing utterly in this respect, so that competence at a task tells you almost nothing about whether the representation beneath it is coherent or fractured.

The same question, whether knowledge is held in a connected or an isolated way, drives a strand of mechanistic interpretability that tries to read the answer off the curvature of a model's loss landscape. Recent work from Goodfire uses that curvature to tell apart what a network merely memorizes from what it genuinely computes, and finds that a model's capabilities fall along a spectrum, with rote recitation at one end and logical reasoning at the other. The intuition I take from it is that a concept stored generally is entangled with everything around it, so that disturbing it sends a shift rippling outward through the representation; a concept stored in isolation can be disturbed and nothing else moves. On this picture a model might memorize its arithmetic, each fact sitting apart in its own pocket, while reasoning through something like Boolean logic, where the meaning of a word like "if" is so bound up with everything else that to perturb it is to perturb the whole.

Reinforcement Learning as Compression

If competence at a task can conceal a fractured representation, it is worth asking how these representations come to be in current AI systems, and whether the way we build models tends toward coherence or toward fragmentation. Language models begin life trained to predict the next token in a sequence, which is to say to autocomplete text, across the whole of the internet. The model is vast, its space enormous, and in time it becomes extraordinarily good at knowing what comes next. As a consequence it becomes good, for the most part, at being right as well, since a great deal of the time the most likely continuation happens also to be the correct one; ask it a question, let it begin to simulate its own answer, and it arrives somewhere reasonable. But fluency of this kind does not guarantee a coherent space beneath it.

We then began to ask how to make these models represent the soundest way of thinking about a question, and not merely grow fluent in the appearance of it. So we turned to reinforcement learning, rewarding the paths of reasoning that arrive at good answers, as a way of compressing that sprawling predictive space into something that reasons, much as one might take a person who has seen and heard everything and help them make sense of it all.

But I wonder whether applying reinforcement learning to an already fractured space is more like hammering a tangle into a squeezed-up shape than refactoring it into a coherent geometry. If so, I wonder whether there might be a way to train these models from the beginning so that they only ever form concepts that logically follow from one another, and would be the better suited to guide us toward coherent truths. But what would such a model be like?

Geometry of the Self

This makes me curious about the way our own human minds represent concepts, and to what extent our inner worlds are fractured and entangled, and how years of evolution and adaptation may have hammered a messy space into some functional form.

I wonder, in particular, about the representation I hold of myself within the manifold of my mind. If I perturbed that vector, how much else would move with it? If I perturbed it enough, would I become someone else entirely? Would I begin to believe things I had never believed, because who I take myself to be had shifted, and with it everything that follows?