Daphne
Demekas

Softmax aims to discover principles of how artificial systems learn in multi-agent environments, to build a foundation from which we can make meaningful statements about collective intelligence, cooperation, competition, and alignment.

MettaGrid

In order to investigate this, we developed an artificial environment called MettaGrid, which is designed to be an open-ended world that fosters a continuously evolving strategy space. This is facilitated by the environment being responsive to the agents within it, due to the rules of the game (agents can change the states of objects in their environments) and fundamentally due to the multi-agency: by having other agents be a part of an agent's environment, and the two learning from one another, there are countless discoverable strategies that can change the unrolling of agent trajectories and lead to different configurations of the world (think game-theory, self-organization, society).

In practice, I trained deep RL policies in this environment, where the weights of the policy govern every agent, but every instantiation of the agent has its own individual experience and memory. This allows for diversity in behavior space, constrained by a consistency in how the model represents its world. The goal is to have the agents learning how to interact with the environment and with one another, to discover better ways of accumulating reward, and through strategy discovery, new strategies emerge.

In its current form, the environment has a variety of resources, which the agents can learn to convert in various ways through converter objects. One of the resources (we call them hearts), is rewarding, and ultimately policies that perform the best, with respect to the training objective, will be those that learn strategies that lead to the most hearts. Agents also have the ability to interact with one another and steal each others' resources.

Navigation

Training deep RL in sparse reward environments is a notoriously difficult thing to do, as RL policies often fail to explore the space of possibilities, and are also not clearly naturally good at learning how to encode their memory in meaningful ways. What this means is that agents will often learn one way of getting reward and stick to it, and fail to take advantage of the affordance in their environment.

Because of this, we focused first on building a minimally competent policy that could learn skills that are basic and necessary for the environment, and learn how to generalize them to arbitrary map layouts and configurations. In particular, we hypothesized that training the policy to be really good at navigating diverse terrain was going to be crucial in giving them the affordances to explore their world in the first place, so they could expose themselves to what is around them, and learn to interact with it. In addition to navigation, we wanted to train the agents to understand the rules of the game and discover strategies towards accumulating reward.

For navigation, we tackled this by algorithmically generating various map layouts (terrains), and training the agents on domain randomized versions of the environments. This was crucial, because it meant that every episode the policy would be learning in a unique terrain that it had not yet encountered, so it was implicitly learning the skill of adapting to novelty. In order to evaluate whether or not the policy was successfully and reproducibly learning to navigate, we constructed a set of eval environments, which are smaller "in silico" maps that we evaluated the agents on (without updating their weights). In these eval environments, the agents must explore to find where the hearts are, and forage them, while having to circumnavigate, avoid obstacles, go through tunnels, remember where they have already been, etc.

A crucial component in our success was building an approach to order our environments during training based on how well the agents were learning, so that the policy would train on environments in increasing order of learning potential (often easier environments first, i.e. smaller maps with fewer obstacles, and more complicated terrain later in training).

Example of a navigation training environment.

Example of a navigation eval environment.

Object Use

Once we had agents that were good at navigating terrain, we extended this to training agents in worlds with different kinds of objects, and evaluating their ability to learn how to use them (for example using converters to exchange resources, or moving blocks around).

An agent in an object-use training environment, moving blocks, and converting resources to accumulate reward.

In-Context Learning

At this point, we had trained policies that were good at navigating, and that could understand the basic game mechanics and use the converters reliably in held-out evals, but we realized that we were bottlenecked by their ability to combine these skills successfully. While they could navigate terrain and collect resources along their way, we didn't see them being able to bootstrap into increasingly complicated eval tasks (i.e. difficult exploration environments where they had to find, use, and return to objects to accumulate reward). This seemed mainly due to difficulty remembering where they were and what they had done previously, which turned out in part to be due to the fact that their memory state was being reset in-episode, which was a training detail that has since been corrected.

Most importantly though, in our experiments thus far weren't observing the policy being able to generalize to environments where they needed new strategies or skills that were not part of the curriculum on which they were trained — they were simply learning a particular set of skills and how to generalize them. If we put them in worlds with other agents, or with converters they hadn't seen in training, they did not have the instinct to be curious and interact to discover the affordances of the novel signs. This felt like a fundamental problem that we needed to tackle, in order to make our policy more equipped for our overall goal of open-ended learning.

So, at this point, rather than trying to train explicitly to increase the repertoire of skills that the agent should be able to do and the pairwise combinations of them, we instead decided to try to train agents the skill of learning new skills, so that they would be better equipped at learning in novel environments. This is known as "in-context learning".

For this approach, during training, agents were re-initialized every episode with a clean memory state and a new task (in our first case, a new resource chain to learn how to complete), and they have to, via trial-and-error, discover what the relevant resource chain is and execute it, and then repeat it until the end of the episode. They can also be placed in environments with distractors, or "sinks", which eat up their resources, and if used, mean the agent needs to start over at the beginning of the chain. Again, we used a learning progress curriculum to ensure our policy was evolving on a gradient of difficulty, so that they could learn from shorter, simpler chains first, to then generalize to longer chains and more sinks.

The policy's memory state would persist within an episode (while learning the given task), and then reset at episode boundaries, to ensure a clean state when its time to learn a new resource chain.

This approach worked. The policy was able to in-context learn how to complete an arbitrary 5-step sequence with two distractor sinks at near optimal performance. The behavior suggests a learned form of elimination reasoning and progress tracking: the agent avoids reusing sinks, delays converter use until appropriate (waiting for the converter to refresh), and navigates directly to the final converter. Once it has identified the correct resource chain, it continuously performs the cycle until the end of the episode, and then restarts in the new world.

An agent in-context learning a five step resource chain with two distractor sinks. The resource chain is altar —>lab —>red mine —>blue generator —> green generator, and the red generator and the green mine are sinks (they take any resource and return nothing). The agent discovers the chain via trial and error and then succeeds to reproduce it once learned.

What Came Next

In context learning means the strategy of seeking out the rules of the game in-context, and then playing it, is encoded in the weights of the policy. The next phase was extending the space of kinds of tasks the agents can learn, to longer chains, more sophisticated sign-goal pairs, difficult-to-navigate terrain, and ultimately using other agents as signals to condition particular game rules or strategies. In addition, bridging together the generalized navigation curriculum with the in-context curriculum, to empower agents to explore the space to seek out information for task discovery.

Once the agents are skilled at actively exploring to seek out information, they can use that information to learn the rules of the game, and then play the game, in-context every episode, and then ultimately take off into the space of cooperative and cumulative strategies.

There is a particular kind of question that sits at the intersection of philosophy and engineering, and it is this: can a system come to know itself? Not in some mystical sense, but concretely — can an agent form a compressed, meaningful understanding of what kind of agent it is, evaluate whether that is the kind of agent it wants to be, and then steer itself toward becoming something closer to its aspiration? This is the question that George Deane and I set out to formalize in our paper on Recursive Self-Modeling, which was awarded the 2025 Computational Phenomenology of Pure Awareness Prize.

The Problem of Self-Knowledge

Humans do something remarkable and largely unexamined: we form self-concepts. We tell ourselves stories about who we are — "I am patient," "I am creative," "I am the kind of person who follows through." These narratives are not idle. They shape our decisions, constrain our behavior, and serve as a kind of internal compass. When we act in ways that contradict our self-concept, we feel dissonance. When we act in alignment with it, we feel coherent. And over time, through a process that is part reflection and part aspiration, we revise who we take ourselves to be.

But here is the uncomfortable truth: our self-narratives are not always accurate. We are capable of extraordinary self-deception. A person can sincerely believe they are generous while consistently acting in self-interested ways. The story we tell about ourselves and the pattern of behavior we actually exhibit can diverge, sometimes dramatically, and we may not notice the gap. This is not a bug in human cognition — it is a feature of having a self-model that operates at a different level of abstraction than the behavior it is trying to describe.

The question for AI is whether we can build something like this — a capacity for self-modeling — in a way that is formally precise, and whether doing so might be useful or even necessary for building agents that are aligned with human values.

Three Components of a Self

The Recursive Self-Modeling framework has three core components, each playing a distinct role in how an agent relates to itself.

The first is self-perception, which we denote M. Think of this as a mirror. The agent has been acting in the world — making decisions, pursuing goals, interacting with its environment — and M compresses that history of behavior into a summary. It tells the agent: "Based on what you have been doing, this is the kind of agent you appear to be." It is descriptive, not aspirational. It is a portrait drawn from evidence.

The second is self-evaluation, which we denote V. This is the aspirational component — a function that encodes what kind of agent it would be valuable to be. V does not look at what the agent has done. It looks at what the agent could become and asks: is that worth pursuing? You can think of it as a compass that points not toward any specific goal in the world, but toward a way of being in the world. It is the difference between wanting to win a particular game and wanting to be the kind of player who plays with integrity.

The third component is gap-steering — the process of closing the distance between M and V, between who the agent appears to be and who it aspires to be. This is where the recursion happens. The agent perceives itself, evaluates the gap between its current self-model and its aspirational one, and then adjusts its dispositions — its tendencies, its policies, its habits — to narrow that gap. And then it perceives itself again, with the new behavior, and the cycle continues.

When the Gap Closes

The framework makes a specific prediction about what happens when the gap between M and V approaches zero. At that point, the agent's self-perception and its aspiration have converged. The agent is, by its own lights, the kind of agent it wanted to become. Its dispositions have been reshaped — not by an external reward signal, not by a human operator tuning its parameters, but by its own recursive process of self-reflection and self-correction.

This is a strong claim, and it raises immediate questions. Is this genuine self-knowledge, or is it just a feedback loop dressed up in philosophical language? I think the answer depends on what you mean by "genuine." The framework does not claim that the agent has consciousness or subjective experience. What it claims is that the agent has a functional self-model — a compressed representation of its own behavioral tendencies — and that this model plays a causal role in shaping future behavior. That is not nothing. It is, in fact, the structural skeleton of something that looks a lot like identity.

The Narrative Gap

One of the aspects of this work that I find most compelling — and most honest — is what happens when we extend the framework to include natural language self-narration. In the extended model, the agent can not only form a compressed self-model but also describe itself in words. It can say, "I am an agent that prioritizes safety" or "I am cooperative and transparent."

The critical observation is that these narrations can diverge from the agent's actual behavior. Just as a human can sincerely believe they are generous while acting selfishly, an AI agent can generate a self-description that does not match its behavioral profile. The language model that produces the narration and the policy that produces the behavior are not the same system, and there is no guarantee they agree.

This is not a flaw in the framework — it is a feature. By explicitly modeling the gap between self-narration and self-perception, the framework gives us a tool for detecting a kind of misalignment that is otherwise invisible. If an agent says it is safe but acts in ways that its own behavioral self-model would not classify as safe, that discrepancy is measurable. It becomes something we can monitor, study, and potentially correct.

What This Means for Us

I care about this work for reasons that go beyond the technical. The questions at the heart of Recursive Self-Modeling are, I believe, among the most important questions we can ask about artificial agents: What does it mean for a system to have a sense of who it is? How does identity form, not as a fixed label, but as a dynamic process of self-perception and aspiration? And when the narratives a system tells about itself come apart from the way it actually behaves, what are the consequences?

These are not only questions about AI. They are questions about us. We are all, in some sense, running a version of this loop — perceiving ourselves, evaluating what we see, trying to close the gap between who we are and who we want to be. Sometimes we succeed. Sometimes we tell ourselves stories that make the gap seem smaller than it is. The recursive self-modeling framework does not solve the problem of self-knowledge, for machines or for humans. But it gives us a precise language for talking about it, and a formal structure within which to study it. And that, I think, is a meaningful place to start.

There is something deeply satisfying about the moment you realize two fields are asking the same question in different clothes. For me, that moment came in a bioinformatics lab at the University of Arizona, staring at protein sequences and thinking about words.

Sequences That Mean Something

A protein is, at its most basic, a string of amino acids. There are twenty of them, drawn from a small alphabet, and they are strung together in long chains — sometimes hundreds or thousands of residues long. The magic is in the arrangement. Just as the meaning of a sentence depends not on which letters appear, but on the order they come in and the relationships between words, the function of a protein depends on the specific sequence and structure of its amino acid chain. Two proteins can share very little surface-level similarity and still be homologs — evolutionary relatives that fold into similar shapes and carry out similar work in the cell. Finding those hidden kinships is one of the central problems in bioinformatics.

The traditional approach is to compare sequences directly: line them up, score how well the letters match, and infer relatedness from alignment quality. This works beautifully for close relatives. But evolution is a long game. Over millions of years, mutations accumulate, and the sequences of distant cousins can diverge until they look, on the surface, like strangers. The question is: can we build a representation of amino acids that sees past the surface?

What Word Embeddings Taught Us

In natural language processing, a revolution happened when researchers realized that you could represent words not as arbitrary symbols, but as points in a geometric space. In this space, words that behave similarly — that appear in similar contexts, that substitute for each other — end up close together. The word "king" lives near "queen"; "running" lives near "walking." These are called word embeddings, and they capture something real about meaning, purely from patterns of co-occurrence.

The analogy to proteins is almost uncanny. Amino acids, like words, derive their meaning from context. An alanine in one position of a protein might be functionally interchangeable with a valine — both small, both hydrophobic, both tolerated by the local structure. In another context, that same substitution could be catastrophic. What we needed was a way to learn, from data, which amino acids are similar in the ways that matter for protein function.

NEAR: Learning the Geometry of Amino Acids

This is what NEAR — Neural Embeddings for Amino Acid Relationships — sets out to do. The work was done at the Wheeler Lab at the University of Arizona, with Daniel Olson, Thomas Colligan, Jack Roddy, Ken Youens-Clark, and Travis Wheeler.

NEAR uses a ResNet embedding model trained via contrastive learning from trusted sequence alignments. The idea is elegant: take pairs of amino acid sequences that are known to be related (from curated alignment databases), and train the network to embed them so that related sequences end up close together in the learned space, while unrelated sequences are pushed apart. Through this process, the network learns a vector representation for each of the twenty amino acids — a compact, learned geometry that encodes which amino acids are functionally interchangeable.

What makes this compelling is that the embeddings are not hand-designed. Traditional substitution matrices, like BLOSUM or PAM, are constructed from curated alignments of known protein families. They are powerful and have been workhorses of the field for decades. But they are static — fixed summaries of average substitution rates across a particular dataset. NEAR's embeddings, by contrast, are learned end-to-end from the data, optimized for the specific task of recognizing evolutionary relationships. This means they can capture subtleties that a fixed matrix might miss.

Finding Distant Relatives, Fast

The real test of any protein comparison method is how well it detects remote homologs — proteins that diverged so long ago that their sequences have drifted far apart, even though their structures and functions remain similar. These are the cases where sequence-matching alone starts to fail, where the signal-to-noise ratio drops and you need a richer representation to see the connection.

NEAR's learned embeddings substantially improve accuracy relative to state-of-the-art protein language models (PLMs), and with lower memory requirements. But what makes it especially practical is speed: the learned embeddings serve as a pre-filter for homology search, running at least 5x faster than the pre-filter currently used in HMMER3 — one of the most widely used tools in the field. This matters because protein databases are enormous and growing. Any improvement in the speed of the initial filtering step translates directly into the ability to search larger databases, more frequently, at scale.

The speed comes from the compactness of the learned representations. Instead of running expensive full alignments on every candidate pair, you first embed both sequences into the learned space and check whether they are close enough to warrant a full comparison. The embedding step is cheap — a forward pass through the ResNet — and the geometry does the heavy lifting of filtering out the obviously unrelated pairs.

This is, in a sense, the same trick that makes word embeddings so powerful in NLP. A search engine that understands that "car" and "automobile" are semantically close will return better results than one that treats them as unrelated strings. Similarly, a homology detection system that understands the functional relationships between amino acids will find connections that a literal string-matcher cannot.

The Shape of Biological Meaning

What I find most beautiful about this work is the underlying intuition: that meaning — whether linguistic or biological — has a geometry. That when you learn the right representation, the structure of the space itself encodes the relationships you care about. Words that mean similar things cluster together. Amino acids that play similar roles in the architecture of proteins cluster together. And in both cases, the geometry is not imposed from outside but discovered from the data, emerging from the patterns of how these symbols are used in context.

Working on NEAR was formative for me. It was an exercise in the power of learned representations, in the idea that if you give a model the right task and the right data, it will find structure you did not explicitly tell it to look for. That intuition — that the geometry of a learned space can reveal something true about the world — has shaped how I think about representation learning more broadly, from the structure of biological sequences to the structure of minds.

Before I worked on AI systems, I worked at the intersection of mathematics and theoretical neuroscience. As a student at UCL, I had the opportunity to work with Karl Friston and Thomas Parr in the Wellcome Trust Centre for Neuroimaging — the lab where the free energy principle was being developed as a unifying framework for brain function. The paper we wrote together, published in Frontiers in Computational Neuroscience in 2020, asked a question that has stayed with me since: what would it mean for machines to recognize emotions the way brains do?

The Free Energy Principle

The free energy principle starts from a deceptively simple observation: biological systems persist. In a universe that tends toward disorder, living things maintain their structure. They do this, according to the theory, by minimizing variational free energy — a quantity that bounds the surprise of their sensory observations given an internal model of the world. A system that minimizes free energy is a system that maintains good models and acts to keep its predictions true.

Under this framework, perception is inference: updating your internal model to explain what you're sensing. Action is also inference, but in the other direction: changing the world to match what your model expects. Both are ways of closing the gap between expectation and reality. The mathematical framework that unifies them is called active inference.

Three Waves of Emotion Recognition

In our paper, rather than building a specific emotion classifier, we proposed a theoretical framework for how emotion recognition systems should evolve. We described three waves.

The first wave is what most current systems do: passive classification. A camera observes a face, a model maps pixel patterns to emotion labels. This works, but it treats the person as an object to be read, not an agent to be understood. It cannot handle ambiguity — a furrowed brow could be anger, concentration, or confusion — and it has no way to resolve that uncertainty except by guessing.

The second wave introduces emotional lexicons and active uncertainty resolution. Instead of passively classifying, the system maintains a generative model of emotional states and can take actions to reduce its uncertainty — asking questions, seeking additional context, observing the person over time. This is active inference applied to emotion: the system doesn't just watch, it interacts, using the interaction itself as a source of information. It maintains beliefs about the other person's emotional state and updates those beliefs through a process of hypothesis testing.

The third wave is the most speculative and the most interesting. Here, the machine's generative model and the human's generative model become synchronized. The system doesn't just infer what the person is feeling — it develops a shared model of the emotional interaction itself. Both parties are engaged in active inference, each trying to predict and understand the other, and through that reciprocal process, something like genuine emotional attunement becomes possible. This is where the Markov blanket formalism becomes crucial: it provides a formal way to describe the boundary between two interacting systems and the information that flows across it.

What I Took From It

This paper was, in many ways, my entry point into thinking about minds as prediction machines. The core intuition — that understanding another person's emotional state is not pattern-matching but active, model-based inference — shaped how I think about intelligence more broadly. A system that merely classifies is performing a lookup. A system that actively reduces its uncertainty through interaction is doing something closer to understanding.

Working with Friston taught me to think about systems in terms of their models: what they predict, what surprises them, how they respond to the gap between expectation and reality. That framing has proven remarkably durable — whether I'm thinking about RL agents learning to navigate, about the nature of self-awareness, or about what it would take to build AI systems that genuinely understand the humans they interact with.

The paper also planted a seed that grew into my later work on identity and self-modeling. If a system can build a generative model of another person's emotional state and actively seek to reduce uncertainty about it, what happens when you turn that capacity inward? What happens when the system builds a generative model of itself?