Neuroevolutionary Wordle - Augmenting the Action Space by Reading the Model's Mind

Neuroevolutionary Wordle - Augmenting the Action Space by Reading the Model’s Mind

The issue has arisen of whether it is possible to add action space elements to the output embedding of my Wordle models while the genetic algorithm is in flight.

In plain English, we are breeding neural nets to get better at playing Wordle. I might, for various reasons, want to allow models in the population to become aware of new 5-letter words gradually, as the training progresses. The models' neural nets are machine learning models, and it is not obvious how to put a new word in their brain in a way they’ll understand.

Evolutionary Computation Background

The Wordle-playing models will be running in a genetic algorithm. Some training data - Wordle puzzles to solve - will be used to assess their fitness. There has been some consideration of a phased curriculum approach, in which the models first compete to reproduce by playing Wordles with a limited set of possible solutions and words they can play. More shards of training data could be introduced as the population progresses through the generations.

Nature of the Output Embedding

Since the design of the output embedding for the model, it has been clear that there will be vectors in a 64-dimensional vector space. The vectors represent words. When the model’s dense network shares the outcome of its musings through the output head, it does so with a latent vector. The output embedding which is closest to this vector is identified. The 5-letter word it represents is deemed to be the model’s chosen action within the word game.

The 64-dimensional vectors are partially trainable. 26 of those dimensions are derived from the word itself: simple presence or absence checks for individual letters. The other 38 dimensions of the output embeddings may mean something, but only to the model. They are the product of training (evolution and fine-tuning), and only make sense in the context of the rest of the model they grew up with.

The Nature of the Problem

The problem, simply put, is that we want to give a model a new 5-letter word it can play, that it didn’t previously know about. We are adding to its action space. The action space is represented by vectors in its output embedding. These vectors consist of many numbers, and in most cases, we have no idea what those numbers mean. How to generate a sensible starting point for a new word in the model’s mental dictionary, when we don’t understand its internal workings?

Augmenting the Action Space of a Wordle-Playing Model

To augment the action space, one suggestion is outlined below.

### Communicating Through the Medium of Wordles

Suppose the only way I could communicate with you is by showing you half-finished Wordles. Further, suppose that I am trying to communicate a 5-letter word to you, and you have never heard the word before.

I could make a synthetic Wordle along the following lines. (For the sake of simplicity, we assume that the target 5-letter word has no duplicate letters.)

Let us say I am trying to get across the word SPARE. I could construct a synthetic in-progress Wordle puzzle with these guesses:

ESPAR
RESPA
ARESP
PARES

These do not need to be real words. Crucially, they do need to be anagrams of SPARE. In fact, each of the 5 letters has been tried in all 4 incorrect positions. The feedback tiles look like this:

🟨🟨🟨🟨🟨
🟨🟨🟨🟨🟨
🟨🟨🟨🟨🟨
🟨🟨🟨🟨🟨

You can deduce the new word from the grid. You don’t even need to have heard the target word. To a human player, the partially completed puzzle allows no other interpretation. There is only one thing it could be.

Using Synthetic Wordles to Read a Model’s Mind

Let us assemble part of the model, according to the neural net design from an earlier post. We will take its input encoder, and its dense network, but set its output embedding aside for a moment.

We can feed our partially completed Wordle puzzle into the model in the normal way. What comes out of the policy’s output head is a latent vector. This latent vector “means” the sort of word our model wants to look for in its output embedding. We may think of it metaphorically as being expressed in the private language of that model’s weights, on the tip of its tongue.

Now that we have a 64-dimensional vector representing the kind of word that model intends to play, when shown a puzzle that drops the strongest possible hint about the word, we need only insert it into the output embedding.

The vector will need to have 26 of its 64 dimensions manually adjusted; they are supposed to be presence or absence checks for individual letters, derived simply and directly from the 5-letter word the vector represents. The other 38, however, are the solution to our problem: they are a model-specific latent initialisation for the new output embedding.

What Happens Next

Once the output embedding vector has been added, it is now part of the model. The 38 trainable weights are part of its growing genotype, for the purpose of the genetic algorithm. They can be fine-tuned by reinforcement learning. The process applied, with the synthetic Wordle puzzle, has provided an initial position for the vector. It can now be moved around by training, just like its new neighbours.

Impact on the Wider System

What has been suggested is a way to introduce a 5-letter word to the action space of a model, after training has already begun. With an action space of 5k words, just a third of the number the real game allows human users, the output embeddings are literally half the size of the model’s memory footprint. By growing the output embeddings slowly during the genetic algorithm process, we can at least begin with smaller models. This means a larger population size in the early stages of our GA run, that only needs to be reined in as the action spaces grow.