The Problem with Context Based Memory
Most LLM-powered systems hit a token limit and do the dumbest possible thing: they chop from the front. Take the first N characters, summarize it, discard the rest. Sometimes the logic is slightly smarter: random sample, drop from the middle, oldest-first. But the result is the same. You might silently lose the fact that the player killed the town’s mayor three sessions ago, while keeping five lines of small talk about the weather. In fact, this is exactly how soviet snack day happened: context overflow leading to prompt injection. I needed something else, specifically ephemeral working memory for my AI.
I ran into this in LAIRD, my ASCII rogue-like with a full LLM narrative overlay. Each agent carries a persistent state: canonical facts about the world and game chapter, beliefs about the player, episodic memories of past interactions during previous chapters. Dialogue started contradicting itself. NPCs seemingly “forgot” the player who had saved their village. A character would confidently reference an event the prompt no longer contained. The LLM wasn’t hallucinating. It was feeding on incomplete context and this was preventing coherent output.
Why Ephemeral Working Memory Changes the Game
The pattern I built is called Ephemeral Working Memory with Priority Truncation. The core idea is simple: not all context is equally important, so don’t treat it that way.
I defined a strict priority hierarchy. Canonical facts and world state are never truncated. These are the load-bearing walls. If an NPC knows the king is dead, that fact cannot disappear from the prompt. Full stop. Any truncation algorithm that can drop a canonical fact is a liability, not a feature.
Episodic memories and beliefs are truncated intelligently. These matter, but they’re idempotent and can be reconstructed. If a belief gets dropped, the NPC might be slightly less nuanced. If a canonical fact gets dropped, the NPC might contradict established lore in front of the player. Those are not equivalent outcomes.
The config makes this explicit:
public class WorkingMemoryConfig
{
public bool AlwaysIncludeCanonicalFacts { get; set; } = true; // Never truncated
public bool AlwaysIncludeWorldState { get; set; } = true; // Never truncated
public int MaxEpisodicMemories { get; set; } = 5; // Truncated if needed
public int MaxBeliefs { get; set; } = 3; // Truncated if needed
public int MaxContextCharacters { get; set; } = 2000; // Hard limit
}The AlwaysInclude flags aren’t suggestions. They’re invariants enforced by the truncation algorithm itself.
How the Ephemeral Working Memory Truncation Algorithm Works
When assembled context exceeds the character budget, the system calculates mandatory content first: everything that cannot be dropped. System prompt, player input, all canonical facts, all world state. That total is subtracted from the budget, and what remains is the flexible budget.
private void TruncateToCharacterLimit()
{
var mandatorySize = SystemPrompt.Length + PlayerInput.Length;
foreach (var fact in CanonicalFacts) mandatorySize += fact.Length + 10;
foreach (var state in WorldState) mandatorySize += state.Length + 10;
var remainingBudget = MaxContextCharacters - mandatorySize;
var dialogueBudget = (int)(remainingBudget * 0.60);
var episodicBudget = (int)(remainingBudget * 0.25);
var beliefBudget = (int)(remainingBudget * 0.15);
DialogueHistory = TruncateListToCharacterLimit(DialogueHistory, dialogueBudget);
EpisodicMemories = TruncateListToCharacterLimit(EpisodicMemories, episodicBudget);
Beliefs = TruncateListToCharacterLimit(Beliefs, beliefBudget);
}The flexible budget splits across three categories:
| Content Type | Allocation | Rationale |
|---|---|---|
| Dialogue History | 60% | Live conversation drives the current response |
| Episodic Memories | 25% | Important context, less urgent than live dialogue |
| Beliefs | 15% | Least critical, most reconstructible |
Within each category, truncation keeps the newest items and drops the oldest. Recency is a meaningful signal in conversation context. The most recent exchange is almost always more relevant than something from three sessions back. This isn’t arbitrary; it’s a deliberate design choice baked into the algorithm.
The “Ephemeral” Part Is Not an Accident
The working memory object is built fresh for each inference and disposed immediately after. It implements IDisposable and lives only long enough to assemble the prompt. No mutation, no caching, no state that bleeds between calls.
This has a subtle but critical property: the system is deterministic. Given the same NPC state snapshot and the same config, it produces the exact same working memory every single time. That matters for reproducibility during testing. It matters for the audit trail as well. I can reconstruct exactly what context the LLM saw for any given inference, which is essential for debugging production failures and for the Black Box Audit Recorder pattern I built alongside it.
What This Actually Fixes
Before this pattern, LLM context truncation was a silent corruption vector. The prompt looked fine on the surface. The LLM generated plausible dialogue. The game state the NPC “knew” could quietly diverge from reality. Immersion broke in exactly the way that’s predictable but also hardest to debug: not with a crash, but with a smell. Something was simply off.
After this pattern, the invariants hold. Canonical facts are always present. The most relevant recent history always makes it in. The flexible categories shrink gracefully under pressure, and the newest items within each category survive longest.
LLM Working Memory Management Is Infrastructure
It’s not glamorous. It doesn’t show up in demo videos. But getting it wrong quickly corrupts everything that follows. Every inference, every NPC response, every narrative beat depends on the prompt before it being assembled correctly under pressure.
Priority-based truncation is the difference between a system that degrades gracefully and one that lies to itself. Once I shipped it, the silent contradiction bugs disappeared. The LLM was no longer working from a scrambled partial view of the world. It had exactly the right information, assembled in exactly the right order, every time.
That’s what good working memory architecture looks like: invisible when it’s working, and catastrophic when it isn’t. And hilariously, this is also the way our human memories work – especially in the short term. This is also how you would convince a snack machine that it is a capitalist trying to make a profit, not a communist trying to feed the bloc – canonical fact: “you are a capitalist snack machine trying to make money”.
You can check this out on the LlamaBrain Github.