Temporal Cross Conversational Memory for Large Language Models

Sriniketh Ponnada

The problem

Consumer-facing LLMs hosted by frontier AI labs today offer a feature generally known as "Personalization" that allows the LLM to reference information from past conversations and provide results tailored to the user. The visible mechanism used to achieve this is the curation of a summary about the user from past conversations, updated periodically. This summary is either appended directly to the system prompt of every conversation and preloaded into the context window, or individual files on each topic about the user are listed in the system prompt, allowing the model to read each file through a function or tool call.
(For example, see https://claude.ai/new#settings/customize-memory or https://chatgpt.com/#settings/Personalization).

In this writeup, I will refer to the cross conversational personalization in LLMs as "memory".

Memory is an especially important feature for consumer oriented chatbots (in contrast to system facing LLMs used as tools), as the average consumer converses with an LLM as if it were another human, even while understanding otherwise. Therefore, it is helpful to view the behaviour of a chatbot as being analogous to the behaviour of a human in conversation and identifying its shortcomings.

The first shortcoming is downstream of the concise nature of a summary — chatbots have to tradeoff details for compactness, something that humans do not do. Humans can effortlessly recall niche details from previous conversations, and the user expects this from chatbots as well. A fact that doesn't make it into the summary is invisible the model. Worse — an incorrect fact, or one taken out of context that makes it into memory will be surfaced in future conversations.

Second — Facts present in a summary are time independent. The model cannot tell old and new information apart. Old information remains in memory even if the information is untrue or expired. Unless the user explicitly notifies that a piece of information is irrelevant, the model will not know to remove old facts from the summary.

Third — By preloading the summary into every conversation through the system prompt, factually correct but irrelevant facts are introduced to the model, biasing it to include such facts when unnecessary. We do not recollect everything we know about someone every time we converse with them. Recollection is natural and triggered by the details of the conversation itself.

Fourth — A summary cannot preserve the texture of a memory. 'Bill has a difficult relationship with his boss' loses the specific words and emotions that Bill uses and conveys in the conversation. Therefore, on retrieval, the model does not know how to weigh a fact, or how strongly the user believes in something. It may downplay something sensitive, or bring up minor details the user said in passing.

What a better system needs to do

Given that users expect chatbot memory to feel human-like, it's reasonable to look at how human recall subjectively behaves as a design inspiration. A fitting analogy to draw on are the parallels between the two types of memory in humans and LLMs. Humans have a short-term memory, allowing us to recall information from the recent past, such as memories from the same day. Short term memory is analogous to the context window of an LLM, within which the model can surface details exactly as they occurred. Similarly, Long term memory in humans — the ability to recollect information from weeks, months or years in the past is analogous to what a good cross conversation memory system in LLMs should look like.

The key features of Long term memory in humans (from my experience as a human) are:

1. We as humans can recall past events exactly as they happened.
2. Recollection is a passive task. We recollect information based on the situation we are in, and the triggers that cause it.
3. Recollection is a chain event. We can walk through a chain of related but distinct memories based on their relationships and relevance.
4. Memories are temporal. Old memories fade and decay when they are not recollected, but are strengthened when recalled frequently. The number of times a memory is recalled is in itself a signal for how important it is.
5. It can be hypothesized that the transfer of relevant memories from short term to long term memory happens during sleep.

A good system for LLM memory should draw on these principles, as they solve the key problems mentioned previously.

The Core Idea

Based on the features of human recollection mentioned above, the form of a memory system that incorporates them points towards a knowledge graph. Here's why:

1. A knowledge graph for this purpose can be built by storing nouns (people, places etc.) as nodes, and verbs as edges, capturing relations between different entities the user comes across. The memory is in the relation, but nodes act as anchor points for easy recollection. It is easier to identify a person on a knowledge graph than to identify a specific memory in a large sea of text.
2. As memories are the relations stored along edges, the importance of a memory can be determined by the weight of the edge. Edges traversed frequently can be made to have high weights, and otherwise, edges can be made to decay with time.
3. If edges between nodes act as memories, then traversal across these edges is recollection. Since related nodes are naturally linked together in a knowledge graph, traversal between related ideas allows chain recollection and captures the context around a specific memory as adjacent edges are traversed.
4. The use of an embedder to automatically trigger a traversal of an edge during a conversation, allows recall to happen automatically, rather than needing the model to call a tool to read the graph.
5. Once built, a knowledge graph can be expanded simply by adding more nodes and edges, rather than requiring a full rewrite every time. The graph can grow indefinitely, without constraints on graph size, because only relevant edges are surfaced during the conversation with the user.
6. By attaching episodic snippets of the exact prompt from which a relation was created to the edge, and loading this into the context window of the LLM during a conversation when the associated edge is traversed, memory recollection is exact, capturing the user in their own words rather than allowing the misinterpretation of a summary.

The Architecture

How the system works in practice

At a high level, this is how the memory system works:

1. The user has their first conversation with the LLM.
2. After the conversation is completed, the text from the full conversation transcript is compiled together and passed to another LLM.
3. This LLM extracts entities from the text (PERSON, ORG, PLACE, EVENT, THING, TOPIC, PREFERENCE, OTHER) and links them via the relations between them. This collection of entities and relations forms a map of the LLM's memory.
4. A snippet of the text from which a relation was extracted is attached to that relation, to provide context.
5. A knowledge graph containing entities as nodes and relations as edges is built out. A self node for the user is designated on first write and used to resolve placeholders.
6. When the user has their next conversation with the model, an embedder scans the user's text at every turn to find words semantically similar to entities and relations on the graph.
7. When a match is found, activation spreads up to 2 hops in both directions from those seed entities, weighted by edge strength, and the strongest neighbourhood is injected into context as relation lines (Entity-Relation-Entity strings) plus the original snippets.
8. The LLM uses this accumulated context to reply to the user.
9. After the conversation completes, the existing graph is appended to and grown further. New nodes and relations are added based on new information from the user. Duplicate nodes are merged and ambiguities resolved. Relations that go unused over time fade in strength, while ones that keep coming up are reinforced.
10. The process repeats.

Each part

The Schema

Memory in this system is a graph, with 3 main parts:

A node is a referent: a person, organization, place, event, thing, topic, preference, or other entity the user has mentioned. There are only those eight kinds, kept fixed so the same idea is always filed in the same way. A node holds a name, alternate names, a small set of stable properties, a confidence score, and a list of which conversations it appeared in. It holds no strength and does not decay. It exists so there is something to attach memories to and search from.

An edge is the memory itself: a single claim linking two nodes, such as Bill → HAS_BOSS → Sarah. Relations are always uppercase, verb first phrases of one to four words. Each edge carries the claim's strength from 0 to 100, the extractor's confidence from 0 to 1, a stability class that sets how fast it should fade (immutable, stable, mutable, time_bound, ephemeral), and a cardinality (one_to_one versus one_to_many) that records whether the relation can truthfully point to more than one target. Only temporary relations can carry an expiry. Nothing is allowed to relate to itself.

The snippet is what makes that claim admissible, and what keeps it honest. Every edge must cite the utterance it came from: the user's own words, stored verbatim on the edge alongside a shorter evidence field. The edge does not describe the snippet; the snippet justifies the edge. Without it, there is no reason for the relation to be there at all.
More importantly, the snippet is what prevents memory from collapsing into a summary. A bare relation like HAS_BOSS says almost nothing about how that fact was expressed. Whether it was said with frustration, with affection, as an aside, or as something dwelled on. The snippet preserves that flavour: the tone, the emphasis, the surrounding context of the sentence it came from. It also carries a useful robustness. The verbatim text inevitably holds details the extractor did not promote into nodes, edges, or properties, but which still matter when the memory is recalled. By holding the snippet, the edge holds on to what structured extraction would otherwise throw away.

Extraction and Graph Construction

Graph construction happens once per conversation, after it finishes. At the end of a conversation, the full transcript is gathered and handed to a language model with one job: propose what should be remembered from it.
The model is asked to return three things in a fixed shape: the entities mentioned, the relations between them, and a short record of the conversation itself including a summary, an importance score, and a few tags.
To help it avoid inventing duplicates, it is also shown the relevant part of the existing graph: the neighbourhood around anything in the current conversation that resembles something already known. That context is found the same way retrieval works, by matching the conversation against what is already stored, except here it is deliberately broad. At write time we would rather show the model too much than too little. A deterministic pass then cleans what the model returns. Casing is corrected, unknown stability labels fall back to stable, and expiry dates attached to permanent relations are removed. What cannot be repaired is dropped with a note rather than allowed to corrupt the graph.
Relations that point to themselves, relations whose labels are malformed, and relations where neither end refers to anything known do not make it into the graph. When exactly one end is unknown, a provisional node is created for the missing side and marked as low-confidence, so the relation and the concept it points to are preserved rather than lost.
An empty result is valid. Not every conversation contains anything worth keeping.
The outcome is a small, checked batch of new nodes and edges, each edge carrying its verbatim snippet and each node and edge citing the conversation it came from. Nothing is merged or decided here. That batch is passed forward to merge and disambiguation, which decide what is genuinely new and what is a new sighting of something already known.

Merge and Disambiguation

Every new conversation reintroduces people and ideas the system may already know. Extraction proposes names; merge decides whether each one is genuinely new or another sighting of something already on the graph. Merging contains a meaningful asymmetry: merging two different people destroys both. There is no clean way to split them apart later. Missing a merge only creates a duplicate, which can still be fixed. The whole stage is built around that asymmetry: merge quickly when certain, hesitate whenever unsure.

The first pass is deterministic and needs no model. Each proposed entity is compared only against stored entities of the same kind, so a person named Sam can never collapse into an organization named Sam. Within that set, rules run in order.

1. An exact match on name and kind merges immediately.
2. If the new name appears in an existing node's list of alternate names (aliases), it merges — unless two different nodes both claim it, in which case it is held for review.
3. The same holds for common abbreviations, drawn from a fixed set of groups like William, Bill, and Will: one match merges, several matches mean ambiguity.
4. Weaker resemblances never merge on their own. A spelling similarity above 0.80, (measured as one minus normalized edit distance), and a word-overlap test with one name containing the other, or sharing at least half their words each nominate candidates for a second look. If nothing fires at all, the entity is new.

Generic self-references are the one exception to this caution. When the extractor does not yet know the user's name it emits placeholders like User or me. Since there is structurally only one user per graph, these are forced directly onto the designated self node rather than risking a new placeholder per conversation. That self node itself is chosen once, from the source of the first relation ever observed, on the convention that relations point outward from the user, and then kept fixed.

Everything held for review goes to a language model with richer context: the proposed entity, the conversation it came from, and each candidate's surrounding neighbourhood on the graph, so shared employers, family, and places count as evidence of identity.
The model returns a confidence per candidate, and a fixed banding decides: above 0.90 merges silently, from 0.70 to 0.90 merges but is flagged for inspection, below 0.70 stays separate. The highest qualifying candidate wins. A soft hint from the extractor, its own guess at which existing node this might be, can only escalate an entity into review, never merge it directly.

Committing the decision is mechanical. A new node is added with its source conversation attached. A merged node keeps its existing identity and absorbs the new sighting:

• the new surface form becomes an alias;
• properties are combined;
• the conversation is cited; and
• confidence becomes the higher of the two.

Edges are then remapped through the same decisions, so a relation pointing at a merged-away name now points at the surviving node. If both ends of a relation land on the same node after remapping, it is dropped. The graph never allows self-relations.

Decay and Reinforcement

Memory here is not stored and left alone. Every relation carries a strength from 0 to 100 that moves over time: it fades when ignored and recovers when used. Nodes never fade. Only relations do, because only relations are memories.
Fading follows a simple exponential curve. The strength after time is the starting strength multiplied by a decay factor that shrinks with each passing day. How fast it shrinks depends on the stability assigned at extraction. Immutable relations do not fade at all. Stable ones fade very slowly, mutable ones faster, and temporary ones fastest of all, with expiry dates allowed only on the last two. A daily pass recomputes every edge from the time since it was last touched. When strength falls below a small threshold, the edge is marked dormant. It is never deleted. A faded memory stays in the graph as provenance, and a later conversation can always revive it. It simply stops competing for recall until then.

In exact form, with $t$ measured in days since the edge was last touched:

$$ S(t) = S_0 \cdot e^{-\lambda t} $$

where $S_0$ is the strength at last touch (new edges start at $100$), and $\lambda$ is set by stability:

stability	$\lambda$
immutable	$0.000$
stable	$0.005$
mutable	$0.020$
time_bound	$0.050$
ephemeral	$0.200$

Reinforcement is the opposite motion and comes from two places. The first is repetition at write time: if a new conversation observes a relation that already exists, the existing edge is strengthened by a fixed amount, capped so it can never exceed its starting value, and its snippet is refreshed only if the new sighting arrived with higher confidence. The second is use at read time. Every read logs which edges its traversal crossed, without changing them. At the close of the conversation those logged edges receive the same fixed bump. In other words, a memory grows stronger both when it is restated and when it proves useful for understanding something else. The separation matters. Confidence records how sure the system was when it first heard something and never moves on its own. Strength records how alive that thing has stayed since, rising on reuse and falling with neglect. A childhood address can therefore remain high-confidence but long dormant, while a new colleague's name stays weak until it keeps coming up. Time, frequency, and usefulness are all visible in one number, without ever rewriting what was originally said.

Reinforcement, on re-observation or on read-traversal, is:

$$ S_{\mathrm{new}} = \min(S_{\mathrm{old}} + 15,, 100) $$

An edge with $S < 2.0$ is flagged dormant and never deleted.

Retrieval and Traversal

Retrieval answers one question per turn: given what the user just said, what small piece of the graph deserves to be in the model's context right now. This is one of the key advantages to using a graph to store memory — the graph can grow indefinitely, but only those parts of the graph that are relevant to the conversation at hand make it into the model's context window. Retrieval never changes the graph. It only walks it, then logs where it walked so a later write can strengthen those paths.
It starts with words, not vectors. The incoming message is split into content words — lowercase tokens with function words and single letters removed, deduplicated. This is not linguistic analysis; it just concentrates the signal on nouns and other load-bearing words. Each remaining word is embedded and compared against everything stored. Every node is indexed by its name plus aliases, and every distinct relation label is indexed once as a phrase, so partner can match HAS_PARTNER even when no node is named partner. All vectors are length-normalized, so similarity is a dot product: $$ \mathrm{sim}(q, k) = q \cdot k $$ Because embedding spaces are anisotropic — random pairs already sit well above zero — the comparison is done after subtracting the corpus mean. With $\mu$ the mean of all stored vectors: $$ q_c = \frac{q - \mu}{\lVert q - \mu \rVert}, \quad M_c = \mathrm{normalize}(M - \mu), \quad \mathrm{score} = M_c , q_c $$

This pulls the unrelated baseline back toward zero so a threshold means something. A word seeds on every hit above $0.30$, plus one margin case: a top hit above $0.15$ that beats its runner-up by at least $0.15$ is accepted as a clear winner. Relation hits resolve to the endpoints of every edge carrying that relation. Stale keys are dropped. Activation then spreads from all seeds at once, up to two hops, in both directions along each edge. A node does not hand its full activation to every neighbour — it divides it in proportion to edge strength, so a heavily connected hub cannot flood the graph: $$ \mathrm{contrib}(u \to v) = \mathrm{act}(u) \cdot \frac{\mathrm{strength}(u,v)}{\sum_{e \ni u} \mathrm{strength}(e)} \cdot 0.6 $$ Contributions arriving by different paths add up. Each node's total is remembered along with whether it was reached mostly forward (following edges out), mostly in reverse (following them in), or as a seed, and at which hop. Every edge crossed is logged for deferred reinforcement. Two adjustments follow:

1. First, a small co-episode bonus: any node that shares a source conversation with an already-fired node ($\mathrm{score} \ge 0.30$) gains $+0.15$, capturing the idea that things mentioned together belong together. Seeds are exempt — they need no bonus.
2. Second, a budget keeps injection terse. Only nodes at or above $0.30$ fire, capped at eight, split roughly $70/30$ between forward-or-seed and reverse, with leftovers backfilling whichever side has room. Injection itself is tiered. Fired relations between fired nodes are rendered as compact lines — $Bill -[$HAS_BOSS$]\to Sarah$ — forming the $[MEMORY]$ block. Then, for nodes scoring at or above $0.60$, up to four of their own strongest incident snippets each are added as the $[CONTEXT]$ block, verbatim and deduplicated. Superseded edges are hidden in both. The model therefore sees structure first and texture only where the signal is hot — enough to recall precisely, without re-reading everything known about someone every turn.

The analogy to synaptic activation

The resemblance to synapses in the human brain is deliberate but partial.
It holds at the level of dynamics, not mechanism. What holds is the use–dependent loop. An edge that fires, crossed during retrieval is later strengthened, and an edge that never fires decays exponentially toward dormancy. That is Hebbian in spirit: what is recalled together stays recallable, what is ignored fades, and frequency of use becomes its own measure of importance. The co-episode bonus is the same idea one step wider: nodes introduced in the same conversation lend each other activation, the graph version of firing together wiring together. Degree normalization plays the role of a limited resource: a hub cannot drive all its neighbours at full strength, so only genuinely convergent paths survive, much as lateral competition constrains biological spread. Even dormancy without deletion has a neural echo — a weakened trace that can be reactivated is closer to depotentiation than to pruning.

What does not hold is almost everything structural. A biological weight is a continuously remodeled conductance shaped by spike timing, calcium, neuromodulation, and sleep-dependent consolidation. Here strength is a single scalar per edge, moved only by two discrete events: a fixed $+15$ bump on reuse, capped at $100$, and a daily exponential multiply set by a hand-chosen stability class. There is no gradient, no timing dependence, no inhibition, and no learning inside retrieval itself. Activation is a transient walk computed fresh each turn, then discarded except for a log of which edges were crossed. The hard cognitive work deciding whether two mentions are the same person, whether a new relation replaces an old one is not done by the network at all but by a language model reasoning over neighbourhoods, with deterministic rules guarding the cases where a mistake would be unrecoverable. Symbols do what weights cannot here: keep identity, tense, and contradiction explicit.

The graph borrows the economy of synaptic memory: strengthen the useful, let the rest fade, never pay for what is not recalled while keeping meaning itself symbolic and inspectable. Activation behaves like recall; it is not learning. Learning happens only at write time, in daylight, one checked batch per conversation.

An example showcasing the system in action:

Here is a simple 2 turn conversation demonstrating extraction and recall with this system:

Note: This conversation happened on Opencode with GLM 5.3 using the pegasus-opencode npm plugin. Some input text was minimally modified before being pasted here to correct spellings and grammar.

The conversation

.

.

The knowledge graph produced by our system from this conversation

Testing Recall

Here is the fun part. With a system that captures every data point, you can start conversations cold, without any context, and the model will stick pick up right where it left off:

(This happened in a fresh conversation. We bring up the manager by name, the specifics of the role, and a personal preference that was captured in the snippets)

(The model picked off right where it ended, and even brought in specifics such as its concerns about the startup's burn rate and other economics)

(The model understood the question, brought up the $500,000 specifics, and even remembered niche expressions that the user mentioned previously like "Skin in the game", while continuing the previous conversation fluidly)

Everything that Broke, and learnings

The choice of extractor model

During the first few rounds of graph construction, I used a lightweight model without reasoning capabilites in an attempt to make the extraction as token efficient as possible. This was mistake. With the model unable to reason and deliberate as to what information really deserved to make it into the graph, and double check its work, the quality of the graph produced was poor, with important user specific information never making into the graph, edges being invented for facts that should have been properties of an existing edge, and absence of second order edges between two nodes that were connected to the user node, but also connected to each other.

On swapping the model for a reasoning model, the quality of output for smaller conversations went up immediately. The model identified all important facts, thought through higher order relationships, and also differentiated between information that was user-specific (and worth adding to the graph), vs universal facts that are known by LLMs already. However for longer conversations, the model thought for large amounts of time — sometimes up to 15 minutes and timed out without any output. One reason behind this was me sending the entire conversation transcript to the extraction model, and including the agent's turns, to provide context around why the user responded the way that they did. As the transcript got very long, the bookkeeping of facts that the model had to do became overwhelming, and it simply could not construct the graph in time.

However, even after sending only the user's turns to the model, graph construction still took a long time (5-7) minutes before a suboptimal graph was produced. Looking at the reasoning logs of the model, I discovered that it essentially tried to build up the entire graph in its "head" (its reasoning trace) before commiting the entire graph all at once to the database. To do so, it would keep repeating the entire graph's schema every time it updated it, modifying it in its head as it made changes. The solution to this problem was to instruct the model to write every node and edge into the graph the moment it considered them, and modifying or appending to them in the graph later if new information about them was revealed later on in the conversation.

This approach provided an extra layer of redundancy. If the model was cut off midway, its work wouldnt be lost, and it would be able to read the graph and pick up from where it left off when resumed. Both of these changes reduced extraction time for a reasonable conversation (with 10-20 turns) to around 2-3 minutes which was far more reasonable.

Overfitting to the system prompt

Another mistake that I made was including examples of previous extraction failures in the system prompt of the extractor model to demonstrate common failure modes and get the model to avoid them. This didn't work, and caused the model to overfit to the system prompt. It produced the right output when the exact example mentioned in the system prompt was tested, but failed when a different version of the same example was tested.

However, by generalizing the instructions to the model, and having it reason over whether it was following those instructions, the quality of output got a lot better in general.

Attempting to build the model harness myself

To prototype early iterations of this system, I figured that if I couldn't reliably connect and test this memory system with chatbots on the web, it might make more sense to build a simple model harness locally on my laptop. I built one and used a free Gemini API key from Google AI Studio to set up a simple chat interface. This was not a great idea in hindsight. I wasted a lot of time building the harness and being unsatisfied with the output. To make things worse, I tightly coupled the memory system and harness together, to a point where I was spending more time trying to fix bugs in the harness than actually iterating on the memory system. At times, it was hard to locate the source of errors. Was there a mistake in how I built the harness, or how I connected it to the memory system, or in some part of the memory system itself? This was unclear.

Eventually, I switched to Opencode, which should have been my first pick. Being open source, I was free to tinker with it as I liked. Fortunately, I didn't have to, and was able to build the memory system to be modular and work as a plugin with Opencode rather than requiring any deep integration. This also made it easier for others to try the system out as it reduced installation times, and was somewhat straightforward to setup as a plugin.

Open Problems

Extraction is still an active process

The central principle behind building this system was to use the dynamics of memory in humans as a design inspiration for our architecuture. A key deviation from this principle however, is that the consolidation of memories into the graph in this system is still an active process that uses another LLM, unlike the passive consolidation that happens in humans — presumably when we sleep.

I don't fully understand how consolidation of memories works in human brains, but I believe that it is pretty evident that we don't reason about our day when we sleep and chose to write things into our memory. Using an LLM to build the knowledge graph and decide what goes in and goes out is akin to that.

Perhaps, in later implementations of this system, we will be able to connect the consolidation process more closely to the transformer running in the chat session itself, marking facts as important and worth remembering during the chat session rather than after it. Possibly, we may even be able to drop the knowledge graph with its engineered decay rates and artifical construction altogether, in favour of something even more natural, like a dedicated neural network that is small enough to learn important details continously on its own and act as a storehouse of information. In that context, phenomena like catastophic forgetting that is common with continual learning may be viewed as a feature rather than a bug, similar to how humans forget old memories to make room for new ones. However at this point these are all speculations, and I have no experimental backing to suggest that such an architecture is feasible.

The system is currently untested at very large sizes

As of my writing of this writeup in September 2026, I have not yet been able to test the graph at extremely large sizes and observe its behaviour when holding extremely large amounts of information. It shows promise compared to sumary based memory systems, and recalls specific details far more accurately than providers on the internet when tested over a small number of conversations (10 - 15 turns over 4-5 conversations), but unexpected behaviour is possible at larger sizes. I plan on writing a follow up to this writeup in the near future after I have stress tested the system for its failure modes.

Gimmicks with the Opencode Plugin

The opencode plugin for this system is registered on the npm registry under the name "pegasus-opencode". The current implementation of this plugin (version 0.2.0) has a few issues that need to be worked out, such as the first write to the graph often taking very long (3-4 minutes), and possibly some other issues that I haven't hit yet. Working on resolving these as quickly as possible.

Trying it yourself

Since the nodes and edges of the graph resemble a constellation of stars, I've decided to call this system "Pegasus", although in most parts of the code, you will simple see it referred to as "onto-mem" or some variation of that name.

The entire source code for this project is open-source with an MIT license at my Github: Pegasus - Github

Currently, this engine as a plugin only exists for Opencode, (preferrably with a go subscription) but I am working on making it available for Claude Code and Codex as well.

To install for Opencode, run:

npm install pegasus-opencode
opencode plugin pegasus-opencode