AI Agent Memory: Short-Term vs Long-Term vs Vector Storage

Why memory is the hardest part of agent design

A stateless chatbot is easy. A stateful agent that remembers your last conversation, your account preferences, your company's internal documents, and what it decided three steps ago in a complex workflow — that is a systems problem. And it is the problem that separates agents that feel genuinely useful from ones that feel like amnesia-with-extra-steps.

Every memory decision — what to store, how to retrieve it, when to summarise it, how long to keep it — has consequences for cost, latency, accuracy, and privacy. The best agents we ship make these decisions explicitly. The worst ones stuff everything into context and hope for the best.

The five types of AI agent memory

Cognitive science distinguishes memory by duration (short-term vs long-term) and by content (episodic, semantic, procedural). Modern AI agent design borrows this taxonomy directly because it turns out to map well to the problems agents actually face.

Short-term memory (the context window)

Short-term memory is what is in the LLM's context at this exact moment — the system prompt, the recent conversation, any retrieved content, and the tool outputs so far. In 2026 context windows range from 128K to 2M tokens depending on the model, which sounds unlimited but isn't. Every token costs money, raises latency, and gives the model more opportunity to get confused.

The three problems with relying on context alone

1. Cost. A 100K-token conversation replayed 20 times a day at USD 3 / 1M input tokens is USD 6 per user per day — unsustainable at scale.
2. Lost-in-the-middle. Even frontier models lose accuracy when relevant information is buried in a long context.
3. Session-bound. Context disappears when the session ends. The agent has amnesia tomorrow.

Best practices for short-term memory

• Keep context lean — only what is genuinely relevant to this turn.
• Summarise aggressively once a conversation exceeds ~8K tokens.
• Structure the context: system prompt → user profile → retrieved facts → recent turns → current turn.
• Never blindly prepend the full history — compress.

Long-term memory

Long-term memory is where the agent stores what happened after a session ends, so the next session doesn't start from zero. The standard pattern: at the end of each session (or at periodic checkpoints), the agent produces a summary. That summary is stored, indexed, and retrieved when the user returns.

What to store in long-term memory

• A running summary of the user's history and preferences.
• Specific commitments the agent made (follow-ups, promises, action items).
• Structured facts extracted from conversations (user name, company, role, preferences).
• Outstanding tasks and their state.

What NOT to store in long-term memory

• Full verbatim transcripts (expensive, privacy risk, low value).
• PII you would not put in a database.
• Information that has a short shelf life — today's weather, transient preferences.

Episodic memory

Episodic memory stores specific events the agent participated in — a conversation with a customer on Tuesday, a support ticket resolved last month, a meeting scheduled last quarter. When the user says "remember when I asked about X?", the agent can retrieve the actual episode.

How episodic memory is typically implemented

1. At the end of each interaction, generate an embedding and metadata (timestamp, user ID, topic tags).
2. Store in a vector database with the rich metadata attached.
3. On retrieval, use hybrid search — semantic similarity + metadata filter (user, date, topic).
4. Inject the top 3–5 most relevant episodes into the next turn's context.

Semantic memory

Semantic memory is the agent's knowledge — not events, but facts. "The customer's account is on the Premium plan." "Our refund policy is 30 days." "The engineering lead for this account is Jamie." Semantic memory is usually the highest-value memory type because it lets the agent act like it knows your business.

Three layers of semantic memory

• User semantic memory: facts about the individual user — name, preferences, role, history.
• Organisational semantic memory: facts about the customer's company — plan, usage patterns, key contacts.
• Domain semantic memory: facts about the world the agent operates in — product docs, policies, SOPs.

Most semantic memory flows through retrieval-augmented generation (RAG). The agent queries a vector database, gets the most relevant facts, and injects them into context. For a deeper look, read how to build an AI agent.

Procedural memory

Procedural memory is learned behaviour — the agent's equivalent of muscle memory. When an agent has processed 10,000 refund cases, it has implicit knowledge about what good looks like. Procedural memory is usually captured in one of three ways:

1. Fine-tuning on examples of high-quality past behaviour.
2. Few-shot examples pulled dynamically into the prompt as reference.
3. Tool policies that encode learned decision rules as code.

Procedural memory is the slowest to build and the most expensive to change — but it is often the most defensible. It is what makes an agent that has been running in your business for a year feel fundamentally different from one you stood up yesterday.

Vector databases in 2026: the landscape

Vector databases store and search embeddings — dense numerical representations of text, images, or other content. They are the workhorse of semantic and episodic memory. The 2026 landscape has consolidated around four leaders.

Our default at Bananalabs is pgvector unless scale or feature requirements push us elsewhere. One database is better than two. If the client is already on Pinecone or Weaviate, we stay there. The product rarely decides; the context does.

Production memory patterns we actually use

Pattern 1: The summary + retrieval hybrid

Our most common pattern. Each session ends with a two-paragraph summary written by the agent. Summary is stored in a SQL row per user. For each new turn, we retrieve the summary plus the top 3 relevant past episodes via vector search. Context stays small; memory feels deep.

Pattern 2: Structured user profile + RAG

For agents where structured facts dominate (B2B sales, account management), we extract user/org facts into a structured profile (Postgres). The agent reads the profile directly into context. RAG layers on top for less structured knowledge like past conversation excerpts.

Pattern 3: The working memory sketchpad

For long autonomous runs (research agents, multi-step ops agents), we give the agent a "sketchpad" — an editable scratch buffer it uses to track state across steps. This is technically short-term memory but persists across LLM calls within a single task. It is how we avoid the agent re-deriving its own conclusions repeatedly.

Pattern 4: Episodic decay

Not all past episodes deserve equal weight. We apply time-decay to retrieval scoring, so last week's conversations rank higher than last year's unless they are explicitly relevant. This prevents the agent from dredging up stale context.

Pattern 5: Opt-in persistent memory

For consumer agents, we make long-term memory an explicit user opt-in with clear controls — what is stored, how to view it, how to delete it. This is a privacy best practice and, post-GDPR, a legal one.

The real cost of memory

Memory seems like a "cheap" feature until you run the math on retrieval-per-turn at scale. Every turn with 5K tokens of retrieved context costs meaningfully more than a turn with 500. Multiply by users and turns per day and memory becomes the dominant line item.

• Embedding storage: pennies per user per month for typical volumes.
• Vector search compute: nearly free at <10M vectors.
• Retrieval tokens in context: the real cost — often 30–60% of total token spend.
• Summarisation calls: cheap if you batch them, expensive if you summarise every turn.

Our rule of thumb: budget 30–50% of total LLM spend for retrieval tokens. If you are not measuring this, start today. For the full cost picture, see the hidden costs of building AI agents.

Memory pitfalls to avoid

1. "Just put everything in context"

The lazy approach that breaks at the second user and the first scale. Long context ≠ memory.

2. No summarisation pipeline

Conversations grow unbounded. Token costs climb. Accuracy drops from lost-in-the-middle. Summarise at logical breakpoints.

3. One retrieval strategy for everything

Semantic search alone misses structured facts. Keyword search alone misses paraphrase. Hybrid search is not optional in production.

4. Storing PII you cannot delete

GDPR requires the right to be forgotten — including from vector indexes. Plan deletion from day one.

5. Infinite retention

Old memories get less accurate over time. Define decay or TTL on every memory type.

6. No memory observability

If you cannot see what the agent retrieved, you cannot debug why it answered wrong. Log every retrieval with the query, the result, and the ranking signal.

7. Treating memory as a database problem only

Memory is an information architecture problem. The vector DB is the easy part. The design of what to remember, when to retrieve, and how to summarise is the hard part.

Memory decision framework

Does the agent need state between sessions?

• No → short-term only. Simpler, cheaper.
• Yes → at minimum, summary-based long-term memory.

Do users expect the agent to recall specific past events?

• Yes → add episodic memory via vector store.

Does the agent need domain knowledge or org-specific facts?

• Yes → semantic memory via RAG.

Will the agent handle thousands of similar cases where consistency matters?

• Yes → procedural memory via fine-tuning or dynamic few-shot.

Do you have regulatory or privacy constraints?

• Yes → plan memory with deletion, audit, and consent primitives from day one. See AI agent security.

The bottom line on AI agent memory

The agents that people love are the ones that feel like they know you. That feeling is not magic — it is a memory architecture with short-term context, distilled long-term summaries, retrievable episodes, structured facts, and learned patterns, all orchestrated cleanly. Teams that get memory right end up with agents that get smarter over time. Teams that skip it end up with agents their users stop trusting.

If memory design feels like a lot — because it is — that is exactly the kind of problem a specialist partner solves for you. Our design baseline at Bananalabs includes a multi-tier memory system out of the gate. The rest of the agent is downstream of getting that right.