Agent memory is the engineering layer that gives a stateless language model continuity. It's the set of systems that decide what an agent stores, what it retrieves, and what it discards, so the model can act on something that happened thirty seconds ago or thirty sessions ago. The model itself remembers nothing. Memory is everything you build around it.
That distinction is the whole game, and most teams get it backwards. They treat memory as a model capability they're waiting on, or a vector database they can buy. It's neither. It's state management, the same class of problem you've solved before in distributed systems wearing a new hat. We built the AI SDK, Fluid compute, and Vercel Workflows around that premise. This piece covers how agent memory actually works, where the common patterns break in production, and why durable execution changes the architecture more than most developers expect.
Copy link to headingWhy stateless models make memory an engineering problem
Every request to a language model arrives cold. No history, no preferences, no record that it already finished step two of a five-step job. The context window, the tokens you send alongside each prompt, is the only thing the model can see, and it behaves like RAM, not storage. When the request ends, it's gone.
I've watched this produce the same three failures across enough deployments that they're predictable. A support agent asks the user to repeat the account number they gave two messages ago. A research agent re-runs a search it already completed, because nothing told it the work was done. A multi-step workflow loses its place when a serverless function times out mid-run. None of these are model-quality problems. You can swap in a smarter model and hit every one of them. They are state management problems, and they get solved in the systems around the model or they don't get solved at all.
Vercel's documentation frames the two tiers plainly: short-term memory holds the current conversation in the context window and clears when the task ends, long-term memory uses external storage like vector databases to persist across sessions and is queried through RAG when historical context is needed. That framing is correct, but the interesting engineering doesn't live inside either tier. It lives in the space between them, in the decisions about what to keep, what to compress, and what to discard. Get those wrong and the agent either forgets what mattered or pays to carry everything that didn't.
Copy link to headingShort-term vs long-term memory
The two-tier model is the right place to start, so it's worth making concrete. Short-term and long-term memory are not two settings of the same thing. They are different storage systems with different failure modes, and they fail in opposite directions.
Short-term memory is fast and coherent and fails by overflowing. Long-term memory is durable and fails by going stale. Almost every architecture decision in this piece is really about managing one of those two failure modes.
Copy link to headingThe four types of agent memory
Short-term versus long-term is the top-level split, but it isn't granular enough to design against. The taxonomy most teams have converged on, drawn from cognitive science, breaks memory into four types. Working memory is the short-term tier already covered, whatever is live in the context window. The other three are all forms of long-term memory, and they answer different questions.
Semantic memory is facts and preferences, the kind of thing a vector store is built to hold. Episodic memory is specific past events, usually logged with timestamps so the agent can recall what happened and when. Procedural memory is learned behavior, the prompts and tool definitions that encode how the agent does its job. Most agents in production only have working memory, which is why they feel competent inside a single session and amnesiac across them. The design work is deciding which of the other three an agent actually needs, and wiring each to storage that fits its retrieval pattern.
Copy link to headingWhen you actually need long-term memory
Here is the call that saves teams the most wasted effort: most agents don't need long-term memory on day one, and building it first is the classic premature-infrastructure mistake. If your agent finishes a task and the next task is independent, short-term memory in the context window is the entire system. You reach for long-term memory when the agent has to carry something across sessions that it can't reconstruct from the current request, a user's stated preferences, the result of a job it ran last week, a fact it learned that should survive a restart. The test is simple. If losing all persisted state between sessions would cost you a few seconds of recomputation, you don't need long-term memory yet. If it would cost you correctness, you do.
Copy link to headingWhy bigger context windows don't fix it
The default everyone reaches for first is the simplest thing that works: append every message to the history and send all of it with each request. No infrastructure, no retrieval latency, maximum coherence. For a five-message chat, it's the correct choice. For a fifty-step agent workflow, the math turns on you.
In my experience, the bill is where this lands first, before anyone notices a quality problem:
A Shaped AI analysis puts context-window stuffing around $0.16 per query at 50,000+ context tokens, against $0.04 to $0.08 for a vector database with reranking at 5,000 to 10,000 tokens.
One developer in a practitioner discussion estimated minimal RAG infrastructure north of $70,000 annually at 850,000 monthly conversations, with full context stuffing costing multiples of that in their setup.
The input side is only half of it. A Redis token analysis argues output tokens can run three to eight times the cost of input tokens, so every unnecessary token you carry compounds on the response.
It gets worse inside multi-step loops, because each tool call adds messages to the history and the history travels with every subsequent call. One developer in a Hacker News discussion described splitting work across invocations as something that "grows in cost superlinearly, as you'll have multiple results you need to summarize to feed into the next stage." Caching helps less than people hope. If an agent always opens with the same twenty-page system instruction, a token economics analysis notes that provider caching can cut cached-prefix cost and latency when the repeated prefix is static. But caching only touches the static prefix. The growing tail of conversation history is exactly the part it can't help you with.
We addressed this in the AI SDK with prepareStep, a callback that receives the step number and the full message history and lets you compress context for longer loops. The AI SDK docs show the basic move: once messages exceed twenty, return only the last ten. A community-contributed context middleware pattern goes further. When the conversation passes the token budget, it summarizes older messages with a small model, preserves the recent ones, truncates large tool results to the first 500 and last 1,000 characters, and trips a circuit breaker that stops compression after three consecutive failures so a bad summarization pass can't crash the session.
This is the part most teams resist, so I'll state it plainly: a bigger context window does not solve the memory problem. Research on long-context retrieval frames the job as deciding if, what, where, and how to retrieve, not filling every token you're allowed. A model with a million-token window still pays for every token you put in it, and still loses the thread when the signal is buried in noise. Context quality is the lever. Context size is the bill.
Copy link to headingThe write side breaks first
Across our community forums and production deployments, one pattern repeats: the write side breaks before the read side does. The hard part of memory management is getting the agent to know what's worth remembering and how to store it. Practitioners make the same point in a thread on memory write quality, where the consensus is that retrieval is relatively easy compared with deciding what should become memory in the first place.
This is why a vector database is not a memory system, even though it gets sold as one. A vector database answers "what is similar to this?" An agent memory system has to answer a harder question: "what does this agent know, and is it still true?" An Atlan analysis lays out the gap directly. Vector databases don't reason about time, don't track contradicting facts, and have no concept that a piece of information has been superseded. Retrieval, including RAG, is how stored knowledge gets read back into the prompt. It is not the memory system itself, and conflating the two is how teams end up with a vector store they mistake for memory. In a production thread, practitioners described recall quality decaying over time, with agents confidently citing context that was relevant at first and now contradicts the current state. One team's mitigation was to track access frequency and demote cold facts that hadn't been touched in two-plus weeks, which they reported cut context pollution by roughly 40%.
The honest answer is that no single store is enough, and that's why the dominant production pattern in 2026 has moved to hybrid architectures combining vectors, graphs, and episodic buffers, a shift also described in temporal memory research. Graph memory retrieves facts through entities and relationships. Vector memory retrieves semantically similar content. Each is good at what the other is bad at. The tradeoff is operational complexity, because now you're running and reconciling more than one store, and you own the logic that decides which one answers a given query.
Be skeptical of the numbers vendors attach to this. Hybrid memory providers like Mem0 report large token savings in their own materials, cutting per-query tokens to a fraction of the roughly 26,000 a full-context conversation can consume, in a Mem0 report, but a benchmark audit found enough inconsistency across memory evaluations to justify treating any single vendor claim with caution. The AI SDK integrates with providers like Letta, Hindsight, and Supermemory, each specializing in a different slice of the memory problem, from archival memory to mental models. Which one fits is really a question of which of the four memory types you most need to persist. The memory integrations let you pick the layer that fits the use case, and the SDK handles the message plumbing underneath. The point isn't which provider wins. It's that the write decision, what becomes memory, stays yours to design.
Copy link to headingWhy serverless and stateful agents are mismatched
Serverless functions and stateful agents want opposite things, and pretending otherwise produces the worst class of agent bugs. A function spins up, handles a request, and dies. An agent needs to hold state across dozens of steps, survive a failure partway through, and resume hours later. My test for whether an agent is production-ready is blunt: if you can't kill it mid-task and restart it without corrupting your database, it isn't ready.
The failure modes are specific, not theoretical. An agent running a four-step sequence that dies on step three can orphan the data it wrote in steps one and two, retry infinitely and duplicate records, or corrupt the steps that come after. Cold starts make it worse the longer the chain gets. Blaxel reported in a third-party benchmark 750ms of total cold-start latency across a five-step chain against 250ms warm, and that penalty multiplies with every tool call you add.
We built Vercel Workflows as the architectural answer to this. Adding use workflow turns reliability into code by automatically retrying failed steps, persisting progress between executions, and giving every run built-in observability, as shown in the Ship AI recap. Vercel detects when a function is durable and provisions the infrastructure to support it in real time. Under the hood, the Workflows architecture runs your workflow and step code on Vercel Functions, uses Vercel Queues for reliable enqueuing and execution, and keeps all state and event logs in managed persistence.
The concrete version is easier to feel than to describe. In our Slack agent guide, the conversation state, the messages array and the current draft, persists on its own. A teammate gives feedback on Monday, another follows up on Wednesday, and the workflow picks up exactly where it left off, with no database to wire up. WorkflowAgent runs the agent loop inside Vercel Workflows, so tool calls, agent state, and human approvals can persist through process restarts and Vercel Function timeouts. A 50-step research agent becomes 50 separate invocations; each one gets the full timeout, and completed steps are never re-run. That last property matters most, because re-running completed steps is how you get duplicate side effects and corrupted state.
Fluid compute handles the cost side of the same problem. Agent workloads spend most of their wall-clock time waiting, for a model response, for a tool to execute, for a human to approve something. Fluid compute lets multiple invocations share a single function instance and bills only for active CPU time, which for workloads with high idle time can cut costs by up to 90% compared to traditional serverless, according to Vercel's AI Cloud overview. Put durability and fluid compute together and an agent can run for hours across dozens of steps without burning budget on idle waiting or losing state when a step fails.
Copy link to headingPersisting state correctly: UIMessage and ModelMessage
The AI SDK distinguishes between UIMessage and ModelMessage, and getting it wrong is the single most common source of persistence bugs I see in Next.js agent applications. The two types do different jobs. UIMessage is the source of truth for application state, carrying every message, its metadata, tool results, and attachments. ModelMessage is a stripped-down representation built for sending to the model. The rule that keeps teams out of trouble: persist UIMessages for UI restoration, and convert to ModelMessages only at the moment you call the model.
We designed the persistence pattern around streamText with an onFinish callback inside toUIMessageStreamResponse:
const result = streamText({ model: openai('gpt-4o'), messages: convertToModelMessages(validatedMessages),});return result.toUIMessageStreamResponse({ originalMessages: validatedMessages, onFinish: ({ messages }) => { saveChat({ chatId: id, messages }); },});convertToModelMessages bridges the client's UIMessage shape, which carries tool-call parts, attachments, and ID metadata, and the provider-neutral model message shape that streamText expects. When you load messages from storage that contain tools, metadata, or custom data parts, validate them with validateUIMessages before processing. The full pattern lives in the message persistence docs.
Two details cause outsized pain when they're missed. The first is consumeStream, which removes backpressure so the result is stored even when the client has already disconnected. This matters because agent tool loops keep running for seconds or minutes after the user navigates away, and without consumeStream the persistence callback never fires, so the work completes and then vanishes. The second is server-side message ID generation. IDs are not generated properly for persistence by default, and without server-generated IDs your messages drift out of sync across sessions and collide on reload. The AI SDK persistence example uses generateMessageId for exactly this. For ephemeral session state that has to survive across serverless function instances, an Upstash pattern stores conversation history externally with a TTL and writes through to a database asynchronously.
Copy link to headingTreating model choice as a cost lever, not an architecture commitment
Memory doesn't exist in isolation from the rest of the agent's bill. Every retrieval adds tokens to the prompt, every tool call is another round trip to a model, and every step in a workflow stacks more cost on top. The AI Gateway routes to hundreds of models through one API endpoint with automatic failover, zero-markup token pricing, and spend controls, which turns model selection into something you tune rather than something you commit to up front.
The pattern I'd push for any memory-heavy agent: send the cheap, mechanical work to cheap models, and reserve the flagship for final output and hard reasoning. Summarization, entity extraction, and compression are the bulk of memory operations, and almost none of them need the most capable model you have access to. Set per-provider timeouts so a slow provider triggers fast failover instead of stalling the run, and configure model-level failover to fall back to a backup when the primary is down, using AI Gateway provider options. The tradeoff is that you're now reasoning about a routing policy, but that's a far cheaper thing to own than a single hardcoded model and a surprise invoice.
You can only tune what you can see. The AI SDK’s WorkflowAgent makes each tool execution part of a durable, observable workflow step, while AI SDK DevTools captures LLM requests, tool calls, and token usage during local development. This is where the composable design pays off. AI Gateway for routing and cost, Fluid compute for execution, Workflows for durability, and the AI SDK for the message protocol are separate layers you assemble, not a monolith you adopt. There is no single "agent memory" abstraction on purpose, because a production agent needs different memory strategies for different parts of the same workflow.
Copy link to headingAgent memory is a state problem, not a memory problem
Scan back through the failure modes in this piece and they're not really about memory at all. Context stuffing is a cost-and-coherence problem. Stale recall is a data-freshness problem. Orphaned steps and lost progress are durability problems. Every one of them is a state management problem that distributed systems engineers would recognize, dressed up in language-model vocabulary. The model stays stateless, and it will keep staying stateless. The work of deciding what to remember, how to compress it, where to persist it, and when to throw it away lives in the systems around the model, which is the good news, because that's the layer you control.
We built durable workflows, Fluid compute, and the UIMessage/ModelMessage separation because these are the exact failure points we watch teams hit. The AI SDK gives you prepareStep for context compression, consumeStream for disconnect-safe persistence, and use workflow for durability, and the long-term memory integrations layer on when the use case actually demands them. Start with the simplest memory that works, watch where it breaks, and add structure at the break. That sequence beats picking a memory architecture on day one every time.
Copy link to headingFAQ
What is the difference between short-term and long-term agent memory?
Short-term memory is the current conversation, held in the context window, and it clears when the task ends. Long-term memory uses external storage like vector databases to persist knowledge across sessions, retrieved through RAG when the agent needs historical context. Short-term memory fails by overflowing, long-term memory fails by going stale.
Do AI agents have memory by default?
No. A language model processes each request in isolation with no record of previous interactions. Any memory an agent appears to have is engineered around the model, either by sending recent history in the context window for short-term memory, or by storing and retrieving information from external systems for long-term memory.
Is a vector database the same as agent memory?
No. A vector database is one possible component of a long-term memory system, good at answering "what is similar to this?" Agent memory has to answer "what does this agent know, and is it still true?" Vector databases don't reason about time or track when a fact has been superseded, which is why production systems increasingly combine vectors with graphs and episodic buffers.
How is agent memory different from RAG?
RAG is a retrieval technique, the mechanism for pulling relevant external information into the prompt. Agent memory is the broader system that decides what to write, what to keep, what to compress, and what to discard over time. RAG is often how long-term memory gets read, but it doesn't address the write-side decisions that determine memory quality.
When should I use Vercel Workflows instead of a single serverless function for an agent?
Use Workflows when your agent runs multi-step tool loops, needs to survive function timeouts, or must resume after failures without re-running completed work. WorkflowAgent runs the agent loop inside a workflow so state can persist across step boundaries. A 50-step research agent becomes 50 separate invocations where completed steps are never re-run.