Closing the Agent Learning Loop with Utility-Ranked Memory
Your agent failed on the same edge case last Tuesday. Someone on the team noticed, tweaked the prompt, and redeployed. This Tuesday, it failed again. Different input, same root cause. The prompt fix was too narrow. The trace was sitting right there in your observability stack, but nothing connected that signal back to the agent's behavior.
This is the default state of most production AI agent systems today. Teams have traces. They have evals. What they do not have is a mechanism that turns failure signals into better retrieval on the next run.
We built Reflect to close that gap. Reflect is an outcome-informed memory layer that stores reflections from past runs, scores them by how useful they actually were, and re-ranks retrieval so your agent improves with every reviewed trace, not just every prompt rewrite.
The missing layer between evals and action
Production agent stacks have three components: observability ( traces), evaluation (pass/fail judgments), and action (the agent itself). The problem is that these layers do not talk to each other.
Your observability stack captures every tool call, every LLM completion, every exception. Your eval suite judges whether the final output was correct. But the agent that runs tomorrow starts from a blank slate. It has no access to why yesterday's runs passed or failed. The eval signal dies in a dashboard.
This is the missing layer: a system that consumes traces, absorbs eval outcomes, and converts both into retrievable guidance for future runs.
Reflect sits between your evals and your agent. It treats traces not as passive audit logs, but as training signal. When a review marks a trace as failed, Reflect does not just file it away. It extracts a reflection — a compressed lesson about what went wrong — and stores it as a memory tied to that task type. When a similar task arrives, that reflection surfaces. The agent acts differently because it now has context about what not to do.
The eval-to-action loop closes because Reflect makes memories outcome-addressable. You do not retrieve by keyword. You retrieve by semantic similarity to the current task, weighted by whether those memories have historically helped or hurt. The eval outcome becomes a first-class signal in the retrieval ranking, not just a post-mortem footnote.
The manual improvement tax
After talking to dozens of enterprise agent teams, we see the same pattern: they can tell when something went wrong, and they can measure pass/fail rates. But improvement still happens outside the data loop:
- A developer reviews a failing trace, rewrites part of the prompt, and redeploys.
- The team swaps to a more capable (and more expensive) model.
- Someone restructures the tool-call harness.
- In extreme cases, the team fine-tunes a custom model.
These are all valid moves. But they share a weakness: they are coarse, slow, and global. A prompt change shifts behavior for every future request, whether or not the change was relevant. A model upgrade costs more across the board, even for tasks the cheaper model handled fine.
More importantly, none of these approaches directly answer the question every agent team actually cares about: what from past experience helped this task succeed, and what made it fail?
Automated prompt optimization can reduce costs by up to 90x while matching or exceeding hand-tuned performance[^1]. But even automated optimization treats the prompt as the unit of improvement. We think the unit should be smaller and more targeted: the individual memory retrieved at inference time.
Why current AI agent memory systems stall
Memory was supposed to solve this. In practice, most agent memory frameworks have focused on user continuity: preferences, profile facts, conversation history, and long-lived personalization. That is useful for chat experiences, but it does not create a self-improving learning system for production agents.
The table below summarizes the dominant approaches and where they fall short for outcome-based learning:
| Approach | What it stores | Retrieval signal | Learns from outcomes? |
|---|---|---|---|
| Conversation buffer (LangChain, most frameworks) | Raw chat history | Recency | No |
| Semantic memory (Mem0, LangMem) | Extracted facts, preferences | Embedding similarity | No |
| Temporal knowledge graph (Zep/Graphiti) | Entity relationships over time | Graph traversal + recency | No |
| Episodic + Reflexion (Reflexion[^2]) | Verbal self-reflections | Similarity to current task | Partially: reflections capture lessons, but retrieval is not ranked by outcome |
| Reflect | Task-linked reflections with utility scores | Similarity weighted by outcome-derived utility | Yes: utility scores update after every reviewed run |
The distinction matters. Systems like Mem0 achieve strong results on conversational benchmarks, including a 26% improvement over OpenAI Memory on LoCoMo[^3]. But their retrieval is driven by semantic similarity and recency, not by whether a memory actually helped produce a good outcome. Reflexion (Shinn et al., NeurIPS 2023) introduced the critical idea that agents can learn from verbal self-reflection, achieving 91% pass@1 on HumanEval[^2]. But Reflexion's memory retrieval does not incorporate a learned utility signal from downstream results.
Production agents need memory that is tied to tasks (not just users), updated by outcomes (not just recency), shaped by failure (not only by similarity), and grounded in reflections about what worked and what did not.
Without that, memory becomes a static index. It retrieves things that sound similar, not things that have a track record of helping.
Utility: a credit score for memories
Think of utility like a credit score. A credit score does not just record that you had a loan. It tracks whether you paid it back. Similarly, utility does not just record that a memory was retrieved. It tracks whether the run it participated in succeeded or failed.
In Reflect, a memory is not stored text with an embedding. It carries:
- the original task the agent was solving
- a reflection about what worked or failed in that run
- the review outcome (pass, fail, or detailed feedback)
- a utility score that changes every time the memory is retrieved and the run is reviewed
The ranking formula combines both signals:
score = (1 - λ) × similarity + λ × utility
Similarity tells you whether a memory is about the same kind of problem. Utility tells you whether that memory has historically helped. You need both. The λ parameter controls the trade-off: at λ = 0, retrieval is pure semantic search; at λ = 1, it ranks entirely by past outcomes.
If a memory was retrieved and the run passed, its utility rises. If it was retrieved and the run failed, its utility falls. Over repeated runs, Reflect starts preferring memories that repeatedly contribute to successful outcomes and down-ranking memories that keep showing up in failures.
That is how the open loop closes. The signal is not going to a dashboard for a human to interpret. It feeds directly into the retrieval ranking.
Memory as reasoning
Most memory systems store facts. User preferences. Conversation history. Document chunks. These are static artifacts. Reflect stores something different: reasoning about outcomes.
Here is the distinction. A fact memory says: "The user prefers dark mode." A reasoning memory says: "When processing a duplicate charge, check settlement status before issuing refunds, because the last time we skipped that check, the customer received two refunds."
Reflect generates these reasoning memories by combining two inputs: the trace (what the agent actually did) and the review (whether it worked). The trace provides the trajectory — the sequence of tool calls, the LLM completions, the retrieved memories that influenced decisions. The review provides the outcome signal — pass, fail, or detailed feedback explaining why.
From these two inputs, Reflect constructs a reflection. This is not a human-written note. It is an LLM-generated compression of the lesson embedded in the trace-review pair: what the agent tried, what went wrong, and what should happen next time. The reflection is embedded and stored as a memory, linked to the original task type.
We call this memory as reasoning because the retrieved content is not raw data. It is processed, contextualized advice. When your agent queries Reflect before a run, it receives not document snippets, but distilled experience: "Last time you saw a task like this, you retrieved memory X, took action Y, and it failed. Consider action Z instead."
This is why traces matter as much as reviews. A review alone says that something failed. The trace says how it failed — which memories were retrieved, which tools were called, where the agent's reasoning went wrong. Reflect uses both to generate a reflection that is grounded in the actual behavior of the system, not just the final outcome.
Over many runs, the memory store accumulates a layer of learned reasoning. Each memory carries not just the reflection text, but a utility score tracking whether following that reflection led to success. Retrieval becomes a form of experience-weighted reasoning: the agent surfaces advice that has survived the test of multiple reviewed runs.
Where utility changes agent behavior: three examples
Utility is an abstract concept until you see it shift real retrieval decisions. Here are three cases where the learning loop changes what the agent actually does.
1. Customer support: learning which resolution strategies work
A support agent handles refund requests. Early on, Reflect retrieves a memory that says: "When a customer reports a duplicate charge, issue an immediate refund." The agent follows this advice and processes a refund, but the customer already had a pending settlement. The run is reviewed as a failure with the feedback: "Did not check existing settlement state before issuing refund."
Two things happen. First, the utility of that "issue an immediate refund" memory drops. Second, Reflect stores a new reflection: "For duplicate charge complaints, check settlement status before initiating any refund." On the next similar ticket, the new reflection ranks higher because its utility starts neutral while the old one has been penalized. The agent now checks settlement state first.
No prompt was rewritten. No model was swapped. The retrieval layer learned from the outcome.
2. Code review: surfacing advice that developers actually accept
A code review agent suggests changes on pull requests. Some suggestions get accepted by reviewers; others get dismissed. Each accepted suggestion is a "pass" review; each dismissed suggestion is a "fail."
Over hundreds of PRs, Reflect learns which categories of advice are consistently useful. Memories about "add null checks before database lookups" accumulate high utility because reviewers almost always accept them. Memories about "rename this variable for clarity" accumulate lower utility because reviewers frequently dismiss them as stylistic noise.
After a few weeks, the agent's suggestions shift. It still retrieves both types of memory, but the retrieval ranking now favors the high-acceptance patterns. The agent naturally focuses on the feedback that developers value.
Why this matters for AI agents in production
Most agent systems today are improved by operators, not by experience. Someone reviews a bad run, updates a prompt, changes a tool call, adds a rule, or swaps to a more expensive model. The whole system shifts for every future request, whether the change was relevant or not.
Utility gives you a narrower and more data-driven unit of improvement. Instead of asking "How do we change the whole agent?", you can ask:
- Which memories were retrieved for this run?
- Which ones actually helped?
- Which ones hurt?
- What reflection should be stored for the next similar task?
That creates a much tighter agent feedback loop. The system gets better because its retrieval layer gets better, not because someone manually retuned the whole stack after every bad example.
This is analogous to how Reflexion demonstrated that verbal self-reflection can push agents from 80% to 91% on HumanEval without any weight updates[^2]. Reflect takes the same insight (agents can learn from linguistic experience) and adds a quantitative ranking signal so that which experiences get surfaced is itself a learned function.
Add Reflect in a few lines
If you are using the Python SDK, the simplest way to wire in the learning loop is with client.trace(). It retrieves memories on entry and automatically records which memories were used when the trace is submitted.
from reflect_sdk import ReflectClient
client = ReflectClient(
base_url="https://api.starlight-search.com",
api_key="rf_live_...",
project_id="support-agent",
)
with client.trace("Handle a duplicate refund request for an enterprise customer") as ctx:
messages = [
{"role": "user", "content": ctx.augmented_task},
]
response = my_llm(ctx.augmented_task)
messages.append({"role": "assistant", "content": response})
ctx.set_output(
trajectory=messages,
final_response=response,
result="pass",
model="gpt-5.4-mini",
)
That one block handles the core work: query memories before the run, inject them into the task, submit the trace after the run, attach the review result, and preserve the retrieved memory IDs so utility can be updated correctly.
The learning loop after the review
Once a review arrives, Reflect does three things:
- It creates a reflection from the task, trajectory, result, and feedback.
- It stores that reflection as a new memory with an initial utility score.
- It updates the utility of the memories retrieved for that run so future ranking improves.
Every reviewed run leaves behind two kinds of value: a new reflection for future similar tasks and a better ordering of existing memories. This is what makes utility different from plain semantic search. Semantic search can only tell you what looks similar. Utility tells you what has earned trust.
Limitations and open questions
We want to be upfront about what we have not solved yet.
Cold start. A new Reflect project has no memories and no utility signal. The first N runs are pure semantic retrieval until enough reviews accumulate to make utility scores meaningful. We are exploring seeding strategies (importing reflections from related projects, bootstrapping from existing evals), but this remains an active area of work.
Utility drift. Task distributions change over time. A memory that was highly useful six months ago may be irrelevant or even harmful for today's tasks. We currently apply time-decay to utility scores, but the optimal decay schedule is task-dependent and hard to set in advance.
Review quality. The entire loop depends on accurate reviews. If pass/fail labels are noisy, or if human reviewers are inconsistent, utility scores will be noisy too. Automated review (using a judge model) helps with throughput but introduces its own failure modes.
Lambda tuning. The balance between similarity and utility (λ) is a hyperparameter. We default to λ = 0.5, but the right value depends on how much outcome signal your project has accumulated. We plan to make this adaptive.
Conclusion
The real bottleneck in agent systems is not lack of traces or lack of evals. It is the lack of a bridge between observation and action. Utility is that bridge. It turns outcomes into a learned ranking signal so your agent retrieves what has worked, not just what sounds relevant. Once you have that, memory stops being a static retrieval layer and starts acting like experience. We are building Reflect to make this the default. Try it at reflect.starlight-search.com and read the full API reference in our docs.
References
[^1]: Databricks Engineering Blog, "Building State-of-the-Art Enterprise Agents 90x Cheaper with Automated Prompt Optimization," 2025. Link
[^2]: N. Shinn, F. Cassano, A. Gopinath, K. R. Sheshadri, P. Karthik, S. Labash, and A. Coskun, "Reflexion: Language Agents with Verbal Reinforcement Learning," NeurIPS 2023. arXiv:2303.11366
[^3]: P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav, "Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory," ECAI 2025. arXiv:2504.19413
[^4]: D. Wu, H. Zhu, and K. Narasimhan, "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory," ICLR 2025. arXiv:2410.10813
[^5]: P. Rasmussen, "Zep: A Temporal Knowledge Graph Architecture for Agent Memory," 2025. arXiv:2501.13956
[^6]: W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y. Zhang, "A-MEM: Agentic Memory for LLM Agents," NeurIPS 2025. arXiv:2502.12110
