Project directory

Memory & evals

Memory, context, evaluations and reliability checks.

Letta

Letta

Letta runs stateful agents with editable memory and persistent conversation state.

  • Runtime. The original Letta server repository points to Letta Code for the current agent runtime, CLI and App Server. Source ↗
  • Memory filesystem. MemFS keeps Markdown memory in Git. Root files enter the system prompt each turn, and nested files can be read on demand. Source ↗

Publisher update ·

Letta Code 0.34.66 ↗

Scheduled · awaiting first completed check

Cognee

Cognee

Cognee turns application data into graph and vector memory with a separate session cache.

  • Write scope. Remember writes into a named graph dataset by default. A session ID targets short-term cache, which can later be bridged into the graph. Source ↗
  • Recall. Recall can search session cache before falling through to persistent graph memory. Dataset filters and query modes scope retrieval. Source ↗

Publisher update ·

Cognee 1.6.10 ↗

Scheduled · awaiting first completed check

Promptfoo

Promptfoo

Promptfoo runs repeatable comparisons of prompts and model providers against configured test cases.

  • Access. The CLI runs locally. Enterprise offers separate managed and on-premises plans. Source ↗
  • Test matrix. A configuration combines prompts, providers and test cases with variable values. Results can be viewed locally or exported as JSON, spreadsheets or HTML. Source ↗

Publisher update ·

Promptfoo 0.126.0 ↗

Scheduled · awaiting first completed check

Claude-Mem

Claude-Mem

Claude-Mem summarizes coding-agent activity and brings relevant observations into later sessions.

  • Capture and recall. An observer summarizes tool activity. A new session gets a compact observation index and can fetch individual records by ID. Source ↗
  • Storage. A local SQLite database stores sessions, summaries and observations. Keyword search uses SQLite FTS5. Source ↗

Publisher update ·

Claude-Mem 16.0.0 ↗

Scheduled · awaiting first completed check

Arize Phoenix

Arize Phoenix

Arize Phoenix collects LLM traces and evaluates changes to prompts, models and retrieval.

  • Deployment. Phoenix runs locally from the Python package or on Docker or Kubernetes. Arize AX is a separate managed platform that accepts the same OpenTelemetry and OpenInference instrumentation. Source ↗
  • License. Elastic License 2.0 permits self-hosting and restricts offering substantial Phoenix functionality to third parties as a hosted service. Source ↗

Publisher update ·

Arize Phoenix 21.2.0 ↗

Scheduled · awaiting first completed check

Inspect AI

Inspect AI

Inspect AI is a Python framework for running and scoring language model and agent evaluations.

  • Task structure. A task combines a dataset of samples, a solver that produces answers or actions, and a scorer that grades the results. Source ↗
  • Scoring. Built-in scorers include text matching, multiple choice, math and model grading. A custom scorer can return its own scores and explanations. Source ↗

Publisher update ·

Inspect AI 0.3.291 ↗

Scheduled · awaiting first completed check

Graphify

Graphify

Graphify maps project files into a queryable graph of code and document relationships.

  • Code extraction. Tree-sitter parses code locally without a model call. Semantic extraction from documents and media uses the assistant model or a configured API key. Source ↗
  • Edge provenance. Graph relationships are labeled EXTRACTED or INFERRED, so callers can distinguish source-backed links from inferred ones. Source ↗

Publisher update ·

Graphify 0.9.112 ↗

Scheduled · awaiting first completed check

Langfuse

Langfuse

Langfuse traces AI applications and connects their observations to prompt versions, datasets and evaluations.

  • Self-hosted access. The core edition runs without a license key. Enterprise and Government keys unlock additional self-hosted features. Source ↗
  • Managed plans. Langfuse Cloud is hosted by the vendor, with a Hobby tier and separate paid plans. Source ↗

Publisher update ·

Langfuse 4.82.0 ↗

Scheduled · awaiting first completed check

Qdrant

Qdrant

Qdrant stores and searches vectors and their associated metadata.

  • Data model. Points carry JSON payloads and can use dense, sparse or multivector fields. Source ↗
  • Hybrid queries. The Query API can fuse multiple retrieval stages with reciprocal rank fusion or distribution-based score fusion. Fusion inside a prefetch runs per shard. Source ↗

Publisher update ·

Qdrant 1.20.2 ↗

Scheduled · awaiting first completed check

Ragas

Ragas

Ragas scores retrieval and generation quality and builds synthetic evaluation datasets for LLM applications.

  • License. The Python package uses the Apache 2.0 license. Source ↗
  • Model access. Model-backed metrics use the configured evaluator, such as an OpenAI client with its own credentials. Source ↗

Publisher update ·

Ragas 0.4.4 ↗

Scheduled · awaiting first completed check

Graphiti

Graphiti

Graphiti builds temporal knowledge graphs from streams of changing information.

  • Fact history. Edges carry validity windows and source episode provenance. An invalidated relationship can remain available for historical queries. Source ↗
  • Retrieval. Episodes update the graph incrementally. Search combines semantic, BM25 and graph retrieval, and optional Pydantic schemas define custom entity and edge types. Source ↗

Publisher update ·

Graphiti 0.31.2 ↗

Scheduled · awaiting first completed check

Supermemory

Supermemory

Supermemory extracts user memories, maintains profiles and retrieves context from memories and documents.

  • User profiles. Profiles combine stable facts with recent activity. Hybrid search retrieves personalized memory and knowledge-base documents together. Source ↗
  • Hosted access. The hosted API uses a Supermemory API key. Container tags scope writes and searches to a user, project or other grouping. Source ↗

Publisher update ·

Supermemory Local 0.0.9 ↗

Scheduled · awaiting first completed check

pgvector

pgvector

pgvector adds vector storage and similarity search to PostgreSQL.

  • Database access. Version 0.8.7 supports PostgreSQL 13 or later. The extension is enabled per database and hosted PostgreSQL providers must make it available. Source ↗
  • License. The PostgreSQL License permits use and redistribution without a fee. Database hosting is separate. Source ↗

Publisher update ·

pgvector 0.8.7 ↗

Scheduled · awaiting first completed check

Mem0

Mem0

Mem0 extracts and retrieves memories for users, sessions and agents.

  • Memory scopes. Applications can scope memories to a user, agent or session. Retrieval supports semantic search with optional keyword and entity signals. Source ↗
  • Delivery. The open-source library runs inside a Python or Node.js app. A self-hosted Docker server adds an HTTP API, dashboard, per-user keys and audit logs. Source ↗

Publisher update ·

Mem0 Python 2.2.1 ↗

Scheduled · awaiting first completed check

DeepEval

DeepEval

DeepEval tests LLM applications with assertions on outputs, components and agent trajectories.

  • Local test runs. Python evaluation tests can run locally with pytest-style assertions or the DeepEval CLI, including in CI. Source ↗
  • Model and cloud access. Local evaluation needs no Confident AI account. Hosted judge metrics require the configured model provider credentials. A Confident AI account enables cloud reports and collaboration. Source ↗

Publisher update ·

DeepEval Python 4.2.4 ↗

Scheduled · awaiting first completed check

Chroma

Chroma

Chroma stores embeddings, documents and metadata for retrieval.

  • Client modes. Python supports in-memory and on-disk persistent clients. Chroma Cloud takes an API key, and key-only database discovery requires a key scoped to a single database. Source ↗
  • Collection queries. Query embeds text with the collection’s configured embedding function or takes vectors with the collection’s dimension. Get retrieves records without similarity ranking. Source ↗

Publisher update ·

Chroma 1.5.9 ↗

Scheduled · awaiting first completed check

Cloudflare Agent Memory is a managed service that extracts and recalls durable context across agent conversations.

  • Ingestion. Conversation ingestion extracts facts, events, instructions and tasks and stores the original messages verbatim. Explicit writes store information the application already identified. Source ↗
  • Recall. Recall combines keyword, topic, vector and transcript search and synthesizes an answer from the ranked results. New facts or instructions can supersede earlier versions without deleting them. Source ↗

Publisher update ·

Cloudflare Agent Memory private beta ↗

Scheduled · awaiting first completed check

LangMem

LangMem

LangMem provides Python components for extracting, storing and retrieving long-term agent memory.

  • In-turn tools. An agent can call memory management and search tools during a conversation. Both use LangGraph store namespaces. Source ↗
  • Background extraction. A background memory manager can extract, consolidate and update memories outside the interactive turn. Functional primitives also support custom storage. Source ↗

Publisher update ·

LangMem 0.0.30 ↗

Scheduled · awaiting first completed check