Memory, context, evaluations and reliability checks.
Letta runs stateful agents with editable memory and persistent conversation state.
- Runtime. The original Letta server repository points to Letta Code for the current agent runtime, CLI and App Server. Source ↗
- Memory filesystem. MemFS keeps Markdown memory in Git. Root files enter the system prompt each turn, and nested files can be read on demand. Source ↗
Scheduled · awaiting first completed check
Cognee turns application data into graph and vector memory with a separate session cache.
- Write scope. Remember writes into a named graph dataset by default. A session ID targets short-term cache, which can later be bridged into the graph. Source ↗
- Recall. Recall can search session cache before falling through to persistent graph memory. Dataset filters and query modes scope retrieval. Source ↗
Scheduled · awaiting first completed check
Promptfoo runs repeatable comparisons of prompts and model providers against configured test cases.
- Access. The CLI runs locally. Enterprise offers separate managed and on-premises plans. Source ↗
- Test matrix. A configuration combines prompts, providers and test cases with variable values. Results can be viewed locally or exported as JSON, spreadsheets or HTML. Source ↗
Scheduled · awaiting first completed check
Claude-Mem summarizes coding-agent activity and brings relevant observations into later sessions.
- Capture and recall. An observer summarizes tool activity. A new session gets a compact observation index and can fetch individual records by ID. Source ↗
- Storage. A local SQLite database stores sessions, summaries and observations. Keyword search uses SQLite FTS5. Source ↗
Scheduled · awaiting first completed check
Arize Phoenix collects LLM traces and evaluates changes to prompts, models and retrieval.
- Deployment. Phoenix runs locally from the Python package or on Docker or Kubernetes. Arize AX is a separate managed platform that accepts the same OpenTelemetry and OpenInference instrumentation. Source ↗
- License. Elastic License 2.0 permits self-hosting and restricts offering substantial Phoenix functionality to third parties as a hosted service. Source ↗
Scheduled · awaiting first completed check
Inspect AI is a Python framework for running and scoring language model and agent evaluations.
- Task structure. A task combines a dataset of samples, a solver that produces answers or actions, and a scorer that grades the results. Source ↗
- Scoring. Built-in scorers include text matching, multiple choice, math and model grading. A custom scorer can return its own scores and explanations. Source ↗
Scheduled · awaiting first completed check
Graphify maps project files into a queryable graph of code and document relationships.
- Code extraction. Tree-sitter parses code locally without a model call. Semantic extraction from documents and media uses the assistant model or a configured API key. Source ↗
- Edge provenance. Graph relationships are labeled EXTRACTED or INFERRED, so callers can distinguish source-backed links from inferred ones. Source ↗
Scheduled · awaiting first completed check
Langfuse traces AI applications and connects their observations to prompt versions, datasets and evaluations.
- Self-hosted access. The core edition runs without a license key. Enterprise and Government keys unlock additional self-hosted features. Source ↗
- Managed plans. Langfuse Cloud is hosted by the vendor, with a Hobby tier and separate paid plans. Source ↗
Scheduled · awaiting first completed check
Qdrant stores and searches vectors and their associated metadata.
- Data model. Points carry JSON payloads and can use dense, sparse or multivector fields. Source ↗
- Hybrid queries. The Query API can fuse multiple retrieval stages with reciprocal rank fusion or distribution-based score fusion. Fusion inside a prefetch runs per shard. Source ↗
Scheduled · awaiting first completed check
Ragas scores retrieval and generation quality and builds synthetic evaluation datasets for LLM applications.
- License. The Python package uses the Apache 2.0 license. Source ↗
- Model access. Model-backed metrics use the configured evaluator, such as an OpenAI client with its own credentials. Source ↗
Scheduled · awaiting first completed check
Graphiti builds temporal knowledge graphs from streams of changing information.
- Fact history. Edges carry validity windows and source episode provenance. An invalidated relationship can remain available for historical queries. Source ↗
- Retrieval. Episodes update the graph incrementally. Search combines semantic, BM25 and graph retrieval, and optional Pydantic schemas define custom entity and edge types. Source ↗
Scheduled · awaiting first completed check
Supermemory extracts user memories, maintains profiles and retrieves context from memories and documents.
- User profiles. Profiles combine stable facts with recent activity. Hybrid search retrieves personalized memory and knowledge-base documents together. Source ↗
- Hosted access. The hosted API uses a Supermemory API key. Container tags scope writes and searches to a user, project or other grouping. Source ↗
Scheduled · awaiting first completed check
pgvector adds vector storage and similarity search to PostgreSQL.
- Database access. Version 0.8.7 supports PostgreSQL 13 or later. The extension is enabled per database and hosted PostgreSQL providers must make it available. Source ↗
- License. The PostgreSQL License permits use and redistribution without a fee. Database hosting is separate. Source ↗
Scheduled · awaiting first completed check
Mem0 extracts and retrieves memories for users, sessions and agents.
- Memory scopes. Applications can scope memories to a user, agent or session. Retrieval supports semantic search with optional keyword and entity signals. Source ↗
- Delivery. The open-source library runs inside a Python or Node.js app. A self-hosted Docker server adds an HTTP API, dashboard, per-user keys and audit logs. Source ↗
Scheduled · awaiting first completed check
DeepEval tests LLM applications with assertions on outputs, components and agent trajectories.
- Local test runs. Python evaluation tests can run locally with pytest-style assertions or the DeepEval CLI, including in CI. Source ↗
- Model and cloud access. Local evaluation needs no Confident AI account. Hosted judge metrics require the configured model provider credentials. A Confident AI account enables cloud reports and collaboration. Source ↗
Scheduled · awaiting first completed check
Chroma stores embeddings, documents and metadata for retrieval.
- Client modes. Python supports in-memory and on-disk persistent clients. Chroma Cloud takes an API key, and key-only database discovery requires a key scoped to a single database. Source ↗
- Collection queries. Query embeds text with the collection’s configured embedding function or takes vectors with the collection’s dimension. Get retrieves records without similarity ranking. Source ↗
Scheduled · awaiting first completed check
Cloudflare Agent Memory is a managed service that extracts and recalls durable context across agent conversations.
- Ingestion. Conversation ingestion extracts facts, events, instructions and tasks and stores the original messages verbatim. Explicit writes store information the application already identified. Source ↗
- Recall. Recall combines keyword, topic, vector and transcript search and synthesizes an answer from the ranked results. New facts or instructions can supersede earlier versions without deleting them. Source ↗
Scheduled · awaiting first completed check
LangMem provides Python components for extracting, storing and retrieving long-term agent memory.
- In-turn tools. An agent can call memory management and search tools during a conversation. Both use LangGraph store namespaces. Source ↗
- Background extraction. A background memory manager can extract, consolidate and update memories outside the interactive turn. Functional primitives also support custom storage. Source ↗
Scheduled · awaiting first completed check