Memory & evals · Project
Inspect AI
Inspect AI is a Python framework for running and scoring language model and agent evaluations.
+ FollowBuilder facts
- Task structure. A task combines a dataset of samples, a solver that produces answers or actions, and a scorer that grades the results. Source ↗
- Scoring. Built-in scorers include text matching, multiple choice, math and model grading. A custom scorer can return its own scores and explanations. Source ↗
- Run inspection. Inspect View opens evaluation logs for per-sample review, including model messages, tool activity and scores. Source ↗
- Access. The MIT-licensed package requires Python 3.11 or later. Source ↗
- Provider credentials. Calls to hosted models use credentials from the model provider configured for the run. Source ↗
Sources checked
Publisher update ·
Inspect AI 0.3.291 ↗The CLI can merge logs from sharded runs and exclude selected samples in a finished log from its aggregate metrics.
Change timeline
0 storiesNo verified releases in this timeline yet.