Memory & evals · Project

Inspect AI

Inspect AI is a Python framework for running and scoring language model and agent evaluations.

+ Follow

Builder facts

  • Task structure. A task combines a dataset of samples, a solver that produces answers or actions, and a scorer that grades the results. Source ↗
  • Scoring. Built-in scorers include text matching, multiple choice, math and model grading. A custom scorer can return its own scores and explanations. Source ↗
  • Run inspection. Inspect View opens evaluation logs for per-sample review, including model messages, tool activity and scores. Source ↗
  • Access. The MIT-licensed package requires Python 3.11 or later. Source ↗
  • Provider credentials. Calls to hosted models use credentials from the model provider configured for the run. Source ↗

Sources checked

Publisher update ·

Inspect AI 0.3.291 ↗

The CLI can merge logs from sharded runs and exclude selected samples in a finished log from its aggregate metrics.

Change timeline

0 stories

No verified releases in this timeline yet.