> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mira-app.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Eval Framework Overview

> Test MIRA's reasoning quality systematically with built-in evaluation tools.

MIRA's **Eval Framework** lets you measure and track the quality of reasoning engine outputs over time. Instead of eyeballing responses, you define test cases, run them against one or both engines, and get structured scores.

## Why evaluate?

* **Catch regressions** after changing a provider, skill, or prompt
* **Compare engines** — NAE vs RLM — on your specific workloads
* **Tune skills** by seeing how instruction changes affect output quality
* **Build confidence** before deploying MIRA to a wider team

## Core concepts

<CardGroup cols={2}>
  <Card title="Eval Cases" icon="list-checks">
    Individual test scenarios — an input, optional documents, and expected output criteria.
  </Card>

  <Card title="Eval Profiles" icon="user-check">
    Named collections of eval cases that represent a repeatable test suite.
  </Card>

  <Card title="Runs" icon="play">
    An execution of a profile against an engine configuration. Each run stores all inputs and
    outputs.
  </Card>

  <Card title="Scores" icon="chart-bar">
    Automated and human-reviewed scores attached to each case output.
  </Card>
</CardGroup>

## Eval case types

| Type                                   | What it tests                                                                                                              |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| **Rule** (`rule`)                      | Output matches a string, regex, keyword, JSON schema, or satisfies a length constraint — fully deterministic, no LLM calls |
| **Semantic similarity** (`similarity`) | Output is semantically close to a reference answer (embedding-based cosine similarity)                                     |
| **LLM judge** (`llm_judge`)            | A second LLM call grades the output against a rubric you write                                                             |
| **Metric** (`metric`)                  | A performance measurement (latency, token count, or tool call count) compared against a threshold                          |

<Note>
  **Human review** is not a separate eval type. It is a post-run override mechanism: after a run
  completes, any result can receive a human score via the Dashboard → **Review** button. Human
  scores are stored alongside (and can override) automated scores.
</Note>

## How evals run

Evals are **automatic**. When you enable the Eval Framework, every agent response is captured and scored against your active eval definitions. There is no manual "run" button — results appear in the dashboard as you use MIRA.

## Workflow

```
Create evals → Activate evals → Assign to profiles → Chat → Review dashboard
```

1. [Create eval definitions](./running-evals) in the Eval Studio
2. [Create a profile](./profiles) and assign evals to it
3. Activate the profile so it evaluates future responses
4. Use MIRA normally — results stream into the [Eval Dashboard](./dashboard)
5. Optionally [compare two conversation runs (A/B)](./ab-comparison) or [submit human review overrides](./human-review)
6. [Export results](./exporting-results) for reporting

## Opening the Eval Framework

Click the **Flask** icon in the left sidebar.

<Note>
  Edit this page — [Open a pull
  request](https://github.com/satyendra2013/mira-app/edit/main/docs/eval-framework/overview.mdx)
</Note>