Skip to main content
A Rule eval passes if the engine’s output satisfies a deterministic rule. No additional LLM calls are required — rule evals are the fastest and most cost-effective type.

​
Rule sub-types

​
When to use

  • The correct answer has a known, fixed form (e.g. a number, a named entity, a required phrase)
  • You need zero-cost, high-speed evaluation with no second LLM call
  • Compliance checks — ensure a required phrase is always present, or a forbidden phrase is never present
  • Output format validation — ensure the engine returns valid JSON matching your schema

​
Configuring a rule eval

1

Open the Eval Studio

Click the Flask icon in the sidebar.
2

Click New

Click + New in the left eval list panel.
3

Select the type

Click the Rule type card.
4

Choose a rule sub-type

Select the sub-type (e.g. keyword_contains, regex) and fill in the required fields.
5

Enter test input and test output

Use the inline test runner to verify the rule against a sample response before activating.
6

Save and activate

Click Save. Once a test passes, click Activate to start auto-evaluating live responses.

​
Scoring

Pass/fail is binary — no partial credit.

​
Regex examples

Edit this page — Open a pull request