Rule sub-types
| Sub-type | Passes when… |
|---|---|
keyword_contains | Output contains all specified keywords (case-insensitive) |
keyword_excludes | Output contains none of the forbidden keywords |
regex | Output matches the JavaScript regular expression |
json_schema | Output is valid JSON matching the given JSON Schema |
length_min | Output character count ≥ minimum |
length_max | Output character count ≤ maximum |
custom_js | Custom JavaScript function returns true |
When to use
- The correct answer has a known, fixed form (e.g. a number, a named entity, a required phrase)
- You need zero-cost, high-speed evaluation with no second LLM call
- Compliance checks — ensure a required phrase is always present, or a forbidden phrase is never present
- Output format validation — ensure the engine returns valid JSON matching your schema
Configuring a rule eval
Choose a rule sub-type
Select the sub-type (e.g.
keyword_contains, regex) and fill in the required fields.Enter test input and test output
Use the inline test runner to verify the rule against a sample response before activating.
Scoring
| Result | Condition |
|---|---|
| ✅ Pass (score: 1.0) | Rule condition satisfied |
| ❌ Fail (score: 0.0) | Rule condition not satisfied |
Regex examples
| Pattern | Matches |
|---|---|
\d{4}-\d{2}-\d{2} | Any ISO date like 2024-03-15 |
GPT-4[o|turbo]? | GPT-4, GPT-4o, GPT-4turbo |
(?i)conclusion | ”Conclusion”, “CONCLUSION”, “conclusion” |
Edit this page — Open a pull
request