Evaluation reports
Evaluation reports summarize your evaluation results over time and send them to email or Slack, so you can track quality without reading every result. PostHog creates a default report configuration for every evaluation you make.
Reports are available for all evaluation types – LLM-as-a-judge, code-based, and sentiment analysis – on any target.
Reports also power Self-driving. Turn on the AI observability source in your inbox settings, and report findings arrive as signals.
Default report configuration
The auto-created report uses these defaults:
| Setting | Default value | Description |
|---|---|---|
| Frequency | Count-based | Generates a report after a set number of evaluation runs |
| Trigger threshold | 100 | A report is generated every 100 new evaluation results |
| Cooldown | 1 hour | Minimum hours between count-triggered reports (1 – 24 hours) |
| Daily run cap | 10 | Maximum report runs per day |
| Enabled | Yes | Reports start generating immediately |
| Delivery targets | None | Reports are viewable in the UI but no notifications are sent |
Reports are viewable directly in the evaluation's Reports tab without configuring any delivery targets.
Report metrics by evaluation type
Reports display different metrics depending on your evaluation's output type:
- Boolean evaluations (LLM-as-a-judge and code-based) – Pass rate, pass/fail/N/A counts, total runs, and comparison to the previous reporting period
- Numeric score and categorical evaluations – Pass rate and pass/fail/N/A counts, based on the evaluation's passing rule. Reports need a passing rule.
- Sentiment analysis evaluations – Positive, neutral, and negative counts with distribution percentages, total runs, and comparison to the previous reporting period
Customizing reports
You can adjust the report configuration after creating an evaluation:
- Change the trigger threshold to generate reports more or less frequently
- Adjust the minimum hours between reports (1 – 24) to control how often count-triggered reports can fire
- Add email or Slack delivery targets to receive notifications when reports are generated
- Disable the report if you don't need automated summaries





