Concept
Insight
A measured change in how an agent or tool behaves, stated with the number it is compared to and how many runs it rests on.
What is measured
For the project: run duration, success rate and cost per run. For each of the most used tools: call latency. A change becomes an insight only when it crosses the threshold below and is outside normal variation (95% confidence). An improvement is not reported as a problem.
What it is compared with
Each value is compared with the same measure over an earlier period, with at least 30 runs on each side:
- if you recorded a deployment in the last 24 hours: the runs since it vs the 7 days before it,
- otherwise: the last 24 hours vs the previous 7 days.
Runs are compared with runs and rates with rates. The insight names both sides, for example “432 ms vs 340 ms in the previous 7 days. Based on 212 calls in the last 24 hours vs 1,840 calls in the previous 7 days”. With fewer runs, the briefing says there is not enough history yet.
How serious it is
| Metric | Minor | Warning | Critical |
|---|---|---|---|
| Average latency | 10% | 20% | 35% |
| Success rate | 3% | 8% | 15% |
| Cost per run | 15% | 30% | 50% |
What comes with it
Steps worth checking for that kind of change, and the runs that show it. Open an insight to see its investigation.