33db776c91
* Fix missing `alt` text in images * Fix `canonical` setting in AlertManager docs * Fix `.`(dot) position in hyperlinks due to a temporary styling issue * Revert "Fix `canonical` setting in AlertManager docs" This reverts commit 65354b36422024e09709e5f5640d646b1cad75e0.
4.7 KiB
| aliases | canonical | description | keywords | labels | title | weight | refs | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rule-evaluation/ | Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state |
|
|
Alert rule evaluation | 108 |
|
Alert rule evaluation
The criteria determining when an alert rule fires are based on two settings:
- Evaluation group: how frequently the alert rule is evaluated.
- Pending period: how long the condition must be met to start firing.
{{< figure src="/media/docs/alerting/alert-rule-evaluation.png" max-width="750px" alt="Set the evaluation behavior of the alert rule in Grafana." caption="Set alert rule evaluation" >}}
Evaluation group
Every alert rule is assigned to an evaluation group. You can assign the alert rule to an existing evaluation group or create a new one.
Each evaluation group contains an evaluation interval that determines how frequently the alert rule is checked. For instance, the evaluation may occur every 10s, 30s, 1m, 10m, etc.
Alert rules in different groups can be evaluated simultaneously.
Grafana-managed alert rules within the same group are evaluated simultaneously. However, data-source managed alert rules within the same group are evaluated one after the other—this is necessary to ensure that recording rules are evaluated before alert rules.
Pending period
You can set a pending period to prevent unnecessary alerts from temporary issues.
The pending period specifies how long the condition must be met before firing, ensuring the condition is consistently met over a consecutive period.
You can also set the pending period to zero to skip it and have the alert fire immediately once the condition is met.
Evaluation example
Keep in mind:
- One alert rule can generate multiple alert instances - one for each time series produced by the alert rule's query.
- Alert instances from the same alert rule may be in different states. For instance, only one observed machine might start firing.
- Only Alerting and Resolved alert instances are routed to manage their notifications.
{{< figure src="/media/docs/alerting/alert-rule-evaluation-overview-statediagram-v2.png" alt="A diagram of the alert instance states and when to route their notifications." max-width="750px" >}}
Consider an alert rule with an evaluation interval set at every 30 seconds and a pending period of 90 seconds. The evaluation occurs as follows:
| Time | Condition | Alert instance state | Pending counter |
|---|---|---|---|
| 00:30 (first evaluation) | Not met | Normal | - |
| 01:00 (second evaluation) | Breached | Pending | 0s |
| 01:30 (third evaluation) | Breached | Pending | 30s |
| 02:00 (fourth evaluation) | Breached | Pending | 60s |
| 02:30 (fifth evaluation) | Breached | Alerting* | 90s |
An alert instance is resolved when it transitions from the Firing to the Normal state. For instance, in the previous example:
| Time | Condition | Alert instance state | Pending counter |
|---|---|---|---|
| 03:00 (sixth evaluation) | Not met | Normal Resolved * | 120s |
| 03:30 (seventh evaluation) | Not met | Normal | 150s |
To learn more about the state changes of alert rules and alert instances, refer to State and health of alert rules.