Files
Lu Shueh Chou 126169f3ca Alerting: Add LogicOr operation option (#89258)
---------

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>
2024-08-16 11:02:14 -04:00

114 lines
5.4 KiB
Markdown

---
aliases:
- ../fundamentals/alert-rules/rule-evaluation/ # /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/rule-evaluation/
canonical: https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rule-evaluation/
description: Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state
keywords:
- grafana
- alerting
- evaluation
labels:
products:
- cloud
- enterprise
- oss
title: Alert rule evaluation
weight: 108
refs:
alerts-state-health:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/state-and-health/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/state-and-health/
---
# Alert rule evaluation
The criteria determining when an alert rule fires are based on two settings:
- [Evaluation group](#evaluation-group): how frequently the alert rule is evaluated.
- [Pending period](#pending-period): how long the condition must be met to start firing.
{{< figure src="/media/docs/alerting/alert-rule-evaluation.png" max-width="750px" alt="Set the evaluation behavior of the alert rule in Grafana." caption="Set alert rule evaluation" >}}
## Evaluation group
Every alert rule is assigned to an evaluation group. You can assign the alert rule to an existing evaluation group or create a new one.
Each evaluation group contains an **evaluation interval** that determines how frequently the alert rule is checked. For instance, the evaluation may occur every `10s`, `30s`, `1m`, `10m`, etc.
**Evaluation strategies**
Alert rules in different groups can be evaluated simultaneously.
- **Grafana-managed** alert rules within the same group are evaluated concurrently—they are evaluated at different times over the same evaluation interval but display the same evaluation timestamp.
- **Data-source managed** alert rules within the same group are evaluated sequentially, one after the other—this is necessary to ensure that recording rules are evaluated before alert rules.
## Pending period
You can set a pending period to prevent unnecessary alerts from temporary issues.
The pending period specifies how long the condition must be met before firing, ensuring the condition is consistently met over a consecutive period.
You can also set the pending period to zero to skip it and have the alert fire immediately once the condition is met.
## Condition operator
There are several condition operators available.
- **and**: Two conditions before and after must be true for the overall condition to be true.
- **or**: If one of conditions before and after are true, the overall condition is true.
- **logic-or**: If the condition before logic-or is true, the overall condition is immediately true, without evaluating subsequent conditions.
Here are some examples of operators.
- `TRUE and TRUE or FALSE and FALSE` evaluate to `FALSE`, because last two conditions return `FALSE`.
- `TRUE and TRUE logic-or FALSE and FALSE` evaluate to `TRUE`, because the preceding condition returns `TRUE`.
## Evaluation example
Keep in mind:
- One alert rule can generate multiple alert instances - one for each time series produced by the alert rule's query.
- Alert instances from the same alert rule may be in different states. For instance, only one observed machine might start firing.
- Only **Alerting** and **Resolved** alert instances are routed to manage their notifications.
{{< figure src="/media/docs/alerting/alert-rule-evaluation-overview-statediagram-v2.png" alt="A diagram of the alert instance states and when to route their notifications." max-width="750px" >}}
<!--
Remove ///
stateDiagram-v2
direction LR
Normal --///> Pending
note right of Normal
Route "Resolved" alert instances
for notifications
end note
Pending --///> Alerting
Alerting --///> Normal: Resolved
note right of Alerting
Route "Alerting" alert instances
for notifications
end note
-->
Consider an alert rule with an **evaluation interval** set at every 30 seconds and a **pending period** of 90 seconds. The evaluation occurs as follows:
| Time | Condition | Alert instance state | Pending counter |
| ------------------------- | --------- | --------------------- | --------------- |
| 00:30 (first evaluation) | Not met | Normal | - |
| 01:00 (second evaluation) | Breached | Pending | 0s |
| 01:30 (third evaluation) | Breached | Pending | 30s |
| 02:00 (fourth evaluation) | Breached | Pending | 60s |
| 02:30 (fifth evaluation) | Breached | Alerting<sup>\*</sup> | 90s |
An alert instance is resolved when it transitions from the `Firing` to the `Normal` state. For instance, in the previous example:
| Time | Condition | Alert instance state | Pending counter |
| -------------------------- | --------- | ----------------------------- | --------------- |
| 03:00 (sixth evaluation) | Not met | Normal <sup>Resolved \*</sup> | 120s |
| 03:30 (seventh evaluation) | Not met | Normal | 150s |
To learn more about the state changes of alert rules and alert instances, refer to [State and health of alert rules](ref:alerts-state-health).