mirror of
https://github.com/grafana/grafana.git
synced 2025-09-18 22:02:52 +08:00
docs(alerting): stale alert instances updates (#103014)
* docs(alerting): stale alert instances updates * correct vale error
This commit is contained in:
@ -68,6 +68,24 @@ Alert instances will be routed for [notifications](ref:notifications) when they
|
|||||||
|
|
||||||
{{< figure src="/media/docs/alerting/alert-rule-evaluation-overview-statediagram-v2.png" alt="A diagram of the alert instance states and when to route their notifications." max-width="750px" >}}
|
{{< figure src="/media/docs/alerting/alert-rule-evaluation-overview-statediagram-v2.png" alt="A diagram of the alert instance states and when to route their notifications." max-width="750px" >}}
|
||||||
|
|
||||||
|
### Stale alert instances (MissingSeries)
|
||||||
|
|
||||||
|
The `No Data` state occurs when the alert rule query runs successfully but returns no data points at all.
|
||||||
|
|
||||||
|
An alert instance is considered stale if the query returns data but its dimension or series has disappeared for two evaluation intervals. In this case, the alert instance transitions to the **Normal (MissingSeries)** state as resolved, and is then evicted.
|
||||||
|
|
||||||
|
1. The alert rule runs and returns data for some label sets.
|
||||||
|
|
||||||
|
1. An alert instance that previously existed is now missing.
|
||||||
|
|
||||||
|
1. Grafana keeps the previous state of the alert instance for two evaluation intervals.
|
||||||
|
|
||||||
|
1. If it remains missing after two intervals, it transitions to the **Normal** state and sets **MissingSeries** in the `grafana_state_reason` annotation.
|
||||||
|
|
||||||
|
1. Stale alert instances in the **Alerting**, **No Data**, or **Error** states transition to the **Normal** state as **Resolved**, and are routed for notifications like other resolved alerts.
|
||||||
|
|
||||||
|
1. The alert instance is removed from the UI.
|
||||||
|
|
||||||
### `No Data` and `Error` alerts
|
### `No Data` and `Error` alerts
|
||||||
|
|
||||||
When an alert rule evaluation results in a `No Data` or `Error` state, Grafana Alerting immediately creates a new alert instance —skipping the pending period—with the following additional labels:
|
When an alert rule evaluation results in a `No Data` or `Error` state, Grafana Alerting immediately creates a new alert instance —skipping the pending period—with the following additional labels:
|
||||||
@ -82,12 +100,6 @@ You can manage these alerts like regular ones by using their labels to apply act
|
|||||||
|
|
||||||
If the alert rule is configured to send notifications directly to a selected contact point (instead of using notification policies), the `DatasourceNoData` and `DatasourceError` alerts are also sent to that contact point. Any additional notification settings defined in the alert rule, such as muting or grouping, are preserved.
|
If the alert rule is configured to send notifications directly to a selected contact point (instead of using notification policies), the `DatasourceNoData` and `DatasourceError` alerts are also sent to that contact point. Any additional notification settings defined in the alert rule, such as muting or grouping, are preserved.
|
||||||
|
|
||||||
### Lifecycle of stale alert instances
|
|
||||||
|
|
||||||
An alert instance is considered stale if its dimension or series has disappeared from the query results entirely for two evaluation intervals.
|
|
||||||
|
|
||||||
Stale alert instances that are in the **Alerting**, **No Data**, or **Error** states transition to the **Normal** state as **Resolved**. Once transitioned, these resolved alert instances are routed for notifications like other resolved alerts.
|
|
||||||
|
|
||||||
## Modify the `No Data` or `Error` state
|
## Modify the `No Data` or `Error` state
|
||||||
|
|
||||||
These states are supported only for Grafana-managed alert rules.
|
These states are supported only for Grafana-managed alert rules.
|
||||||
@ -121,7 +133,7 @@ However, in situations where strict monitoring is critical, relying solely on th
|
|||||||
|
|
||||||
Occasionally, an alert instance may be in a state that isn't immediately clear to everyone. For example:
|
Occasionally, an alert instance may be in a state that isn't immediately clear to everyone. For example:
|
||||||
|
|
||||||
- Stale alert instances in the `Alerting` state transition to the `Normal` state when the series disappear.
|
- [Stale alert instances](#stale-alert-instances-missingseries) in the `Alerting` state transition to the `Normal` state when the series disappear.
|
||||||
- If "no data" handling is configured to transition to a state other than `No Data`.
|
- If "no data" handling is configured to transition to a state other than `No Data`.
|
||||||
- If "error" handling is configured to transition to a state other than `Error`.
|
- If "error" handling is configured to transition to a state other than `Error`.
|
||||||
- If the alert rule is deleted, paused, or updated in some cases, the alert instance also transitions to the `Normal` state.
|
- If the alert rule is deleted, paused, or updated in some cases, the alert instance also transitions to the `Normal` state.
|
||||||
@ -130,7 +142,7 @@ In these situations, the evaluation state may differ from the alert state, and i
|
|||||||
|
|
||||||
The `grafana_state_reason` annotation is included in these situations, providing the reason that explains why the alert instance transitioned to its current state. For example:
|
The `grafana_state_reason` annotation is included in these situations, providing the reason that explains why the alert instance transitioned to its current state. For example:
|
||||||
|
|
||||||
- Stale alert instances in the `Normal` state include the `grafana_state_reason` annotation with the value **MissingSeries**.
|
- [Stale alert instances](#stale-alert-instances-missingseries) in the `Normal` state include the `grafana_state_reason` annotation with the value **MissingSeries**.
|
||||||
- If "no data" or "error" handling transitions to the `Normal` state, the `grafana_state_reason` annotation is included with the value **No Data** or **Error**, respectively.
|
- If "no data" or "error" handling transitions to the `Normal` state, the `grafana_state_reason` annotation is included with the value **No Data** or **Error**, respectively.
|
||||||
- If the alert rule is deleted or paused, the `grafana_state_reason` is set to **Paused** or **RuleDeleted**. For some updates, it is set to **Updated**.
|
- If the alert rule is deleted or paused, the `grafana_state_reason` is set to **Paused** or **RuleDeleted**. For some updates, it is set to **Updated**.
|
||||||
|
|
||||||
|
Reference in New Issue
Block a user