5c1b263664
* docs(alerting): improve `Intro > Alert rule evaluation` docs * Update Introduction to Grafana Alerting * Simplify `Intro > Alert rules` and related docs * minor copy change phrasing GMA and DS differences * fix vale error
19 KiB
| aliases | canonical | description | keywords | labels | title | weight | refs | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-data-source-managed-rule/ | Configure data source-managed alert rules alert for an external Grafana Mimir or Loki instance |
|
|
Configure data source-managed alert rules | 400 |
|
Configure data source-managed alert rules
Data source-managed alert rules are alert rules that are stored in the data source, such as in Prometheus or Loki, rather than in Grafana.
In Grafana Alerting, you can:
- Create and edit data source-managed rules for Grafana Mimir and Grafana Loki data sources.
- View rules from Prometheus data sources when Manage alerts via Alerting UI is enabled. However, you cannot create or edit these rules in Grafana.
- Import data source-managed rules from Loki, Mimir, and Prometheus into Grafana-managed rules.
{{< admonition type="note" >}} Data source-managed rules are supported for horizontal scalability, but they can introduce more operational complexity than Grafana-managed alert rules.
We recommend using Grafana-managed alert rules whenever possible, as they provide a richer feature set and better integration with the full Grafana Alerting workflow. {{< /admonition >}}
Comparison with Grafana-managed rules
The table below compares Grafana-managed and data source-managed alert rules.
Feature |
Grafana-managed alert rule |
Data source-managed alert rule |
|---|---|---|
| Supported data sources | All backend data sources enabling the alerting option |
Only supports creating rules for Mimir and Loki data sources |
| Mix and match data sources | Yes | No |
| Add expressions to transform |
Yes | No |
| No data and error states | Yes | No |
| Stale alert instances | Yes | No |
| Images in alert notifications | Yes | No |
| Role-based access control | Yes | No |
| Alert state history | Yes | No |
| Alert version history | Yes | No |
| Terraform provisioning | Yes | No |
| Recording rules | Yes | Yes |
| Organization | Organize and manage access with folders | Use namespaces |
| Alert rule evaluation | Alert evaluation is done in Grafana | Alert rule evaluation is done in the data source and allow for horizontal scaling |
| Scaling | Alert rules are stored in the Grafana database. | Alert rules are stored within the data source and allow for horizontal scaling |
The following diagram shows the architecture of a Mimir setup that uses data source-managed alert rules.
{{< figure src="/media/docs/alerting/mimir-managed-alerting-architecture-v2.png" max-width="750px" alt="Data source-managed alerting architecture based on Grafana Mimir" >}}
Create data source-managed alert rules
To create or edit data source-managed alert rules, follow these instructions.
Before you begin
Verify that you have write permission to the Mimir or Loki data source. Otherwise, you cannot create or update data source-managed alert rules.
Enable the Ruler API
For more information, refer to the Mimir Ruler API or Loki Ruler API.
-
Mimir - use the
/prometheusprefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the Query API and Ruler API are under the same URL. You cannot provide a separate URL for the Ruler API. -
Loki - The
localrule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.
Permissions
Alert rules for Mimir or Loki instances can be edited or deleted by users with Editor or Admin roles.
If you do not want to manage alert rules for a particular data source, go to its settings and clear the Manage alerts via Alerting UI checkbox.
Provisioning
Note that if you delete an alert resource created in the UI, you can no longer retrieve it.
To backup and manage alert rules, you can provision alerting resources using options such as configuration files, Terraform, or the Alerting API.
Set alert rule name
{{< docs/shared lookup="alerts/configure-alert-rule-name.md" source="grafana" version="<GRAFANA_VERSION>" >}}
Define query and condition
Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.
{{< admonition type="note" >}} By default, new alert rules are Grafana-managed. To switch to Data source-managed, follow these instructions. {{< /admonition >}}
-
Select a Prometheus-based data source from the drop-down list.
You can also click Open advanced data source picker to find more options.
-
Enter a PromQL or LogQL query, including the alert condition.
-
In the Rule type option, select Data source-managed.
-
Click Preview alerts.
Set alert evaluation behavior
Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.
-
Select a namespace or click + New namespace.
-
Select an evaluation group or click + New evaluation group.
If you are creating a new evaluation group, specify the interval for the group.
All rules within the same group are evaluated sequentially over the same time interval. You can reorder them from the Alert rules page.
-
Enter a pending period.
The pending period is the period in which an alert rule can be in breach of the condition until it fires.
Once a condition is met, the alert goes into the Pending state. If the condition remains active for the duration specified, the alert transitions to the Firing state, else it reverts to the Normal state.
Configure labels and notifications
Add labels to your alert rules to set which notification policy should handle your firing alert instances.
All alert rules and instances, irrespective of their labels, match the default notification policy. If there are no nested policies, or no nested policies match the labels in the alert rule or alert instance, then the default notification policy is the matching policy.
-
Add labels if you want to change the way your notifications are routed.
Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.
Configure notification message
Use annotations to add information to alert messages that can help respond to the alert.
Annotations are included by default in notification messages, and can use text or templates to display dynamic data from queries.
Grafana provides several optional annotations.
-
Optional: Add a summary.
Short summary of what happened and why.
-
Optional: Add a description.
Description of what the alert rule does.
-
Optional: Add a Runbook URL.
Webpage where you keep your runbook for the alert
-
Optional: Add a custom annotation.
Add any additional information that could help address the alert.
-
Optional: Link dashboard and panel.
Link the alert rule to a panel to facilitate alert investigation.
-
Click Save rule.