Files
Pepe Cano 813632dff1 Alerting Docs: Update Alert rule and Notification Templating (#94069)
* Update links to the new location of the Templates intro page

* Set alert rule template structure

* Update the `Template notifications` structure/sidebar

* Move examples to examples pages

* Update menu titles and URL paths

* WIP reference

* added intro to examples.md

* reference page section intros

* examples title

* Template

* Continue Alert Template Reference

* example sections

* Almost finish with Reference page

* weight pages

* WIP - template variables Intro

* Edits Alert rule templates + Differences with notification templates

* examples-draft-2

* how to map annotations to notif templates

* examples v2

* moved example

* Continue Template annotations and labels

* Preview annotation templates

* Complete `Template annotations and labels`

* Update `Differences with notification templates`

* Polish a bit `Differences with notification tempaltes`

* added logic examples to `Examples`

* Finish sharing Template language

* Minor: set `anchor` link

* Minor edit

* Minor edit

* Minor changes

* Template language changes

* Document missing functions for alert rule templates

* 'Examples' v3

* `Examples` v3.1

* `Examples` v3.2

* added context/intro to `Examples`

* added a how to section to `Examples`

* moved legacy info in `Examples`

* added examples, sections in `Examples`

* formatting

* examples v5

* examples v5.1

* Complete update to `Notification template reference`

* Update `$values` description

* moved steps to `index`

* fixed urls

* removed section in `examples`

* added refs

* fixed urls

* refs fix

* blended the use case into the explanation

* Notification template reference: Minor readibility changes

* Edit `$value` in Alert rule template reference

* Edit `Manage notification templates

* links to ref and lang pages

* Minor copy changes

* Edit Template annotation overview + Prepare Template notifications overview

* `Select` vs `Choose`

* Template Overview

* Fix `heading` does not match `title`

* examples v6

* examples v6.1

* Fix spell errors

* Run `prettier --write` to fix md format errors

* examples v7

* examples v7.1

* examples v7.2

* examples v7.4

* examples v7.5

* examples v7.6

* all pretty, no pity

* Update alert rule examples

* Complete `Labels and annotations template examples`

* Minor changes

* Use `.` notation instead of `index` in examples

* rename `extendeddata` to `notificationdata`

* Complete Notification examples

* minor changes

* Set `go` language when displaying template examples

* Revert "minor changes"

This reverts commit f647c55c8718a4b5c5ab51b3ef88ef0605de3c84.

* Update example and admonition for consistency

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/configure-notifications/template-notifications/manage-notification-templates.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/configure-notifications/template-notifications/manage-notification-templates.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/shared/alerts/template-language.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/shared/alerts/template-language.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/configure-notifications/template-notifications/manage-notification-templates.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/configure-notifications/template-notifications/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/language.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/examples.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/examples.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/configure-notifications/template-notifications/examples.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/alerting-rules/templates/examples.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/configure-notifications/template-notifications/examples.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/configure-notifications/template-notifications/manage-notification-templates.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/configure-notifications/template-notifications/manage-notification-templates.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/configure-notifications/template-notifications/manage-notification-templates.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/configure-notifications/template-notifications/manage-notification-templates.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/configure-notifications/template-notifications/manage-notification-templates.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* use `add` instead of `enrich`

* fix prettier content errors

* minor copy change

* Specify which Alert properties are exclusive to Grafana-managed alerts

* Use `Monitor` instead of `Meta monitoring`

* Be more detailed about notifation template limitations

---------

Co-authored-by: tonypowa <tonypowa@gmail.com>
Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>
2024-10-23 12:43:28 +02:00

12 KiB

canonical description keywords labels title menuTitle weight refs
https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/examples/ Examples of notification templates
grafana
alerting
templating
notification templates
products
cloud
enterprise
oss
Notification template examples Examples 103
template-annotations-and-labels template-notifications manage-notification-templates reference reference-notification-data reference-alert language
pattern destination
/docs/grafana/ /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/
pattern destination
/docs/grafana-cloud/ /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/
pattern destination
/docs/grafana/ /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/
pattern destination
/docs/grafana-cloud/ /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/
pattern destination
/docs/grafana/ /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/manage-notification-templates/
pattern destination
/docs/grafana-cloud/ /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/manage-notification-templates/
pattern destination
/docs/grafana/ /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/
pattern destination
/docs/grafana-cloud/ /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/
pattern destination
/docs/grafana/ /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/#notification-data
pattern destination
/docs/grafana-cloud/ /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/#notification-data
pattern destination
/docs/grafana/ /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/#alert
pattern destination
/docs/grafana-cloud/ /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/#alert
pattern destination
/docs/grafana/ /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/language/
pattern destination
/docs/grafana-cloud/ /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/language/

Notification template examples

Notification templates allows you to change the default notification messages.

You can modify the content and format of notification messages. For example, you can customize the content to show only specific information or adjust the format to suit a particular contact point, such as Slack or Email.

{{% admonition type="note" %}} Avoid adding extra information about alert instances in notification templates, as this information is only be visible in the notification message.

Instead, you should use annotations or labels to add information directly to the alert, ensuring it's also visible in the alert state and alert history within Grafana. You can then print the new alert annotation or label in notification templates. {{% /admonition %}}

This page provides various examples illustrating how to template common notification messages. For more details about notification templates, refer to:

Basic examples

Notification templates can access the notification data using the dot (.). The following examples demonstrate some basic uses of the template language.

For instance, to check if there are common labels (.CommonLabels) for all alerts in the notification, use if:

{{ define "custom_message" -}}
{{ if .CommonLabels }}
Alerts have common labels
{{ else }}
There are no common labels
{{ end }}
{{ end }}

To iterate on the alerts in the notification and print a specific label, use range and index:

{{ define "custom_message" -}}
{{ range .Alerts }}
The name of the alert is {{ index .Labels "alertname" }}
{{ end }}
{{ end }}

Alternatively, you can use the . notation to print the value of the key.

{{ define "custom_message" -}}
{{ range .Alerts }}
The name of the alert is {{ .Labels.alertname }}
{{ end }}
{{ end }}
The name of the alert is InstanceDown

The name of the alert is CpuOverload

Print alerts with summary and description

Here's an example that displays the summary and description annotations for each alert in the notification.

{{ define "custom.alerts" -}}
{{ len .Alerts }} alert(s)
{{ range .Alerts -}}
  {{ template "alert.summary_and_description" . -}}
{{ end -}}
{{ end -}}
{{ define "alert.summary_and_description" }}
  Summary: {{.Annotations.summary}}
  Status: {{ .Status }}
  Description: {{.Annotations.description}}
{{ end -}}

In this example:

  • A template (alert.summary_and_description) is defined to print the summary, status, and description of one alert.
  • The main template custom.alerts iterates the list of alerts (.Alerts) in notification data, executing the alert.summary_and_description template to print the details of each alert.

The notification message would look like this:

2 alert(s)

  Summary: The database server db1 has exceeded 75% of available disk space.
  Status: firing
  Description: This alert fires when a database server is at risk of running out of disk space. You should take measures to increase the maximum available disk space as soon as possible to avoid possible corruption.

  Summary: The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes.
  Status: resolved
  Description: This alert fires when a web server responds with more 5xx errors than is expected. This could be an issue with the web server or a backend service.

Print firing and resolved alerts

The following example is similar to the previous one, but it separates firing and resolved alerts.

{{ define "custom.firing_and_resolved_alerts" -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ range .Alerts.Resolved -}}
  {{ template "alert.summary_and_description" . -}}
{{ end }}
{{ len .Alerts.Firing }} firing alert(s)
{{ range .Alerts.Firing -}}
  {{ template "alert.summary_and_description" . -}}
{{ end -}}
{{ end -}}
{{ define "alert.summary_and_description" }}
  Summary: {{.Annotations.summary}}
  Status: {{ .Status }}
  Description: {{.Annotations.description}}
{{ end -}}

Instead of .Alerts, the template accesses .Alerts.Firing and .Alerts.Resolved separately to print details for each alert.

The output might now look like this:

1 resolved alert(s)

  Summary: The database server db1 has exceeded 75% of available disk space.
  Status: resolved
  Description: This alert fires when a database server is at risk of running out of disk space. You should take measures to increase the maximum available disk space as soon as possible to avoid possible corruption.

1 firing alert(s)

  Summary: The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes.
  Status: firing
  Description: This alert fires when a web server responds with more 5xx errors than is expected. This could be an issue with the web server or a backend service.

Print common labels and annotations

This example displays only the labels and annotations that are common to all alerts in the notification.

{{ define "custom.common_labels_and_annotations" -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ len .Alerts.Firing }} firing alert(s)
Common labels: {{ len .CommonLabels.SortedPairs }}
{{ range .CommonLabels.SortedPairs -}}
- {{ .Name }} = {{ .Value }}
{{ end }}
Common annotations: {{ len .CommonAnnotations.SortedPairs }}
{{ range .CommonAnnotations.SortedPairs }}
- {{ .Name }} = {{ .Value }}
{{ end }}
{{ end -}}

Note that .CommonAnnotations and .CommonLabels are part of notification data.

1 resolved alert(s)
1 firing alert(s)
Common labels: 2
- grafana_folder = server_alerts
- team = server_admin

Common annotations: 0

Print individual labels and annotations

This example displays all labels and annotations for each alert in the notification.

{{ define "custom.alert_labels_and_annotations" -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ range .Alerts.Resolved -}}
  {{ template "alert.labels_and_annotations" . -}}
{{ end }}
{{ len .Alerts.Firing }} firing alert(s)
{{ range .Alerts.Firing -}}
  {{ template "alert.labels_and_annotations" . -}}
{{ end -}}
{{ end -}}
{{ define "alert.labels_and_annotations" }}
Alert labels: {{ len .Labels.SortedPairs }}
{{ range .Labels.SortedPairs -}}
- {{ .Name }} = {{ .Value }}
{{ end -}}
Alert annotations: {{ len .Annotations.SortedPairs }}
{{ range .Annotations.SortedPairs -}}
- {{ .Name }} = {{ .Value }}
{{ end -}}
{{ end -}}

In this example:

  • The custom.alert_labels_and_annotations template iterates over the list of resolved and firing alerts, similar to previous examples. It then executes alert.labels_and_annotations for each alert.
  • The alert.labels_and_annotations template prints all the alert labels and annotations by accessing .Labels.SortedPairs and .Annotations.SortedPairs.
1 resolved alert(s)

Alert labels: 4
- alertname = db_server_disk_space
- grafana_folder = server_alerts
- server = db1
- team = server_admin

Alert annotations: 2
- summary = The database server db1 has exceeded 75% of available disk space.
- description = This alert fires when a database server is at risk of running out of disk space. You should take measures to increase the maximum available disk space as soon as possible to avoid possible corruption.

1 firing alert(s)

Alert labels: 4
- alertname = web_server_http_errors
- grafana_folder = server_alerts
- server = web1
- team = server_admin

Alert annotations: 2
- summary = The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes.
- description = This alert fires when a web server responds with more 5xx errors than is expected. This could be an issue with the web server or a backend service.

Print URLs for runbook and alert data in Grafana

Note that the following example works only for Grafana-managed alerts. It displays some alert data such as DashboardURL, PanelURL, and SilenceURL, which are exclusive to Grafana-managed alerts.

{{ define "custom.alert_additional_details" -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ range .Alerts.Resolved -}}
  {{ template "alert.additional_details" . -}}
{{ end }}
{{ len .Alerts.Firing }} firing alert(s)
{{ range .Alerts.Firing -}}
  {{ template "alert.additional_details" . -}}
{{ end -}}
{{ end -}}
{{ define "alert.additional_details" }}
- Dashboard: {{ .DashboardURL }}
- Panel: {{ .PanelURL }}
- AlertGenerator: {{ .GeneratorURL }}
- Silence: {{ .SilenceURL }}
- RunbookURL: {{ .Annotations.runbook_url}}
{{ end -}}

The output of this template looks like this:

1 resolved alert(s)

- Dashboard: https://example.com/d/
- Panel: https://example.com/d/
- AlertGenerator: ?orgId=1
- Silence: https://example.com/alerting/silence/new
- RunbookURL: https://example.com/on-call/db_server_disk_space

1 firing alert(s)

- Dashboard: https://example.com/d/
- Panel: https://example.com/d/
- AlertGenerator: ?orgId=1
- Silence: https://example.com/alerting/silence/new
- RunbookURL: https://example.com/on-call/web_server_http_errors