Files
Pepe Cano c84388f550 docs(alerting): Examples of dynamic labels and dynamic thresholds (#105776)
* docs(alerting): Add dynamic thresholds example

* update intro

* docs(alerting): Example of dynamic labels

* fix template example

* Update docs/sources/alerting/best-practices/dynamic-labels.md

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>

* Update docs/sources/alerting/best-practices/dynamic-labels.md

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>

* Update docs/sources/alerting/best-practices/dynamic-labels.md

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>

* Update docs/sources/alerting/best-practices/dynamic-labels.md

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>

* Update docs/sources/alerting/best-practices/dynamic-labels.md

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>

* Update docs/sources/alerting/best-practices/dynamic-thresholds.md

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>

* Update docs/sources/alerting/best-practices/dynamic-thresholds.md

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>

* Update docs/sources/alerting/best-practices/dynamic-thresholds.md

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>

* Update docs/sources/alerting/best-practices/dynamic-labels.md

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>

* Update docs/sources/alerting/best-practices/dynamic-labels.md

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>

* Update docs/sources/alerting/best-practices/dynamic-labels.md

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>

* Update docs/sources/alerting/best-practices/dynamic-labels.md

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>

* Update docs/sources/alerting/best-practices/dynamic-labels.md

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>

* Update docs/sources/alerting/best-practices/dynamic-labels.md

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>

* Update docs/sources/alerting/best-practices/dynamic-labels.md

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>

* fix typo

* fix typo

* Add section `CSV data with Infinity` in Table data example

* Link dynamic threshold example to tabular data requirements

* minor copy changes

* minor heading fix

* Add links (admonition) to Grafana Play examples

* Use `Caveat` instead of `Gotcha`

* Dynamic thresholds: caution message when Math operates on missing series

* Exampleof latency threshold based on traffic

---------

Co-authored-by: Johnny Kartheiser <140559259+JohnnyK-Grafana@users.noreply.github.com>
2025-05-27 17:36:32 +02:00

5.9 KiB
Raw Blame History

canonical description keywords labels menuTitle title weight refs
https://grafana.com/docs/grafana/latest/alerting/best-practices/table-data This example shows how to create an alert rule using table data.
grafana
products
cloud
enterprise
oss
Examples of table data Example of alerting on tabular data 1102
testdata-data-source multi-dimensional-example infinity-csv
pattern destination
/docs/grafana/ /docs/grafana/<GRAFANA_VERSION>/datasources/testdata/
pattern destination
/docs/grafana-cloud/ /docs/grafana-cloud/connect-externally-hosted/data-sources/testdata/
pattern destination
/docs/grafana/ /docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/multi-dimensional-alerts/
pattern destination
/docs/grafana-cloud/ /docs/grafana-cloud/alerting-and-irm/alerting/best-practices/multi-dimensional-alerts/
pattern destination
/docs/grafana/ /docs/plugins/yesoreyeram-infinity-datasource/latest/csv/

Example of alerting on tabular data

Not all data sources return time series data. SQL databases, CSV files, and some APIs often return results as rows or arrays of columns or fields — commonly referred to as tabular data.

This example shows how to create an alert rule using data in table format. Grafana treats each row as a separate alert instance, as long as the data meets the expected format.

How Grafana Alerting evaluates tabular data

When a query returns data in table format, Grafana transforms each row into a separate alert instance.

To evaluate each row (alert instance), it expects:

  1. Only one numeric column. This is the value used for evaluating the alert condition.
  2. Non-numeric columns. These columns defines the label set. The column name becomes a label name; and the cell value becomes the label value.
  3. Unique label sets per row. Each row must be uniquely identifiable by its labels. This ensures each row represents a distinct alert instance.

{{< admonition type="caution" >}} These three conditions must be met—otherwise, Grafana cant evaluate the table data and the rule will fail. {{< /admonition >}}

Example overview

Imagine you store disk usage in a DiskSpace table and you want to trigger alerts when the available space drops below 5%.

Time Host Disk PercentFree
2021-06-07 web1 /etc 3
2021-06-07 web2 /var 4
2021-06-07 web3 /var 8

To calculate the free space per Host and Disk in this case, you can use $__timeFilter to filter by time but without returning the date to Grafana:

SELECT
  Host,
  Disk,
  AVG(PercentFree) AS PercentFree
FROM DiskSpace
WHERE $__timeFilter(Time)
GROUP BY Host, Disk

This query returns the following table response:

Host Disk PercentFree
web1 /etc 3
web2 /var 4
web3 /var 8

When Alerting evaluates the query response, the data is transformed into three alert instances as previously detailed:

  • The numeric column becomes the value for the alert condition.
  • Additional columns define the label set for each alert instance.
Alert instance Value
{Host="web1", Disk="/etc"} 3
{Host="web2", Disk="/var"} 4
{Host="web3", Disk="/var"} 8

Finally, an alert condition that checks for less than 5% of free space ($A < 5) would result in two alert instances firing:

Alert instance Value State
{Host="web1", Disk="/etc"} 3 Firing
{Host="web2", Disk="/var"} 4 Firing
{Host="web3", Disk="/var"} 8 Normal

Try it with TestData

To test this quickly, you can simulate the table using the TestData data source:

  1. Add the TestData data source through the Connections menu.

  2. Go to Alerting and create an alert rule

  3. Select TestData as the data source.

  4. From Scenario, select CSV Content and paste this CSV:

    host, disk, percentFree
    web1, /etc, 3
    web2, /var, 4
    web3, /var, 8
    
  5. Set a condition like $A < 5 and Preview the alert.

    Grafana evaluates the table data and fires the two first alert instances.

    {{< figure src="/media/docs/alerting/example-table-data-preview.png" max-width="750px" alt="Alert preview with tabular data using the TestData data source" >}}

    {{< docs/play title="this alert example" url="https://play.grafana.org/alerting/grafana/eemqylh1l8tfkf/view" >}}

CSV data with Infinity

Note that when the Infinity plugin fetches CSV data, all the columns are parsed and returned as strings. By default, this causes the query expression to fail in Alerting.

To make it work, you need to format the CSV data as expected by Grafana Alerting.

In the query editor, specify the column names and their types to ensure that only one column is treated as a number.

{{< figure src="/media/docs/alerting/example-table-data-infinity-csv-data.png" max-width="750px" alt="Using the Infinity data source plugin to fetch CSV data in Alerting" >}}

Differences with time series data

Working with time series is similar—each series is treated as a separate alert instance, based on its label set.

The key difference is the data format:

  • Time series data contains multiple values over time, each with its own timestamp. To evaluate the alert condition, alert rules must reduce each series to a single number using a function like last(), avg(), or max().
  • Tabular data doesnt require reduction, as each row contains only a single numeric value used to evaluate the alert condition.

For comparison, see the multi-dimensional time series data example.