mirror of
https://github.com/grafana/grafana.git
synced 2025-09-18 05:32:50 +08:00
DOCS: Prometheus data sources documentation overhaul (#104427)
* initial updates * additional updates - index and config dos * made edits, renamed the index files. * Close unclosed shortcode Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * updated main and config docs * edits to query editor and templates * final edits to landing page * added updates * made a few additional edits * edits * final edits and changes per dev feedback * reviewed titles and headings; ran prettier * added ref URIs * Update docs/sources/datasources/prometheus/template-variables/_index.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update docs/sources/datasources/prometheus/configure/_index.md Co-authored-by: ismail simsek <ismailsimsek09@gmail.com> * Update docs/sources/datasources/prometheus/template-variables/_index.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * updates based on feedback * ran prettier * updates due to feedback --------- Signed-off-by: Jack Baldry <jack.baldry@grafana.com> Co-authored-by: Jack Baldry <jack.baldry@grafana.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: ismail simsek <ismailsimsek09@gmail.com>
This commit is contained in:
Larissa Wandzura
@ -26,21 +26,11 @@ refs:
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/getting-started/get-started-grafana-prometheus/#get-started-with-grafana-and-prometheus
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/getting-started/get-started-grafana-prometheus/#get-started-with-grafana-and-prometheus
|
||||
configure-grafana-configuration-file-location:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#configuration-file-location
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#configuration-file-location
|
||||
provisioning-data-sources:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources
|
||||
explore:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
set-up-grafana-monitoring:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/setup-grafana/set-up-grafana-monitoring/
|
||||
@ -73,145 +63,103 @@ refs:
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/fundamentals/intro-to-prometheus/
|
||||
configure-prometheus-data-source:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/configure-prometheus-data-source/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/configure
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/configure-prometheus-data-source/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/configure
|
||||
annotate-visualizations:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/visualizations/dashboards/build-dashboards/annotate-visualizations/
|
||||
recorded-queries:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/recorded-queries/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/recorded-queries/
|
||||
transformations:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/transform-data/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/visualizations/panels-visualizations/query-transform-data/transform-data/
|
||||
alerting:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/
|
||||
visualizations:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/visualizations/panels-visualizations/visualizations/
|
||||
variables:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/dashboards/variables/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/visualizations/dashboards/variables/
|
||||
|
||||
---
|
||||
|
||||
# Prometheus data source
|
||||
|
||||
Prometheus is an open source database that uses a telemetry collector agent to scrape and store metrics used for monitoring and alerting. Grafana provides native support for Prometheus. If you are just getting started with Prometheus, see [What is Prometheus?](ref:intro-to-prometheus).
|
||||
Prometheus is an open source database that uses a telemetry collector agent to scrape and store metrics used for monitoring and alerting.
|
||||
|
||||
{{% admonition type="tip" %}}
|
||||
For instructions on downloading Prometheus see [Get started with Grafana and Prometheus](ref:get-started-prometheus).
|
||||
Grafana provides native support for Prometheus, so you don't need to install a plugin.
|
||||
|
||||
If you’re ready to start visualizing your metrics, check out our Prometheus Learning Journeys:
|
||||
|
||||
- [Connect to a Prometheus data source in Grafana Cloud](https://www.grafana.com/docs/learning-journeys/prometheus/) to visualize your metrics directly from where they are stored.
|
||||
- [Send metrics to Grafana Cloud using Prometheus remote write](https://www.grafana.com/docs/learning-journeys/prom-remote-write/) to explore Grafana Cloud without making significant changes to your existing configuration.
|
||||
{{% /admonition %}}
|
||||
|
||||
For instructions on how to add a data source to Grafana, refer to the [administration documentation](ref:administration-documentation).
|
||||
Only users with the organization `administrator` role can add data sources and edit existing data sources.
|
||||
Administrators can also [configure the data source via YAML](#provision-the-data-source) with Grafana's provisioning system.
|
||||
|
||||
Once you've added the Prometheus data source, you can [configure it](ref:configure-prometheus-data-source) so that your Grafana instance's users can create queries in its [query editor](query-editor/) when they [build dashboards](ref:build-dashboards), use [Explore](ref:explore), and [annotate visualizations](ref:annotate-visualizations).
|
||||
|
||||
The following guides will help you get started with the Prometheus data source:
|
||||
The following documentation will help you get started working with Prometheus and Grafana:
|
||||
|
||||
- [What is Prometheus?](ref:intro-to-prometheus)
|
||||
- [Prometheus data model](https://prometheus.io/docs/concepts/data_model/)
|
||||
- [Getting started](https://prometheus.io/docs/prometheus/latest/getting_started/)
|
||||
- [Configure the Prometheus data source](ref:configure-prometheus-data-source)
|
||||
- [Prometheus query editor](query-editor/)
|
||||
- [Template variables](template-variables/)
|
||||
|
||||
## Exemplars
|
||||
|
||||
In Prometheus, an **exemplar** is a specific trace that represents a measurement taken within a given time interval. While metrics provide an aggregated view of your system, and traces offer a detailed view of individual requests, exemplars serve as a bridge between the two, linking high-level metrics to specific traces for deeper insights.
|
||||
|
||||
Exemplars associate higher-cardinality metadata from a specific event with traditional time series data. Refer to [Introduction to exemplars](ref:exemplars) in the Prometheus documentation for detailed information on how they work.
|
||||
|
||||
Grafana can show exemplar data alongside a metric both in Explore and in Dashboards.
|
||||
|
||||
{{< figure src="/static/img/docs/v74/exemplars.png" class="docs-image--no-shadow" caption="Exemplar window" >}}
|
||||
|
||||
You add exemplars when you configure the Prometheus data source.
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/exemplars-10-1.png" max-width="500px" class="docs-image--no-shadow" >}}
|
||||
|
||||
## Prometheus API
|
||||
|
||||
The Prometheus data source also works with other projects that implement the [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/).
|
||||
|
||||
For more information on how to query other Prometheus-compatible projects from Grafana, refer to the specific project's documentation:
|
||||
For more information on how to query other Prometheus-compatible projects from Grafana, refer to the specific product's documentation:
|
||||
|
||||
- [Grafana Mimir](/docs/mimir/latest/)
|
||||
- [Thanos](https://thanos.io/tip/components/query.md/)
|
||||
|
||||
## Provision the data source
|
||||
|
||||
You can define and configure the data source in YAML files as part of Grafana's provisioning system.
|
||||
For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](ref:provisioning-data-sources).
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
Once you have provisioned a data source you cannot edit it.
|
||||
{{% /admonition %}}
|
||||
|
||||
### Provisioning example
|
||||
|
||||
```yaml
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
# Access mode - proxy (server in the UI) or direct (browser in the UI).
|
||||
url: http://localhost:9090
|
||||
jsonData:
|
||||
httpMethod: POST
|
||||
manageAlerts: true
|
||||
prometheusType: Prometheus
|
||||
prometheusVersion: 2.44.0
|
||||
cacheLevel: 'High'
|
||||
disableRecordingRules: false
|
||||
incrementalQueryOverlapWindow: 10m
|
||||
exemplarTraceIdDestinations:
|
||||
# Field with internal link pointing to data source in Grafana.
|
||||
# datasourceUid value can be anything, but it should be unique across all defined data source uids.
|
||||
- datasourceUid: my_jaeger_uid
|
||||
name: traceID
|
||||
|
||||
# Field with external link.
|
||||
- name: traceID
|
||||
url: 'http://localhost:3000/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Jaeger%22,%7B%22query%22:%22$${__value.raw}%22%7D%5D'
|
||||
```
|
||||
|
||||
## View Grafana metrics with Prometheus
|
||||
|
||||
Grafana exposes metrics for Prometheus on the `/metrics` endpoint.
|
||||
We also bundle a dashboard within Grafana so you can start viewing your metrics faster.
|
||||
Grafana exposes metrics for Prometheus on the `/metrics` endpoint and includes a pre-built dashboard to help you start visualizing your metrics immediately.
|
||||
|
||||
**To import the bundled dashboard:**
|
||||
Complete the following steps to import the pre-built dashboard:
|
||||
|
||||
1. Navigate to the data source's [configuration page](ref:configure-prometheus-data-source).
|
||||
1. Select the **Dashboards** tab.
|
||||
|
||||
This displays dashboards for Grafana and Prometheus.
|
||||
|
||||
1. Select **Import** for the dashboard to import.
|
||||
1. Navigate to the Prometheus [configuration page](ref:configure-prometheus-data-source).
|
||||
1. Click the **Dashboards** tab.
|
||||
1. Locate the **Grafana metrics** dashboard in the list and click **Import**.
|
||||
|
||||
For details about these metrics, refer to [Internal Grafana metrics](ref:set-up-grafana-monitoring).
|
||||
|
||||
## Amazon Managed Service for Prometheus
|
||||
|
||||
The Prometheus data source with Amazon Managed Service for Prometheus is deprecated. Please use the [Amazon Managed service for Prometheus data source](https://grafana.com/grafana/plugins/grafana-amazonprometheus-datasource/). Migrations steps are detailed in the link.
|
||||
Grafana has deprecated the Prometheus data source for Amazon Managed Service for Prometheus. Use the [Amazon Managed Service for Prometheus data source](https://grafana.com/grafana/plugins/grafana-amazonprometheus-datasource/) instead. The linked documentation outlines the migration steps.
|
||||
|
||||
## Azure authentication settings
|
||||
## Get the most out of the Prometheus data source
|
||||
|
||||
The Prometheus data source works with Azure authentication. To configure Azure authentication see [Configure Azure Active Directory (AD) authentication](/docs/grafana/latest/datasources/azure-monitor/#configure-azure-active-directory-ad-authentication).
|
||||
After you install and configure Prometheus you can:
|
||||
|
||||
In Grafana Enterprise, update the .ini configuration file: [Configure Grafana](ref:configure-grafana). Depending on your setup, the .ini file is located [here](ref:configure-grafana-configuration-file-location).
|
||||
Add the following setting in the **[auth]** section :
|
||||
|
||||
```bash
|
||||
[auth]
|
||||
azure_auth_enabled = true
|
||||
```
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
If you are using Azure authentication settings do not enable `Forward OAuth identity`. Both use the same HTTP authorization headers. Azure settings will get overwritten by the Oauth token.
|
||||
{{% /admonition %}}
|
||||
|
||||
## Exemplars
|
||||
|
||||
Exemplars associate higher-cardinality metadata from a specific event with traditional time series data. See [Introduction to exemplars](ref:exemplars) in Prometheus documentation for detailed information on how they work.
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
Available in Prometheus v2.26 and higher with Grafana v7.4 and higher.
|
||||
{{% /admonition %}}
|
||||
|
||||
Grafana can show exemplars data alongside a metric both in Explore and in Dashboards.
|
||||
|
||||
{{< figure src="/static/img/docs/v74/exemplars.png" class="docs-image--no-shadow" caption="Screenshot showing the detail window of an Exemplar" >}}
|
||||
|
||||
See the Exemplars section in [Configure Prometheus data source](ref:configure-prometheus-data-source).
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/exemplars-10-1.png" max-width="500px" class="docs-image--no-shadow" caption="Exemplars" >}}
|
||||
|
||||
## Incremental dashboard queries (beta)
|
||||
|
||||
As of Grafana 10, the Prometheus data source can be configured to query live dashboards incrementally, instead of re-querying the entire duration on each dashboard refresh.
|
||||
|
||||
This can be toggled on or off in the data source configuration or provisioning file (under `incrementalQuerying` in jsonData).
|
||||
Additionally, the amount of overlap between incremental queries can be configured using the `incrementalQueryOverlapWindow` jsonData field, the default value is `10m` (10 minutes).
|
||||
|
||||
Increasing the duration of the `incrementalQueryOverlapWindow` will increase the size of every incremental query, but might be helpful for instances that have inconsistent results for recent data.
|
||||
|
||||
## Recording Rules (beta)
|
||||
|
||||
The Prometheus data source can be configured to disable recording rules under the data source configuration or provisioning file (under `disableRecordingRules` in jsonData).
|
||||
- Create a wide variety of [visualizations](ref:visualizations)
|
||||
- Configure and use [templates and variables](ref:variables)
|
||||
- Add [transformations](ref:transformations)
|
||||
- Add [annotations](ref:annotate-visualizations)
|
||||
- Set up [alerting](ref:alerting)
|
||||
- Create [recorded queries](ref:recorded-queries)
|
||||
|
||||
@ -1,200 +0,0 @@
|
||||
---
|
||||
aliases:
|
||||
- ../data-sources/prometheus/
|
||||
- ../features/datasources/prometheus/
|
||||
description: Guide for configuring Prometheus in Grafana
|
||||
keywords:
|
||||
- grafana
|
||||
- prometheus
|
||||
- guide
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Configure Prometheus
|
||||
title: Configure the Prometheus data source
|
||||
weight: 200
|
||||
refs:
|
||||
intro-to-prometheus:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/fundamentals/intro-to-prometheus/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/fundamentals/intro-to-prometheus/
|
||||
exemplars:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/fundamentals/exemplars/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/fundamentals/exemplars/
|
||||
configure-data-links-value-variables:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-data-links/#value-variables
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-data-links/#value-variables
|
||||
alerting-alert-rules:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/
|
||||
---
|
||||
|
||||
# Configure Prometheus
|
||||
|
||||
Grafana ships with built-in support for Prometheus. If you are new to Prometheus the following documentation will help you get started working with Prometheus and Grafana:
|
||||
|
||||
- [What is Prometheus?](ref:intro-to-prometheus)
|
||||
- [Prometheus data model](https://prometheus.io/docs/concepts/data_model/)
|
||||
- [Getting started](https://prometheus.io/docs/prometheus/latest/getting_started/)
|
||||
|
||||
## Configure the data source
|
||||
|
||||
{{< shared id="add-prom-data-source" >}}
|
||||
|
||||
To add the Prometheus data source, complete the following steps:
|
||||
|
||||
1. Click **Connections** in the left-side menu.
|
||||
1. Under **Connections**, click **Add new connection**.
|
||||
1. Enter `Prometheus` in the search bar.
|
||||
1. Click **Prometheus data source**.
|
||||
1. Click **Add new data source** in the upper right.
|
||||
1. Enter a name for the data source.
|
||||
|
||||
{{< /shared >}}
|
||||
|
||||
You will be taken to the **Settings** tab where you will set up your Prometheus configuration.
|
||||
|
||||
## Configuration options
|
||||
|
||||
The following is a list of configuration options for Prometheus.
|
||||
|
||||
The first option to configure is the name of your connection:
|
||||
|
||||
- **Name** - The data source name. This is how you refer to the data source in panels and queries. Examples: prometheus-1, prom-metrics.
|
||||
|
||||
- **Default** - Toggle to select as the default name in dashboard panels. When you go to a dashboard panel this will be the default selected data source.
|
||||
|
||||
### Connection section
|
||||
|
||||
- **Prometheus server URL** - The URL of your Prometheus server. {{< shared id="prom-data-source-url" >}}
|
||||
If your Prometheus server is local, use `http://localhost:9090`. If it's on a server within a network, this is the URL with the port where you are running Prometheus. Example: `http://prometheus.example.orgname:9090`.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
|
||||
If you're running Grafana and Prometheus together in different container environments, each localhost refers to its own container - if the server URL is localhost:9090, that means port 9090 inside the Grafana container, not port 9090 on the host machine.
|
||||
|
||||
You should use the IP address of the Prometheus container, or the hostname if you are using Docker Compose. Alternatively, you can consider `http://host.docker.internal:9090`.
|
||||
|
||||
{{< /admonition >}}
|
||||
|
||||
{{< /shared >}}
|
||||
|
||||
### Authentication section
|
||||
|
||||
There are several authentication methods you can choose in the Authentication section.
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
|
||||
Use TLS (Transport Layer Security) for an additional layer of security when working with Prometheus. For information on setting up TLS encryption with Prometheus see [Securing Prometheus API and UI Endpoints Using TLS Encryption](https://prometheus.io/docs/guides/tls-encryption/). You must add TLS settings to your Prometheus configuration file **prior** to setting these options in Grafana.
|
||||
|
||||
{{% /admonition %}}
|
||||
|
||||
- **Basic authentication** - The most common authentication method. Use your `data source` user name and `data source` password to connect.
|
||||
|
||||
- **With credentials** - Toggle on to enable credentials such as cookies or auth headers to be sent with cross-site requests.
|
||||
|
||||
- **TLS client authentication** - Toggle on to use client authentication. When enabled, add the `Server name`, `Client cert` and `Client key`. The client provides a certificate that is validated by the server to establish the client's trusted identity. The client key encrypts the data between client and server.
|
||||
|
||||
- **With CA cert** - Authenticate with a CA certificate. Follow the instructions of the CA (Certificate Authority) to download the certificate file.
|
||||
|
||||
- **Skip TLS verify** - Toggle on to bypass TLS certificate validation.
|
||||
|
||||
- **Forward OAuth identity** - Forward the OAuth access token (and also the OIDC ID token if available) of the user querying the data source.
|
||||
|
||||
### Custom HTTP headers
|
||||
|
||||
- **Header** - Add a custom header. This allows custom headers to be passed based on the needs of your Prometheus instance.
|
||||
|
||||
- **Value** - The value of the header.
|
||||
|
||||
## Advanced settings
|
||||
|
||||
Following are additional configuration options.
|
||||
|
||||
### Advanced HTTP settings
|
||||
|
||||
- **Allowed cookies** - Specify cookies by name that should be forwarded to the data source. The Grafana proxy deletes all forwarded cookies by default.
|
||||
|
||||
- **Timeout** - The HTTP request timeout. This must be in seconds. The default is 30 seconds.
|
||||
|
||||
### Alerting
|
||||
|
||||
- **Manage alerts via Alerting UI** - Toggle to enable [data source-managed rules in Grafana Alerting](ref:alerting-alert-rules) for this data source. For `Mimir`, it enables managing data source-managed rules and alerts. For `Prometheus`, it only supports viewing existing rules and alerts, which are displayed as data source-managed.
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
|
||||
The **Manage alerts via Alerting UI** toggle is enabled by default. You can change this behavior by setting the [default_manage_alerts_ui_toggle](../../../setup-grafana/configure-grafana/#default_manage_alerts_ui_toggle) option in the Grafana configuration file.
|
||||
|
||||
{{% /admonition %}}
|
||||
|
||||
### Interval behavior
|
||||
|
||||
- **Scrape interval** - Set to the typical scrape and evaluation interval configured in Prometheus. The default is `15s`.
|
||||
|
||||
- **Query timeout** - The default is `60s`.
|
||||
|
||||
### Query editor
|
||||
|
||||
- **Default editor** - Sets a default editor. Options are `Builder` or `Code`. For information on query editor types see [Prometheus query editor](../query-editor/).
|
||||
|
||||
- **Disable metrics lookup** - Toggle on to disable the metrics chooser and metric/label support in the query field's autocomplete. This helps if you have performance issues with large Prometheus instances.
|
||||
|
||||
### Performance
|
||||
|
||||
- **Prometheus type** - The type of your Prometheus server. There are four options: `Prometheus`, `Cortex`, `Mimir`, and `Thanos`.
|
||||
|
||||
- **Cache level** - The browser caching level for editor queries. There are four options: `Low`, `Medium`, `High`, or `None`.
|
||||
|
||||
- **Incremental querying (beta)** - Changes the default behavior of relative queries to always request fresh data from the Prometheus instance. Enable this option to decrease database and network load.
|
||||
|
||||
- **Disable recording rules (beta)** - Toggle on to disable the recording rules. Enable this option to improve dashboard performance.
|
||||
|
||||
### Other
|
||||
|
||||
- **Custom query parameters** - Add custom parameters to the Prometheus query URL. For example `timeout`, `partial_response`, `dedup`, or `max_source_resolution`. Multiple parameters should be concatenated together with an '&'.
|
||||
|
||||
- **HTTP method** - Use either `POST` or `GET` HTTP method to query your data source. `POST` is the recommended and pre-selected method as it allows bigger queries. Change to `GET` if you have a Prometheus version older than 2.1 or if `POST` requests are restricted in your network.
|
||||
|
||||
### Exemplars
|
||||
|
||||
Support for exemplars is available only for the Prometheus data source. If this is your first time working with exemplars see [Introduction to exemplars](ref:exemplars). An exemplar is a specific trace representative of measurement taken in a given time interval.
|
||||
|
||||
- **Internal link** - Toggle on to enable an internal link. When enabled, reveals the data source selector. Select the backend tracing data store for your exemplar data.
|
||||
|
||||
- **URL** - _(Visible if you **disable** `Internal link`)_ Defines the external link's URL trace backend. You can interpolate the value from the field by using the [`${__value.raw}` macro](ref:configure-data-links-value-variables).
|
||||
|
||||
- **Data source** - _(Visible if you **enable** `Internal link`)_ The data source the exemplar will navigate to.
|
||||
|
||||
- **URL label** - Adds a custom display label to override the value of the `Label name` field.
|
||||
|
||||
- **Label name** - The name of the field in the `labels` object used to obtain the traceID property.
|
||||
|
||||
- **Remove exemplar link** - Click to remove existing links.
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
Refer to the following troubleshooting information, as required.
|
||||
|
||||
#### Data doesn't appear in Explore metrics
|
||||
|
||||
<!-- vale Grafana.Spelling = NO -->
|
||||
|
||||
If metric data doesn't appear in Explore after you've successfully tested a connection to a Prometheus data source or sent
|
||||
metrics to Grafana Cloud, ensure that you've selected the correct data source in the **Data source** drop-down menu. If
|
||||
you've used remote_write to send metrics to Grafana Cloud, the data source name follows the convention
|
||||
`grafanacloud-stackname-prom`.
|
||||
|
||||
<!-- vale Grafana.Spelling = YES -->
|
||||
|
||||
The following image shows the **Data source** field in Explore metrics.
|
||||
|
||||
|
||||
300
docs/sources/datasources/prometheus/configure/_index.md
Normal file
300
docs/sources/datasources/prometheus/configure/_index.md
Normal file
@ -0,0 +1,300 @@
|
||||
---
|
||||
aliases:
|
||||
- ../data-sources/prometheus/
|
||||
- ../features/datasources/prometheus/
|
||||
description: Guide for configuring Prometheus in Grafana
|
||||
keywords:
|
||||
- grafana
|
||||
- prometheus
|
||||
- guide
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Configure the Prometheus data source
|
||||
title: Configure the Prometheus data source
|
||||
weight: 200
|
||||
refs:
|
||||
intro-to-prometheus:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/fundamentals/intro-to-prometheus/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/fundamentals/intro-to-prometheus/
|
||||
exemplars:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/fundamentals/exemplars/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/fundamentals/exemplars/
|
||||
configure-data-links-value-variables:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-data-links/#value-variables
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/configure-data-links/#value-variables
|
||||
alerting-alert-rules:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/
|
||||
add-a-data-source:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/#add-a-data-source
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/#add-a-data-source
|
||||
prom-query-editor:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/query-editor
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/prometheus/query-editor
|
||||
default-manage-alerts-ui-toggle:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#default_manage_alerts_ui_toggle
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#default_manage_alerts_ui_toggle
|
||||
provision-grafana:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/provisioning/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/provisioning/
|
||||
manage-alerts-toggle:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#default_manage_alerts_ui_toggle
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#default_manage_alerts_ui_toggle
|
||||
private-data-source-connect:
|
||||
- pattern: /docs/grafana/
|
||||
destination: docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/
|
||||
configure-pdc:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc
|
||||
azure-active-directory:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/#configure-azure-active-directory-ad-authentication
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/azure-monitor/#configure-azure-active-directory-ad-authentication
|
||||
configure-grafana-configuration-file-location:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#configuration-file-location
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#configuration-file-location
|
||||
---
|
||||
|
||||
# Configure the Prometheus data source
|
||||
|
||||
This document provides instructions for configuring the Prometheus data source and explains the available configuration options. Grafana includes built-in support for Prometheus, so you don't need to install a plugin. For general information on adding a data source to Grafana, refer to [Add a data source](ref:add-a-data-source).
|
||||
|
||||
## Before you begin
|
||||
|
||||
- You must have the `Organization administrator` role to add a data source. Administrators can also configure a data source via YAML files.
|
||||
|
||||
- Know which Prometheus-compatible database you are using.
|
||||
|
||||
- Familiarize yourself with your Prometheus security configuration and gather any necessary security certificates and client keys.
|
||||
|
||||
- Verify that data from Prometheus is being written to your Grafana instance.
|
||||
|
||||
## Configure the data source using the UI
|
||||
|
||||
{{< shared id="add-prom-data-source" >}}
|
||||
|
||||
To add the Prometheus data source, complete the following steps:
|
||||
|
||||
1. Click **Connections** in the left-side menu.
|
||||
1. Under **Connections**, click **Add new connection**.
|
||||
1. Enter `Prometheus` in the search bar.
|
||||
1. Click **Prometheus data source**.
|
||||
1. Click **Add new data source** in the upper right.
|
||||
|
||||
{{< /shared >}}
|
||||
|
||||
Grafana takes you to the **Settings** tab where you will set up your Prometheus configuration.
|
||||
|
||||
## Configuration options
|
||||
|
||||
Following is a list of configuration options for Prometheus.
|
||||
|
||||
- **Name** - The data source name. Sets the name you use to refer to the data source in panels and queries. Examples: prometheus-1, prom-metrics.
|
||||
- **Default** - Toggle to select as the default name in dashboard panels. When you go to a dashboard panel this will be the default selected data source.
|
||||
|
||||
**Connection:**
|
||||
|
||||
- **Prometheus server URL** - The URL of your Prometheus server. {{< shared id="prom-data-source-url" >}}
|
||||
If Prometheus is running locally, use `http://localhost:9090`. If it's hosted on a networked server, provide the server’s URL along with the port where Prometheus is running. Example: `http://prometheus.example.orgname:9090`.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
When running Grafana and Prometheus in separate containers, localhost refers to each container’s own network namespace. This means that `localhost:9090` points to port 9090 inside the Grafana container, not on the host machine.
|
||||
|
||||
Use the IP address of the Prometheus container, or the hostname if you are using Docker Compose. Alternatively, you can use `http://host.docker.internal:9090` to reference the host machine.
|
||||
{{< /admonition >}}
|
||||
|
||||
{{< /shared >}}
|
||||
|
||||
**Authentication:**
|
||||
|
||||
There are three authentication options for the Prometheus data source.
|
||||
|
||||
- **Basic authentication** - The most common authentication method.
|
||||
|
||||
- **User** - The username you use to connect to the data source.
|
||||
- **Password** - The password you use to connect to the data source.
|
||||
|
||||
- **Forward OAuth identity** - Forward the OAuth access token (and also the OIDC ID token if available) of the user querying the data source.
|
||||
|
||||
- **No authentication** - Allows access to the data source without any authentication.
|
||||
|
||||
**TLS settings:**
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
Use TLS (Transport Layer Security) for an additional layer of security when working with Prometheus. For information on setting up TLS encryption with Prometheus refer to [Securing Prometheus API and UI Endpoints Using TLS Encryption](https://prometheus.io/docs/guides/tls-encryption/). You must add TLS settings to your Prometheus configuration file **prior** to setting these options in Grafana.
|
||||
{{< /admonition >}}
|
||||
|
||||
- **Add self-signed certificate** - Check the box to authenticate with a CA certificate. Follow the instructions of the CA (Certificate Authority) to download the certificate file. Required for verifying self-signed TLS certificates.
|
||||
- **CA certificate** - Add your certificate.
|
||||
- **TLS client authentication** - Check the box to enable TLS client authentication.
|
||||
- **Server name** - Add the server name, which is used to verify the hostname on the returned certificate.
|
||||
- **Client certificate** - The client certificate is generated from a Certificate Authority or its self-signed. Follow the instructions of the CA (Certificate Authority) to download the certificate file.
|
||||
- **Client key** - Add your client key, which can also be generated from a Certificate Authority (CA) or be self-signed. The client key encrypts data between the client and server.
|
||||
- **Skip TLS verify** - Toggle on to bypass TLS certificate validation. Skipping TLS certificate validation is not recommended unless absolutely necessary or for testing purposes.
|
||||
|
||||
**HTTP headers:**
|
||||
|
||||
Pass along additional information and metadata about the request or response.
|
||||
|
||||
- **Header** - Add a custom header. This allows custom headers to be passed based on the needs of your Prometheus instance.
|
||||
- **Value** - The value of the header.
|
||||
|
||||
**Advanced settings:**
|
||||
|
||||
Following are optional configuration settings you can configure for more control over your data source.
|
||||
|
||||
- **Advanced HTTP settings:**
|
||||
- **Allowed cookies** - Specify cookies by name that should be forwarded to the data source. The Grafana proxy deletes all forwarded cookies by default.
|
||||
- **Timeout** - The HTTP request timeout, must be in seconds.
|
||||
|
||||
**Alerting:**
|
||||
|
||||
- **Manage alerts via Alerting UI** -Toggled on by default. This enables [data source-managed rules in Grafana Alerting](ref:alerting-alert-rules) for this data source. For `Mimir`, it enables managing data source-managed rules and alerts. For `Prometheus`, it only supports viewing existing rules and alerts, which are displayed as data source-managed. Change this by setting the [`default_manage_alerts_ui_toggle`](ref:manage-alerts-toggle) option in the `grafana.ini` configuration file.
|
||||
|
||||
**Interval behavior:**
|
||||
|
||||
- **Scrape interval** - Sets the standard scrape and evaluation interval in Prometheus. The default is `15s`. This interval determines how often Prometheus scrapes targets. Set it to match the typical scrape and evaluation interval in your Prometheus configuration file. If you set a higher value than your Prometheus configuration, Grafana will evaluate data at this interval, resulting in fewer data points.
|
||||
- **Query timeout** - Sets the Prometheus query timeout. The default is `60s`. Without a timeout, complex or inefficient queries can run indefinitely, consuming CPU and memory resources.
|
||||
|
||||
**Query editor:**
|
||||
|
||||
- **Default editor** - Sets the default query editor. Options are `Builder` or `Code`. `Builder` mode helps you build queries using a visual interface. `Code` mode is geared for the experienced Prometheus user with prior expertise in PromQL. For more details on editor types refer to [Prometheus query editor](ref:prom-query-editor). You can switch easily code editors in the Query editor UI.
|
||||
- **Disable metrics lookup** - Toggle on to disable the metrics chooser and metric and label support in the query field's autocomplete. This can improve performance for large Prometheus instances.
|
||||
|
||||
**Performance:**
|
||||
|
||||
- **Prometheus type** - Select the type of your Prometheus-compatible database, such as Prometheus, Cortex, Mimir, or Thanos. Changing this setting will save your current configuration. Different database types support different APIs. For example, some allow `regex` matching for label queries to improve performance, while others provide a metadata API. Setting this incorrectly may cause unexpected behavior when querying metrics and labels. Refer to your Prometheus documentation to ensure you select the correct type.
|
||||
- **Cache level** - Sets the browser caching level for editor queries. There are four options: `Low`, `Medium`, `High`, or `None`. Higher cache settings are recommended for high cardinality data sources.
|
||||
- **Incremental querying (beta)** - Toggle on to enable incremental querying. Enabling this feature changes the default behavior of relative queries. Instead of always requesting fresh data from the Prometheus instance, Grafana will cache query results and only fetch new records. This helps reduce database and network load.
|
||||
- **Query overlap window** - If you are using incremental querying, specify a duration (e.g., 10m, 120s, or 0s). The default is `10m`. This is a buffer of time added to incremental queries and this value is added to the duration of each incremental request.
|
||||
- **Disable recording rules (beta)** - Toggle on to disable the recording rules. When recording rules are disabled, Grafana won't fetch and parse recording rules from Prometheus, improving dashboard performance by reducing processing overhead..
|
||||
|
||||
**Other settings:**
|
||||
|
||||
- **Custom query parameters** - Add custom parameters to the Prometheus query URL, which allow for more control over how queries are executed. Examples: `timeout`, `partial_response`, `dedup`, or `max_source_resolution`. Multiple parameters should be joined using `&`.
|
||||
- **HTTP method** - Select either the `POST` or `GET` HTTP method to query your data source. `POST`is recommended and selected by default, as it supports larger queries. Select `GET` if you're using Prometheus version 2.1 or older, or if your network restricts `POST` requests.
|
||||
Toggle on
|
||||
- **Use series endpoint** - Enabling this option makes Grafana use the series endpoint (/api/v1/series) with the match[] parameter instead of the label values endpoint (/api/v1/label/<label_name>/values). While the label values endpoint is generally more performant, some users may prefer the series endpoint because it supports the `POST` method, whereas the label values endpoint only allows `GET` requests.
|
||||
|
||||
**Exemplars:**
|
||||
|
||||
Support for exemplars is available only for the Prometheus data source. For more information on exemplars refer to [Introduction to exemplars](ref:exemplars). An exemplar is a trace that represents a specific measurement taken within a given time interval.
|
||||
|
||||
Click the **+ sign** to add exemplars.
|
||||
|
||||
- **Internal link** - Toggle on to enable an internal link. This will display the data source selector, where you can choose the backend tracing data store for your exemplar data.
|
||||
- **URL** - _(Visible if you **disable** `Internal link`)_ Defines the external link's URL trace backend. You can interpolate the value from the field by using the [`${__value.raw}` macro](ref:configure-data-links-value-variables).
|
||||
- **Data source** - _(Visible when`Internal link` is enabled.)_ Select the data source that the exemplar will link to from the drop-down.
|
||||
- **URL label** - Adds a custom display label to override the value of the `Label name` field.
|
||||
- **Label name** - The name of the field in the `labels` object used to obtain the traceID property.
|
||||
- **Remove exemplar link** - Click the **X** to remove existing links.
|
||||
|
||||
You can add multiple exemplars.
|
||||
|
||||
- **Private data source connect** - _Only for Grafana Cloud users._ Private data source connect, or PDC, allows you to establish a private, secured connection between a Grafana Cloud instance, or stack, and data sources secured within a private network. Click the drop-down to locate the URL for PDC. For more information regarding Grafana PDC refer to [Private data source connect (PDC)](ref:private-data-source-connect) and [Configure Grafana private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc) for steps on setting up a PDC connection.
|
||||
|
||||
Click **Manage private data source connect** to be taken to your PDC connection page, where you’ll find your PDC configuration details.
|
||||
|
||||
After you have configured your Prometheus data source options, click **Save & test** at the bottom to test out your data source connection.
|
||||
|
||||
You should see a confirmation dialog box that says:
|
||||
|
||||
**Successfully queried the Prometheus API.**
|
||||
|
||||
**Next, you can start to visualize data by building a dashboard, or by querying data in the Explore view.**
|
||||
|
||||
You can also remove a connection by clicking **Delete**.
|
||||
|
||||
## Provision the Prometheus data source
|
||||
|
||||
You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, and for available configuration options, refer to [Provision Grafana](ref:provision-grafana).
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
After you have provisioned a data source you cannot edit it.
|
||||
{{< /admonition >}}
|
||||
|
||||
**Example of a Prometheus data source configuration:**
|
||||
|
||||
```yaml
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://localhost:9090
|
||||
jsonData:
|
||||
httpMethod: POST
|
||||
manageAlerts: true
|
||||
prometheusType: Prometheus
|
||||
prometheusVersion: 3.3.0
|
||||
cacheLevel: 'High'
|
||||
disableRecordingRules: false
|
||||
incrementalQueryOverlapWindow: 10m
|
||||
exemplarTraceIdDestinations:
|
||||
# Field with internal link pointing to data source in Grafana.
|
||||
# datasourceUid value can be anything, but it should be unique across all defined data source uids.
|
||||
- datasourceUid: my_jaeger_uid
|
||||
name: traceID
|
||||
|
||||
# Field with external link.
|
||||
- name: traceID
|
||||
url: 'http://localhost:3000/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Jaeger%22,%7B%22query%22:%22$${__value.raw}%22%7D%5D'
|
||||
```
|
||||
|
||||
## Azure authentication settings
|
||||
|
||||
The Prometheus data source works with Azure authentication. To configure Azure authentication refer to [Configure Azure Active Directory (AD) authentication](ref:azure-active-directory).
|
||||
|
||||
In Grafana Enterprise, you need to update the .ini configuration file. Refer to [Configuration file location](ref:configure-grafana-configuration-file-location) to locate your .ini file.
|
||||
|
||||
Add the following setting in the **[auth]** section of the .ini configuration file:
|
||||
|
||||
```bash
|
||||
[auth]
|
||||
azure_auth_enabled = true
|
||||
```
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
If you are using Azure authentication, don't enable `Forward OAuth identity`. Both methods use the same HTTP authorization headers, and the OAuth token will override your Azure credentials.
|
||||
{{% /admonition %}}
|
||||
|
||||
## Recording rules (beta)
|
||||
|
||||
You can configure the Prometheus data source to disable recording rules in the data source configuration or provisioning file under `disableRecordingRules` in jsonData.
|
||||
|
||||
## Troubleshooting configuration issues
|
||||
|
||||
Refer to the following troubleshooting information as needed.
|
||||
|
||||
**Data doesn't appear in Metrics Drilldown:**
|
||||
|
||||
If you have successfully tested the connection to a Prometheus data source or are sending metrics to Grafana Cloud and there is no metric data appearing in Explore, make sure you've selected the correct data source from the data source drop-down menu. When using `remote_write` to send metrics to Grafana Cloud, the data source name follows the convention `grafanacloud-stackname-prom`.
|
||||
238
docs/sources/datasources/prometheus/query-editor/_index.md
Normal file
238
docs/sources/datasources/prometheus/query-editor/_index.md
Normal file
@ -0,0 +1,238 @@
|
||||
---
|
||||
aliases:
|
||||
- ../../data-sources/prometheus/query-editor/
|
||||
description: Guide for using the Prometheus data source's query editor
|
||||
keywords:
|
||||
- grafana
|
||||
- prometheus
|
||||
- logs
|
||||
- queries
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Prometheus query editor
|
||||
title: Prometheus query editor
|
||||
weight: 300
|
||||
refs:
|
||||
query-transform-data:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/
|
||||
table:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/table/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/table/
|
||||
exemplars:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/fundamentals/exemplars/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/fundamentals/exemplars/
|
||||
heatmap:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/heatmap/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/heatmap/
|
||||
time-series-transform:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/time-series/#transform
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/time-series/#transform
|
||||
explore:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
---
|
||||
|
||||
# Prometheus query editor
|
||||
|
||||
Grafana provides a query editor for the Prometheus data source to create queries in PromQL. For more information about PromQL, see [Querying Prometheus](http://prometheus.io/docs/querying/basics/). The Prometheus query editor is located on the [Explore page](ref:explore). You can also access the PostgreSQL query editor from a dashboard panel. Click the ellipsis in the upper right of the panel and select **Edit**.
|
||||
|
||||
For general documentation on querying data sources in Grafana, refer to [Query and transform data](ref:query-transform-data). For options and functions common to all query editors, refer to [Query editors](ref:query-transform-data).
|
||||
|
||||
The Prometheus query editor has two modes:
|
||||
|
||||
- [Builder mode](#builder-mode)
|
||||
- [Code mode](#code-mode)
|
||||
|
||||
|
||||
|
||||
Grafana synchronizes both modes, allowing you to switch between them. Grafana also displays a warning message if it detects an issue with the query while switching modes.
|
||||
|
||||
You can configure Prometheus-specific options in the query editor by setting several options regardless of mode.
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/options.png" max-width="500px" class="docs-image--no-shadow" caption="Options" >}}
|
||||
|
||||
## Builder mode
|
||||
|
||||
**Builder mode** helps you build queries using a visual interface. This option is best for users who have limited experience working with Prometheus and PromQL.
|
||||
|
||||
The following video demonstrates how to use the visual Prometheus query builder:
|
||||
|
||||
{{< vimeo 720004179 >}}
|
||||
|
||||
Builder mode contains the following components:
|
||||
|
||||
- **Kick start your query** - Click to view a list of predefined operation patterns that help you quickly build queries with multiple operations. These include:
|
||||
|
||||
- Rate query starters
|
||||
- Histogram query starters
|
||||
- Binary query starters
|
||||
|
||||
Click the arrow next to each to see the available options to add to your query.
|
||||
|
||||
- **Explain** - Toggle on to display a step-by-step explanation of all query components and operations.
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/explain-results.png" max-width="500px" class="docs-image--no-shadow" caption="Explain results" >}}
|
||||
|
||||
- **Builder/Code** - Click the corresponding **Builder** or **Code** tab on the toolbar to select an editor mode.
|
||||
|
||||
If you select Builder mode you will see the following options:
|
||||
|
||||
- **Metric** - Select a metric from the drop-down. Click the icon to open Metrics explorer, where you can search for metrics by name and filter by type if your instance has a large number of metrics. Refer to [Metrics explorer](#metrics-explorer) for more detail on using this feature.
|
||||
- **Label filters** - Select label filters from the drop-down. Select an operator and a value.
|
||||
Select desired labels and their values from the drop-down list.
|
||||
When a metric is selected, the data source requests available labels and their values from the server.
|
||||
Use the `+` button to add a label, and the `x` button to remove a label.
|
||||
|
||||
Click **+ Operations** to select from a list of operations including Aggregations, Range functions, Functions, Binary operations, Trigonometric and Time functions. You can select multiple operations. Refer to [Operations](#operations) for more detail.
|
||||
|
||||
**Options:**
|
||||
|
||||
- **Legend**- Lets you customize the name for the time series. You can use a predefined or custom format.
|
||||
|
||||
- **Auto** - Displays unique labels. Also displays all overlapping labels if a series has multiple labels.
|
||||
- **Verbose** - Displays all label names.
|
||||
- **Custom** - Lets you customize the legend using label templates. For example, `{{hostname}}` is replaced with the value of the `hostname` label. To switch to a different legend mode, clear the input and click outside the field.
|
||||
|
||||
- **Min step** - Sets the minimum interval between data points returned by the query. For example, setting this to `1h` suggests that data is collected or displayed at hourly intervals. This setting supports the `$__interval` and `$__rate_interval` macros. Note that the time range of the query is aligned to this step size, which may adjust the actual start and end times of the returned data.
|
||||
|
||||
- **Format** - Determines how the data from your Prometheus query is interpreted and visualized in a panel. Choose from the following format options:
|
||||
|
||||
- **Time series** - The default format. Refer to [Time series kind formats](https://grafana.com/developers/dataplane/timeseries/) for information on time series data frames and how time and value fields are structured.
|
||||
- **Table** - Displays data in table format. This format works only in a [Table panel](ref:table).
|
||||
- **Heatmap** - Displays Histogram-type metrics in a [Heatmap panel](ref:heatmap) by converting cumulative histograms to regular ones and sorting the series by the bucket bound. Converts cumulative histogram data into regular histogram format and sorts the series by bucket boundaries for proper display.
|
||||
|
||||
- **Type** - This setting determines the query type. These include:
|
||||
- **Both** - The default option. Returns results for both a **Range** query and an **Instant** query.
|
||||
- **Range** - Returns a range vector - a set of time series
|
||||
a set of time series where each series includes multiple data points over a period of time. You can choose to visualize the data as lines, bars, points, stacked lines, or stacked bars.
|
||||
- **Instant** - Returns a single data point per series — the most recent value within the selected time range. The results can be displayed in a table or as raw data. To visualize instant query results in a time series panel, start by adding field override, then add a property to the override called `Transform`, and set the Transform value to `Constant` in the drop-down. For more information, refer to the [Time Series Transform option documentation](ref:time-series-transform).
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
Grafana adjusts the query time range to align with the dynamically calculated step interval. This alignment ensures consistent metric visualization and supports Prometheus's result caching requirements. However, this alignment can cause minor visual differences, such as a slight gap at the graph's right edge or a shifted start time. For example, a `15s` step aligns timestamps to Unix times divisible by 15 seconds. A `1w` `minstep` aligns the range to the start of the week, which for Prometheus is Thursday at 00:00 UTC.
|
||||
{{% /admonition %}}
|
||||
|
||||
- **Exemplars** - Toggle on to run a query that includes exemplars in the graph. Exemplars are unique to Prometheus. For more information see [Introduction to exemplars](ref:exemplars).
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
There is no option to add exemplars with an **Instant** query type.
|
||||
{{% /admonition %}}
|
||||
|
||||
### Filter metrics
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/metrics-and-labels.png" max-width="500px" class="docs-image--no-shadow" caption="Metric and label filters" >}}
|
||||
|
||||
When you are ready to create a query, you can choose the specific metric name from the drop-down list under **Metric**.
|
||||
The data source provides the list of available metrics based on the selected time range.
|
||||
You can also enter text into the selector when the drop-down is open to search and filter the list.
|
||||
|
||||
#### Metrics explorer in Builder mode
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/screenshot-grafana-prometheus-metrics-explorer-2.png" max-width="500px" class="docs-image--no-shadow" caption="Metrics explorer" >}}
|
||||
|
||||
Click the **Open book icon** to open the Metrics explorer, where you can search for and filter all the metrics in your instance.
|
||||
|
||||
If you would like to explore your metrics in the query builder further, you can open the **Metrics explorer** by clicking the first option in the metric select component of the query builder.
|
||||
|
||||
The Metrics explorer displays all metrics in a paginated table list. The list shows the total number of metrics, as well as the name, type, and description for each metric. You can enter text into the search input to filter results.
|
||||
You can also filter by type.
|
||||
|
||||
The following options are included under the **Additional Settings** drop-down:
|
||||
|
||||
- **Include description in search**: Toggle on to search by both name and description.
|
||||
- **Include results with no metadata**: Toggle on to include metrics that lack type or description metadata.
|
||||
- **Disable text wrap**: Toggle on to disable text wrapping.
|
||||
- **Enable regex search**: Toggle on to filter metric names by regex search, which uses an additional call on the Prometheus API.
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
The Metrics explorer (Builder mode) and [Metrics browser (Code mode)](#metrics-browser-in-code-mode) are separate elements. The Metrics explorer does not have the ability to browse labels yet, but the Metrics browser can display all labels on a metric name.
|
||||
{{% /admonition %}}
|
||||
|
||||
### Operations
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/operations.png" max-width="500px" class="docs-image--no-shadow" caption="Operations" >}}
|
||||
|
||||
Select the **+ Operations** button to add operations to your query.
|
||||
|
||||
The query editor groups operations into the following sections:
|
||||
|
||||
- Aggregations - for additional information see [Aggregation operators](https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators).
|
||||
- Range functions - for additional information see [Functions](https://prometheus.io/docs/prometheus/latest/querying/functions/#functions).
|
||||
- Functions - for additional information see [Functions](https://prometheus.io/docs/prometheus/latest/querying/functions/#functions).
|
||||
- Binary operations - for additional information see [Binary operators](https://prometheus.io/docs/prometheus/latest/querying/operators/#binary-operators).
|
||||
- Trigonometric - for additional information see [Trigonometric functions](https://prometheus.io/docs/prometheus/latest/querying/functions/#trigonometric-functions).
|
||||
- Time functions - for additional information see [Functions](https://prometheus.io/docs/prometheus/latest/querying/functions/#functions).
|
||||
|
||||
All operations have function parameters under the operation header. Click the `operator` to see a full list of supported functions. Some operations allow you to apply specific labels to functions.
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/use-function-by-label-9-5.png" max-width="500px" class="docs-image--no-shadow" caption="Functions and labels" >}}
|
||||
|
||||
Some operations are only valid when used in a specific order. If you add an operation in a way that would create an invalid or illogical query, the query editor automatically places it in the correct position to maintain a valid query structure.
|
||||
|
||||
### Hints
|
||||
|
||||
The query editor can detect which operations are most appropriate for certain selected metrics.
|
||||
When it does, it displays a hint next to the **+ Operations** button.
|
||||
|
||||
To add the operation to your query, click the **Hint**.
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/hint-example.png" max-width="500px" class="docs-image--no-shadow" caption="Hint" >}}
|
||||
|
||||
When you are satisfied with your query, click **Run query**.
|
||||
|
||||
## Code mode
|
||||
|
||||
**Code mode** is for the experienced Prometheus user with prior expertise in PromQL, Prometheus' query language. The Code mode editor allows you to create queries just as you would in Prometheus. For more information about PromQL see [Querying Prometheus](http://prometheus.io/docs/querying/basics/).
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/code-mode.png" max-width="500px" class="docs-image--no-shadow" caption="Code mode" >}}
|
||||
|
||||
The user interface (UI) also lets you select metrics, labels, filters, and operations.
|
||||
|
||||
You can write complex queries using the text editor with autocompletion features and syntax highlighting. Code mode's autocomplete feature works automatically while typing. The query editor can autocomplete static functions, aggregations, keywords, and also dynamic items like metrics and labels. The autocompletion drop-down includes documentation for the suggested items where available.
|
||||
|
||||
It also contains a [Metrics browser](#metrics-browser-in-code-mode) to further help you write queries. To open the Metrics browser, click the arrow next to **Metrics browser**.
|
||||
|
||||
### Metrics browser in Code mode
|
||||
|
||||
The Metrics browser locates metrics and selects relevant labels to help you build basic queries.
|
||||
When you click **Metrics browser** in `Code` mode, it displays all available metrics and labels.
|
||||
If supported by your Prometheus instance, each metric also displays its `HELP` and `TYPE` as a tooltip.
|
||||
|
||||
{{< figure alt="Prometheus query editor metrics browser" src="/media/docs/prometheus/Metrics-browser-V10-prom-query-editor.png" caption="Metrics browser" >}}
|
||||
|
||||
When you select a metric under **Step 1**, the browser narrows down the available labels to show only the ones applicable to the metric.
|
||||
You can then select one or more labels shown in **Step 2**.
|
||||
Select one or more values in **Step 3** for each label to tighten your query scope.
|
||||
In **Step 4**, you can select **Use query** to run the query, **Use as rate query** to add the rate operation to your query (`$__rate_interval`), **Validate selector** to verify the selector is valid and show the number of series found, or **Clear** to clear your selections and start over.
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
If you don't remember the exact metric name, you can start by selecting a few labels to filter the list. This helps you find relevant label values and narrow down your options.
|
||||
{{% /admonition %}}
|
||||
|
||||
All lists in the Metrics browser include a search field to quickly filter metrics or labels by keyword.
|
||||
In the **Values** section, there's a single search field that filters across all selected labels, making it easier to find matching values. For example, if you have labels like `app`, `job`, and `job_name`, only one of them might contain the value you're looking for.
|
||||
|
||||
When you are satisfied with your query, click **Run query**.
|
||||
|
||||
## Incremental dashboard queries (beta)
|
||||
|
||||
Starting with Grafana v10, the Prometheus data source supports incremental querying for live dashboards. Instead of re-querying the entire time range on each refresh, Grafana can fetch only new data since the last query.
|
||||
|
||||
You can enable or disable this feature in the data source configuration or provisioning file using the `incrementalQuerying` field in `jsonData`.
|
||||
|
||||
You can also control the overlap between consecutive incremental queries using the `incrementalQueryOverlapWindow` field in `jsonData`. By default, this is set to `10m` (10 minutes). Increasing the `incrementalQueryOverlapWindow` value increases the time range covered by each incremental query. This can help in environments where the most recent data may be delayed or incomplete.
|
||||
@ -1,255 +0,0 @@
|
||||
---
|
||||
aliases:
|
||||
- ../../data-sources/prometheus/query-editor/
|
||||
description: Guide for using the Prometheus data source's query editor
|
||||
keywords:
|
||||
- grafana
|
||||
- prometheus
|
||||
- logs
|
||||
- queries
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Query editor
|
||||
title: Prometheus query editor
|
||||
weight: 300
|
||||
refs:
|
||||
query-transform-data:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/
|
||||
table:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/table/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/table/
|
||||
exemplars:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/fundamentals/exemplars/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/fundamentals/exemplars/
|
||||
heatmap:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/heatmap/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/heatmap/
|
||||
time-series-transform:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/time-series/#transform
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/time-series/#transform
|
||||
---
|
||||
|
||||
# Prometheus query editor
|
||||
|
||||
Grafana provides a query editor for the Prometheus data source to create queries in PromQL. For more information about PromQL, see [Querying Prometheus](http://prometheus.io/docs/querying/basics/).
|
||||
|
||||
For general documentation on querying data sources in Grafana, see [Query and transform data](ref:query-transform-data).
|
||||
|
||||
For options and functions common to all query editors, see [Query editors](ref:query-transform-data).
|
||||
|
||||
## Choose a query editing mode
|
||||
|
||||
The Prometheus query editor has two modes:
|
||||
|
||||
- [Builder mode](#builder-mode)
|
||||
- [Code mode](#code-mode)
|
||||
|
||||
Each mode is explained in greater detail below.
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/editing-mode.png" max-width="500px" class="docs-image--no-shadow" caption="Query editor mode" >}}
|
||||
|
||||
Both modes are synchronized, so you can switch between them. However, if there is an issue with the query while switching modes, a warning message will appear.
|
||||
|
||||
## Toolbar elements
|
||||
|
||||
The query editor toolbar contains the following elements:
|
||||
|
||||
- **Kick start your query** - Click to see a list of operation patterns that help you quickly get started adding multiple operations to your query. These include:
|
||||
|
||||
- Rate query starters
|
||||
- Histogram query starters
|
||||
- Binary query starters
|
||||
|
||||
Click the arrow next to each to see available options to add to your query.
|
||||
|
||||
- **Explain** - Toggle to display a step-by-step explanation of all query components and operations.
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/explain-results.png" max-width="500px" class="docs-image--no-shadow" caption="Explain results" >}}
|
||||
|
||||
- **Builder/Code** - Click the corresponding **Builder** or **Code** tab on the toolbar to select a editor mode.
|
||||
|
||||
## Configure common options
|
||||
|
||||
You can configure Prometheus-specific options in the query editor by setting several options regardless of mode.
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/options.png" max-width="500px" class="docs-image--no-shadow" caption="Options" >}}
|
||||
|
||||
### Legend
|
||||
|
||||
The **Legend** setting defines the time series's name. You can use a predefined or custom format.
|
||||
|
||||
- **Auto** - Displays unique labels. Also displays all overlapping labels if a series has multiple labels.
|
||||
- **Verbose** - Displays all label names.
|
||||
- **Custom** - Uses templating to select which labels will be included. For example, `{{hostname}}` is replaced by the label value for the label `hostname`. Clear the input and click outside of it to select another mode.
|
||||
|
||||
### Min step
|
||||
|
||||
The **Min step** setting defines the lower bounds on the interval between data points.
|
||||
For example, set this to `1h` to hint that measurements are taken hourly.
|
||||
This setting supports the `$__interval` and `$__rate_interval` macros. Be aware that the query range dates are aligned to the step and this can change the start and end of the range.
|
||||
|
||||
### Format
|
||||
|
||||
Switch between the following format options:
|
||||
|
||||
- **Time series** - The default time series format. See [Time series kind formats](https://grafana.com/developers/dataplane/timeseries/) for information on time series data frames and how time and value fields are structured.
|
||||
- **Table** - This works only in a [Table panel](ref:table).
|
||||
- **Heatmap** - Displays metrics of the Histogram type on a [Heatmap panel](ref:heatmap) by converting cumulative histograms to regular ones and sorting the series by the bucket bound.
|
||||
|
||||
### Type
|
||||
|
||||
The **Type** setting sets the query type. These include:
|
||||
|
||||
- **Both** - The default option. Returns results for both a **Range** query and an **Instant** query.
|
||||
- **Range** - Returns a range vector consisting of a set of time series data containing a range of data points over time for each time series. You can choose lines, bars, points, stacked lines or stacked bars
|
||||
- **Instant** - Returns one data point per query and only the most recent point in the time range provided. The results can be shown in table format or as raw data. To depict instant query results in the time series panel, first add a field override, next add a property to the override named `Transform`, and finally select `Constant` from the **Transform** dropdown.
|
||||
|
||||
For more information, refer to the [Time Series Transform option documentation](ref:time-series-transform).
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
Grafana modifies the request dates for queries to align them with the dynamically calculated step.
|
||||
This ensures a consistent display of metrics data and Prometheus requires this for caching results. But, aligning the range with the step can result in a small gap of data at the right edge of a graph or change the start date of the range. For example, a 15s step aligns the range to Unix time divisible by 15s and a 1w minstep aligns the range to the start of the week on a Thursday.
|
||||
{{% /admonition %}}
|
||||
|
||||
### Exemplars
|
||||
|
||||
Toggle **Exemplars** to run a query that includes exemplars in the graph. Exemplars are unique to Prometheus. For more information see [Introduction to exemplars](ref:exemplars).
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
There is no option to add exemplars with an **Instant** query type.
|
||||
{{% /admonition %}}
|
||||
|
||||
### Inspector
|
||||
|
||||
Click **Inspector** to get detailed statistics regarding your query. Inspector functions as a kind of debugging tool that "inspects" your query. It provides query statistics under **Stats**, request response time under **Query**, data frame details under **{} JSON**, and the shape of your data under **Data**.
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/insepctor-9-5.png" max-width="500px" class="docs-image--no-shadow" caption="Inspector" >}}
|
||||
|
||||
## Builder mode
|
||||
|
||||
**Builder mode** helps you build queries using a visual interface. This option is best for users who have limited or no previous experience working with Prometheus and PromQL.
|
||||
|
||||
This video demonstrates how to use the visual Prometheus query builder:
|
||||
|
||||
{{< vimeo 720004179 >}}
|
||||
|
||||
</br>
|
||||
|
||||
### Metrics
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/metrics-and-labels.png" max-width="500px" class="docs-image--no-shadow" caption="Metric and label filters" >}}
|
||||
|
||||
When you are ready to create a query, you can choose the specific metric name from the dropdown list under **Metric**.
|
||||
The data source requests the list of available metrics from the Prometheus server based on the selected time rage.
|
||||
You can also enter text into the selector when the dropdown is open to search and filter the list.
|
||||
|
||||
#### Metrics explorer
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/screenshot-grafana-prometheus-metrics-explorer-2.png" max-width="500px" class="docs-image--no-shadow" caption="Metrics explorer" >}}
|
||||
|
||||
If you would like to explore your metrics in the query builder further, you can open the **Metrics Explorer** by clicking the first option in the metric select component of the query builder.
|
||||
|
||||
The metrics explorer is different than the metrics browser. The metrics explorer is only found in the query builder section. The metrics browser is only found in the code editor. The metrics explorer does not have the ability to browse labels yet, but the metrics browser can display all labels on a metric name.
|
||||
|
||||
The metrics explorer displays all metrics in a paginated table list. The list shows the total number of metrics, as well as the name, type and description for each metric. You can enter text into the search input to filter results.
|
||||
You can also filter by type.
|
||||
|
||||
There are also additional settings for the following items:
|
||||
|
||||
- Include description in search. Search by name **and** description
|
||||
- Include results with no metadata. Many Prometheus metrics have no metadata. This allows users to include metrics with undefined type and description.
|
||||
- Disable text wrap.
|
||||
- Enable regex search. This uses the Prometheus API to enable regex search for the metric name.
|
||||
|
||||
### Label filters
|
||||
|
||||
Select desired labels and their values from the dropdown list.
|
||||
When a metric is selected, the data source requests available labels and their values from the server.
|
||||
Use the `+` button to add a label, and the `x` button to remove a label.
|
||||
|
||||
### Operations
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/operations.png" max-width="500px" class="docs-image--no-shadow" caption="Operations" >}}
|
||||
|
||||
Select the **+ Operations** button to add operations to your query.
|
||||
|
||||
The query editor groups operations into the following sections:
|
||||
|
||||
- Aggregations - for additional information see [Aggregation operators](https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators).
|
||||
- Range functions - for additional information see [Functions](https://prometheus.io/docs/prometheus/latest/querying/functions/#functions).
|
||||
- Functions - for additional information see [Functions](https://prometheus.io/docs/prometheus/latest/querying/functions/#functions).
|
||||
- Binary operations - for additional information see [Binary operators](https://prometheus.io/docs/prometheus/latest/querying/operators/#binary-operators).
|
||||
- Trigonometric - for additional information see [Trigonometric functions](https://prometheus.io/docs/prometheus/latest/querying/functions/#trigonometric-functions).
|
||||
- Time functions - for additional information see [Functions](https://prometheus.io/docs/prometheus/latest/querying/functions/#functions).
|
||||
|
||||
All operations have function parameters under the operation header. Click the `operator` to see a full list of supported functions. Some operations allow you to apply specific labels to functions.
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/use-function-by-label-9-5.png" max-width="500px" class="docs-image--no-shadow" caption="Functions and labels" >}}
|
||||
|
||||
Some operations make sense only when used in a specific order.
|
||||
If adding an operation would result in nonsensical query, the query editor adds the operation to the correct place.
|
||||
|
||||
#### Hints
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/hint-example.png" max-width="500px" class="docs-image--no-shadow" caption="Hint" >}}
|
||||
|
||||
The query editor can detect which operations are most appropriate for some selected metrics.
|
||||
If it does, it displays a hint next to the **+ Operations** button.
|
||||
|
||||
To add the operation to your query, click the **Hint**.
|
||||
|
||||
Once you are satisfied with your query, click **Run query**.
|
||||
|
||||
## Code mode
|
||||
|
||||
**Code mode** is for the experienced Prometheus user with prior expertise in PromQL, Prometheus' query language. The Code mode editor allows you to create queries just as you would in Prometheus. For more information about PromQL see [Querying Prometheus](http://prometheus.io/docs/querying/basics/).
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/code-mode.png" max-width="500px" class="docs-image--no-shadow" caption="Code mode" >}}
|
||||
|
||||
The user interface (UI) also lets you select metrics, labels, filters and operations.
|
||||
|
||||
You can write complex queries using the text editor with autocompletion features and syntax highlighting.
|
||||
It also contains a [Metrics browser](#metrics-browser) to further help you write queries.
|
||||
|
||||
### Use autocomplete
|
||||
|
||||
Code mode's autocomplete feature works automatically while typing. The query editor can autocomplete static functions, aggregations, keywords, and also dynamic items like metrics and labels.
|
||||
The autocompletion dropdown includes documentation for the suggested items where available.
|
||||
|
||||
### Metrics browser
|
||||
|
||||
The metrics browser locates metrics and selects relevant labels to help you build basic queries.
|
||||
When you click **Metrics browser** in `Code` mode, it displays all available metrics and labels.
|
||||
If supported by your Prometheus instance, each metric also displays its `HELP` and `TYPE` as a tooltip.
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/metric-browser.png" max-width="500px" class="docs-image--no-shadow" caption="Metrics browser" >}}
|
||||
|
||||
When you select a metric under Step 1, the browser narrows down the available labels to show only the ones applicable to the metric.
|
||||
You can then select one or more labels shown in Step 2.
|
||||
Select one or more values in Step 3 for each label to tighten your query scope.
|
||||
In Step 4, you can select **Use query** to run the query, **Use as rate query** to add the rate operation to your query (`$__rate_interval`), **Validate selector** to verify the selector is valid and show the number of series found, or **Clear** to clear your selections and start over.
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
If you do not remember a metric name, you can also select a few labels to narrow down the list, then find relevant label values.
|
||||
{{% /admonition %}}
|
||||
|
||||
All lists in the metrics browser have a search field above them to quickly filter for metrics or labels that match a certain string.
|
||||
The values section has only one search field, and its filtering applies to all labels to help you find values across labels once selected.
|
||||
|
||||
For example, among your labels `app`, `job`, `job_name` only one might have the value you are looking for.
|
||||
|
||||
Once you are satisfied with your query, click **Run query**.
|
||||
@ -13,7 +13,7 @@ labels:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Template variables
|
||||
menuTitle: Prometheus template variables
|
||||
title: Prometheus template variables
|
||||
weight: 400
|
||||
refs:
|
||||
@ -42,13 +42,13 @@ refs:
|
||||
# Prometheus template variables
|
||||
|
||||
Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana refers to such variables as **template** variables.
|
||||
Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard.
|
||||
Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the displayed data.
|
||||
|
||||
For an introduction to templating and template variables, see [Templating](ref:variables) and [Add and manage variables](ref:add-template-variables).
|
||||
For an introduction to templating and template variables, refer to [Templating](ref:variables) and [Add and manage variables](ref:add-template-variables).
|
||||
|
||||
## Use query variables
|
||||
|
||||
You have the option to use several different variable types, but variables of the type `Query` will query Prometheus for a list of metrics, labels, label values, a query result or a series.
|
||||
Grafana supports several types of variables, but Query variables are specifically used to query Prometheus. They can return a list of metrics, labels, label values, query results, or series.
|
||||
|
||||
Select a Prometheus data source query type and enter the required inputs:
|
||||
|
||||
@ -61,18 +61,18 @@ Select a Prometheus data source query type and enter the required inputs:
|
||||
| `Series query` | `metric`, `label` or both | Returns a list of time series associated with the entered data. | /api/v1/series |
|
||||
| `Classic query` | classic query string | Deprecated, classic version of variable query editor. Enter a string with the query type using a syntax like the following: `label_values(<metric>, <label>)` | all |
|
||||
|
||||
For details on _metric names_, _label names_, and _label values_, refer to the [Prometheus documentation](http://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).
|
||||
For details on `metric names`, `label names`, and `label values`, refer to the [Prometheus documentation](http://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).
|
||||
|
||||
### Query options
|
||||
|
||||
Under the query variable type, you can set the following query options:
|
||||
With the query variable type, you can set the following query options:
|
||||
|
||||
| Option | Description |
|
||||
| --------------------- | ------------------------------------------------------------------------------------------------------- |
|
||||
| **Data source** | Select your data source from the dropdown list. |
|
||||
| **Data source** | Select your data source from the drop-down list. |
|
||||
| **Select query type** | Options are `default`, `value` and `metric name`. Each query type hits a different Prometheus endpoint. |
|
||||
| **Regex** | Optional, if you want to extract part of a series name or metric node segment. |
|
||||
| **Sort** | Default is `disabled`. Options include `alphabetical`, `numerical` and `alphabetical case-sensitive`. |
|
||||
| **Sort** | Default is `disabled`. Options include `alphabetical`, `numerical`, and `alphabetical case-sensitive`. |
|
||||
| **Refresh** | When to update the values for the variable. Options are `On dashboard load` and `On time range change`. |
|
||||
|
||||
### Selection options
|
||||
@ -85,34 +85,41 @@ The following selection options are available:
|
||||
|
||||
### Use interval and range variables
|
||||
|
||||
You can use some global built-in variables in query variables, for example, `$__interval`, `$__interval_ms`, `$__range`, `$__range_s` and `$__range_ms`.
|
||||
For details, see [Global built-in variables](ref:add-template-variables-global-variables).
|
||||
You can use global built-in variables in query variables, including the following:
|
||||
|
||||
- `$__interval`
|
||||
- `$__interval_ms`
|
||||
- `$__range`
|
||||
- `$__range_s`
|
||||
- `$__range_ms`
|
||||
|
||||
For details, refer to [Global built-in variables](ref:add-template-variables-global-variables).
|
||||
The `label_values` function doesn't support queries, so you can use these variables in conjunction with the `query_result` function to filter variable queries.
|
||||
|
||||
Make sure to set the variable's `refresh` trigger to be `On Time Range Change` to get the correct instances when changing the time range on the dashboard.
|
||||
Configure the variable’s `refresh` setting to `On Time Range Change` to ensure it dynamically queries and displays the correct instances when the dashboard time range is modified.
|
||||
|
||||
**Example:**
|
||||
|
||||
Populate a variable with the busiest 5 request instances based on average QPS over the time range shown in the dashboard:
|
||||
Populate a variable with the top 5 busiest request instances ranked by average QPS over the dashboard's selected time range:
|
||||
|
||||
```
|
||||
Query: query_result(topk(5, sum(rate(http_requests_total[$__range])) by (instance)))
|
||||
query_result(topk(5, sum(rate(http_requests_total[$__range])) by (instance)))
|
||||
Regex: /"([^"]+)"/
|
||||
```
|
||||
|
||||
Populate a variable with the instances having a certain state over the time range shown in the dashboard, using `$__range_s`:
|
||||
|
||||
```
|
||||
Query: query_result(max_over_time(<metric>[${__range_s}s]) != <state>)
|
||||
query_result(max_over_time(<metric>[${__range_s}s]) != <state>)
|
||||
Regex:
|
||||
```
|
||||
|
||||
## Use `$__rate_interval`
|
||||
|
||||
We recommend using `$__rate_interval` in the `rate` and `increase` functions instead of `$__interval` or a fixed interval value.
|
||||
Because `$__rate_interval` is always at least four times the value of the Scrape interval, it avoid problems specific to Prometheus.
|
||||
Grafana recommends using `$__rate_interval` with the `rate` and `increase` functions instead of `$__interval` or a fixed interval value.
|
||||
Since `$__rate_interval` is always at least four times the scrape interval, it helps avoid issues specific to Prometheus, such as gaps or inaccuracies in query results.
|
||||
|
||||
For example, instead of using:
|
||||
For example, instead of using the following:
|
||||
|
||||
```
|
||||
rate(http_requests_total[5m])
|
||||
@ -124,20 +131,28 @@ or:
|
||||
rate(http_requests_total[$__interval])
|
||||
```
|
||||
|
||||
We recommend that you use:
|
||||
Use the following:
|
||||
|
||||
```
|
||||
rate(http_requests_total[$__rate_interval])
|
||||
```
|
||||
|
||||
The value of `$__rate_interval` is defined as
|
||||
<!-- The value of `$__rate_interval` is defined as
|
||||
*max(`$__interval` + *Scrape interval*, 4 \* *Scrape interval*)*,
|
||||
where _Scrape interval_ is the "Min step" setting (also known as `query*interval`, a setting per PromQL query) if any is set.
|
||||
Otherwise, Grafana uses the Prometheus data source's "Scrape interval" setting.
|
||||
Otherwise, Grafana uses the Prometheus data source's `scrape interval` setting. -->
|
||||
|
||||
The "Min interval" setting in the panel is modified by the resolution setting, and therefore doesn't have any effect on _Scrape interval_.
|
||||
The value of `$__rate_interval` is calculated as:
|
||||
|
||||
For details, refer to the [Grafana blog](/blog/2020/09/28/new-in-grafana-7.2-__rate_interval-for-prometheus-rate-queries-that-just-work/).
|
||||
```
|
||||
max($__interval + scrape_interval, 4 * scrape_interval)
|
||||
```
|
||||
|
||||
Here, `scrape_interval` refers to the `min step` setting (also known as `query_interval`) specified per PromQL query, if set. If not, Grafana falls back to the Prometheus data source’s scrape interval setting.
|
||||
|
||||
The `min interval` setting in the panel is modified by the resolution setting, and therefore doesn't have any effect on `scrape interval`.
|
||||
|
||||
For details, refer to the Grafana blog [$\_\_rate_interval for Prometheus rate queries that just work](https://grafana.com/blog/2020/09/28/new-in-grafana-7.2-__rate_interval-for-prometheus-rate-queries-that-just-work/).
|
||||
|
||||
## Choose a variable syntax
|
||||
|
||||
@ -146,9 +161,8 @@ The Prometheus data source supports two variable syntaxes for use in the **Query
|
||||
- `$<varname>`, for example `rate(http_requests_total{job=~"$job"}[$_rate_interval])`, which is easier to read and write but does not allow you to use a variable in the middle of a word.
|
||||
- `[[varname]]`, for example `rate(http_requests_total{job=~"[[job]]"}[$_rate_interval])`
|
||||
|
||||
If you've enabled the _Multi-value_ or _Include all value_ options, Grafana converts the labels from plain text to a regex-compatible string, which requires you to use `=~` instead of `=`.
|
||||
If you've enabled the `Multi-value` or `Include all value` options, Grafana converts the labels from plain text to a regex-compatible string, which requires you to use `=~` instead of `=`.
|
||||
|
||||
## Use the ad hoc filters variable type
|
||||
|
||||
Prometheus supports the special [ad hoc filters](ref:add-template-variables-add-ad-hoc-filters) variable type, which you can use to specify any number of label/value filters on the fly.
|
||||
These filters are automatically applied to all your Prometheus queries.
|
||||
Prometheus supports the special [ad hoc filters](ref:add-template-variables-add-ad-hoc-filters) variable type, which allows you to dynamically apply label/value filters across your dashboards. These filters are automatically added to all Prometheus queries, allowing dynamic filtering without modifying individual queries.
|
||||
Reference in New Issue
Block a user