mirror of
https://github.com/grafana/grafana.git
synced 2025-09-18 12:22:44 +08:00
Docs: Update the Elasticsearch data source docs (#73415)
* added new config doc * edits to config doc * updates to config and index docs * minor edits to config page * updates to query editor * query editor update, added data source keyword * index file updates * updates to quey editor, index * edits to index, configure and query editor docs * final updates to config and index docs * Update docs/sources/datasources/elasticsearch/configure-elasticsearch-data-source.md Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com> * Update docs/sources/datasources/elasticsearch/query-editor/index.md Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com> * Update docs/sources/datasources/elasticsearch/query-editor/index.md Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com> * Update docs/sources/datasources/elasticsearch/configure-elasticsearch-data-source.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/datasources/elasticsearch/configure-elasticsearch-data-source.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/datasources/elasticsearch/configure-elasticsearch-data-source.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/datasources/elasticsearch/configure-elasticsearch-data-source.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/datasources/elasticsearch/configure-elasticsearch-data-source.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * Update docs/sources/datasources/elasticsearch/configure-elasticsearch-data-source.md Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com> * added date histogram options * suggestions from Chris * added bucket aggregations options * bucket aggregation * updates based on feedback * removed panels info under logs query * inspector now query inspector --------- Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com> Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com>
This commit is contained in:
lwandz13
@ -7,6 +7,7 @@ keywords:
|
||||
- grafana
|
||||
- elasticsearch
|
||||
- guide
|
||||
- data source
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
@ -19,17 +20,15 @@ weight: 325
|
||||
|
||||
# Elasticsearch data source
|
||||
|
||||
Grafana ships with built-in support for Elasticsearch.
|
||||
You can make many types of queries to visualize logs or metrics stored in Elasticsearch, and annotate graphs with log events stored in Elasticsearch.
|
||||
Elasticsearch is a search and analytics engine used for a variety of use cases.
|
||||
You can create many types of queries to visualize logs or metrics stored in Elasticsearch, and annotate graphs with log events stored in Elasticsearch.
|
||||
|
||||
This topic explains configuring and querying specific to the Elasticsearch data source.
|
||||
For general documentation on querying data sources in Grafana, see [Query and transform data][query-transform-data].
|
||||
The following will help you get started working with Elasticsearch and Grafana:
|
||||
|
||||
For instructions on how to add a data source to Grafana, refer to the [administration documentation][data-source-management].
|
||||
Only users with the organization administrator role can add data sources.
|
||||
Administrators can also [configure the data source via YAML](#provision-the-data-source) with Grafana's provisioning system.
|
||||
|
||||
Once you've added the Elasticsearch data source, you can [configure it](#configure-the-data-source) so that your Grafana instance's users can create queries in its [query editor]({{< relref "./query-editor" >}}) when they [build dashboards][build-dashboards] and use [Explore][explore].
|
||||
- [What is Elasticsearch?](https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-intro.html)
|
||||
- [Configure the Elasticsearch data source](/docs/grafana/latest/datasources/elasticsearch/configure-elasticsearch-data-source/)
|
||||
- [Elasticsearch query editor]({{< relref "./query-editor/" >}})
|
||||
- [Elasticsearch template variables]({{< relref "./template-variables/" >}})
|
||||
|
||||
## Supported Elasticsearch versions
|
||||
|
||||
@ -40,113 +39,7 @@ This data source supports these versions of Elasticsearch:
|
||||
|
||||
Our maintenance policy for Elasticsearch data source is aligned with the [Elastic Product End of Life Dates](https://www.elastic.co/support/eol) and we ensure proper functionality for supported versions. If you are using an Elasticsearch with version that is past its end-of-life (EOL), you can still execute queries, but you will receive a notification in the query builder indicating that the version of Elasticsearch you are using is no longer supported. It's important to note that in such cases, we do not guarantee the correctness of the functionality, and we will not be addressing any related issues.
|
||||
|
||||
## Configure the data source
|
||||
|
||||
To configure basic settings for the data source, complete the following steps:
|
||||
|
||||
1. Click **Connections** in the left-side menu.
|
||||
1. Under Your connections, click **Data sources**.
|
||||
1. Enter `Elasticsearch` in the search bar.
|
||||
1. Click **Elasticsearch**.
|
||||
|
||||
The **Settings** tab of the data source is displayed.
|
||||
|
||||
1. Set the data source's basic configuration options:
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | -------------------------------------------------------------------------- |
|
||||
| **Name** | Sets the name you use to refer to the data source in panels and queries. |
|
||||
| **Default** | Sets the data source that's pre-selected for new panels. |
|
||||
| **Url** | Sets the HTTP protocol, IP, and port of your Elasticsearch server. |
|
||||
| **Access** | Don't modify Access. Use `Server (default)` or the data source won't work. |
|
||||
|
||||
You must also configure settings specific to the Elasticsearch data source. These options are described in the sections below.
|
||||
|
||||
### Index settings
|
||||
|
||||
{{< figure src="/static/img/docs/elasticsearch/elasticsearch-ds-details-7-4.png" max-width="500px" class="docs-image--right" caption="Elasticsearch data source details" >}}
|
||||
|
||||
Use the index settings to specify a default for the `time field` and your Elasticsearch index's name.
|
||||
You can use a time pattern, such as `YYYY.MM.DD`, or a wildcard for the index name.
|
||||
|
||||
### Configure Min time interval
|
||||
|
||||
The **Min time interval** setting defines a lower limit for the auto group-by time interval.
|
||||
|
||||
This value _must_ be formatted as a number followed by a valid time identifier:
|
||||
|
||||
| Identifier | Description |
|
||||
| ---------- | ----------- |
|
||||
| `y` | year |
|
||||
| `M` | month |
|
||||
| `w` | week |
|
||||
| `d` | day |
|
||||
| `h` | hour |
|
||||
| `m` | minute |
|
||||
| `s` | second |
|
||||
| `ms` | millisecond |
|
||||
|
||||
We recommend setting this value to match your Elasticsearch write frequency.
|
||||
For example, set this to `1m` if Elasticsearch writes data every minute.
|
||||
|
||||
You can also override this setting in a dashboard panel under its data source options.
|
||||
|
||||
### X-Pack enabled
|
||||
|
||||
Toggle this to enable `X-Pack`-specific features and options, which provide the [query editor]({{< relref "./query-editor" >}}) with additional aggregations, such as `Rate` and `Top Metrics`.
|
||||
|
||||
#### Include frozen indices
|
||||
|
||||
When the "X-Pack enabled" setting is active and the configured Elasticsearch version is higher than `6.6.0`, you can configure Grafana to not ignore [frozen indices](https://www.elastic.co/guide/en/elasticsearch/reference/7.13/frozen-indices.html) when performing search requests.
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
Frozen indices are [deprecated in Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/frozen-indices.html) since v7.14.
|
||||
{{% /admonition %}}
|
||||
|
||||
### Logs
|
||||
|
||||
You can optionally configure the two Logs parameters **Message field name** and **Level field name** to determine which fields the data source uses for log messages and log levels when visualizing logs in [Explore][explore].
|
||||
|
||||
For example, if you're using a default setup of Filebeat for shipping logs to Elasticsearch, set:
|
||||
|
||||
- **Message field name:** `message`
|
||||
- **Level field name:** `fields.level`
|
||||
|
||||
### Data links
|
||||
|
||||
Data links create a link from a specified field that can be accessed in Explore's logs view.
|
||||
|
||||
Each data link configuration consists of:
|
||||
|
||||
| Parameter | Description |
|
||||
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Field** | Sets the name of the field used by the data link. |
|
||||
| **URL/query** | Sets the full link URL if the link is external. If the link is internal, this input serves as a query for the target data source.<br/>In both cases, you can interpolate the value from the field with the `${__value.raw }` macro. |
|
||||
| **URL Label** | (Optional) Sets a custom display label for the link. The link label defaults to the full external URL or name of the linked internal data source and is overridden by this setting. |
|
||||
| **Internal link** | Sets whether the link is internal or external. For an internal link, you can select the target data source with a data source selector. This supports only tracing data sources. |
|
||||
|
||||
### Configure Amazon Elasticsearch Service
|
||||
|
||||
If you use Amazon Elasticsearch Service, you can use Grafana's Elasticsearch data source to visualize data from it.
|
||||
|
||||
If you use an AWS Identity and Access Management (IAM) policy to control access to your Amazon Elasticsearch Service domain, you must use AWS Signature Version 4 (AWS SigV4) to sign all requests to that domain.
|
||||
|
||||
For details on AWS SigV4, refer to the [AWS documentation](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html).
|
||||
|
||||
#### AWS Signature Version 4 authentication
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
Available in Grafana v7.3 and higher.
|
||||
{{% /admonition %}}
|
||||
|
||||
To sign requests to your Amazon Elasticsearch Service domain, you can enable SigV4 in Grafana's [configuration][configure-grafana-sigv4-auth-enabled].
|
||||
|
||||
Once AWS SigV4 is enabled, you can configure it on the Elasticsearch data source configuration page.
|
||||
For more information about AWS authentication options, refer to [AWS authentication]({{< relref "../aws-cloudwatch/aws-authentication" >}}).
|
||||
|
||||
{{< figure src="/static/img/docs/v73/elasticsearch-sigv4-config-editor.png" max-width="500px" class="docs-image--no-shadow" caption="SigV4 configuration for AWS Elasticsearch Service" >}}
|
||||
|
||||
### Provision the data source
|
||||
## Provision the data source
|
||||
|
||||
You can define and configure the data source in YAML files as part of Grafana's provisioning system.
|
||||
For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana][provisioning-data-sources].
|
||||
@ -157,9 +50,9 @@ You should now use the `index` field in `jsonData` to store the index name.
|
||||
Please see the examples below.
|
||||
{{% /admonition %}}
|
||||
|
||||
#### Provisioning examples
|
||||
### Provisioning examples
|
||||
|
||||
**Basic provisioning:**
|
||||
**Basic provisioning**
|
||||
|
||||
```yaml
|
||||
apiVersion: 1
|
||||
@ -175,7 +68,7 @@ datasources:
|
||||
timeField: '@timestamp'
|
||||
```
|
||||
|
||||
**Provision for logs:**
|
||||
**Provision for logs**
|
||||
|
||||
```yaml
|
||||
apiVersion: 1
|
||||
@ -197,6 +90,27 @@ datasources:
|
||||
url: '$${__value.raw}' # Careful about the double "$$" because of env var expansion
|
||||
```
|
||||
|
||||
## Configure Amazon Elasticsearch Service
|
||||
|
||||
If you use Amazon Elasticsearch Service, you can use Grafana's Elasticsearch data source to visualize data from it.
|
||||
|
||||
If you use an AWS Identity and Access Management (IAM) policy to control access to your Amazon Elasticsearch Service domain, you must use AWS Signature Version 4 (AWS SigV4) to sign all requests to that domain.
|
||||
|
||||
For details on AWS SigV4, refer to the [AWS documentation](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html).
|
||||
|
||||
### AWS Signature Version 4 authentication
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
Available in Grafana v7.3 and higher.
|
||||
{{% /admonition %}}
|
||||
|
||||
To sign requests to your Amazon Elasticsearch Service domain, you can enable SigV4 in Grafana's [configuration]({{< relref "../../setup-grafana/configure-grafana/#sigv4_auth_enabled" >}}).
|
||||
|
||||
Once AWS SigV4 is enabled, you can configure it on the Elasticsearch data source configuration page.
|
||||
For more information about AWS authentication options, refer to [AWS authentication]({{< relref "../aws-cloudwatch/aws-authentication/" >}}).
|
||||
|
||||
{{< figure src="/static/img/docs/v73/elasticsearch-sigv4-config-editor.png" max-width="500px" class="docs-image--no-shadow" caption="SigV4 configuration for AWS Elasticsearch Service" >}}
|
||||
|
||||
## Query the data source
|
||||
|
||||
You can select multiple metrics and group by multiple terms or filters when using the Elasticsearch query editor.
|
||||
|
||||
@ -0,0 +1,153 @@
|
||||
---
|
||||
aliases:
|
||||
- ../data-sources/elasticsearch/
|
||||
- ../features/datasources/elasticsearch/
|
||||
description: Guide for configuring the Elasticsearch data source in Grafana
|
||||
keywords:
|
||||
- grafana
|
||||
- elasticsearch
|
||||
- guide
|
||||
- data source
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Configure Elasticsearch
|
||||
title: Configure the Elasticsearch data source
|
||||
weight: 200
|
||||
---
|
||||
|
||||
# Configure the Elasticsearch data source
|
||||
|
||||
Grafana ships with built-in support for Elasticsearch.
|
||||
You can make many types of queries to visualize logs or metrics stored in Elasticsearch, and annotate graphs with log events stored in Elasticsearch.
|
||||
|
||||
For general documentation on querying data sources in Grafana, see [Query and transform data]({{< relref "../../panels-visualizations/query-transform-data" >}}).
|
||||
|
||||
For instructions on how to add a data source to Grafana, refer to the [administration documentation]({{< relref "../../administration/data-source-management/" >}}).
|
||||
Only users with the organization administrator role can add data sources.
|
||||
Administrators can also [configure the data source via YAML]({{< relref "#provision-the-data-source" >}}) with Grafana's provisioning system.
|
||||
|
||||
## Configure the data source
|
||||
|
||||
To add the Elasticsearch data source, complete the following steps:
|
||||
|
||||
1. Click **Connections** in the left-side menu.
|
||||
1. Under **Connections**, click **Add new connection**.
|
||||
1. Enter `Elasticsearch` in the search bar.
|
||||
1. Select **Elasticsearch data source**.
|
||||
1. Click **Create a Elasticsearch data source** in the upper right.
|
||||
|
||||
You will be taken to the **Settings** tab where you will set up your Elasticsearch configuration.
|
||||
|
||||
## Configuration options
|
||||
|
||||
The following is a list of configuration options for Elasticsearch.
|
||||
|
||||
The first option to configure is the name of your connection:
|
||||
|
||||
- **Name** - The data source name. This is how you refer to the data source in panels and queries. Examples: elastic-1, elasticsearch_metrics.
|
||||
|
||||
- **Default** - Toggle to select as the default data source option. When you go to a dashboard panel or Explore, this will be the default selected data source.
|
||||
|
||||
### HTTP section
|
||||
|
||||
- **URL** - The URL of your Elasticsearch server. If your Elasticsearch server is local, use `<http://localhost:9200>`. If it is on a server within a network, this is the URL with port where you are running Elasticsearch. Example: `<http://elasticsearch.example.orgname:9200>`.
|
||||
|
||||
- **Allowed cookies** - Specify cookies by name that should be forwarded to the data source. The Grafana proxy deletes all forwarded cookies by default.
|
||||
|
||||
- **Timeout** - The HTTP request timeout. This must be in seconds. There is no default, so this setting is up to you.
|
||||
|
||||
### Auth section
|
||||
|
||||
There are several authentication methods you can choose in the Authentication section.
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
Use TLS (Transport Layer Security) for an additional layer of security when working with Elasticsearch. For information on setting up TLS encryption with Elasticsearch see [Configure TLS](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/configuring-tls.html#configuring-tls). You must add TLS settings to your Elasticsearch configuration file **prior** to setting these options in Grafana.
|
||||
{{% /admonition %}}
|
||||
|
||||
- **Basic authentication** - The most common authentication method. Use your `data source` user name and `data source` password to connect.
|
||||
|
||||
- **With credentials** - Toggle to enable credentials such as cookies or auth headers to be sent with cross-site requests.
|
||||
|
||||
- **TLS client authentication** - Toggle to use client authentication. When enabled, add the `Server name`, `Client cert` and `Client key`. The client provides a certificate that is validated by the server to establish the client's trusted identity. The client key encrypts the data between client and server.
|
||||
|
||||
- **With CA cert** - Toggle to authenticate with a CA certificate. Follow the instructions of the CA (Certificate Authority) to download the certificate file.
|
||||
|
||||
- **Skip TLS verify** - Toggle on to bypass TLS certificate validation.
|
||||
|
||||
- **Forward OAuth identity** - Forward the OAuth access token (and the OIDC ID token if available) of the user querying the data source.
|
||||
|
||||
### Custom HTTP headers
|
||||
|
||||
- **Header** - Add a custom header. This allows custom headers to be passed based on the needs of your Elasticsearch instance.
|
||||
|
||||
- **Value** - The value of the header.
|
||||
|
||||
### Elasticsearch details
|
||||
|
||||
The following settings are specific to the Elasticsearch data source.
|
||||
|
||||
- **Index name** - Use the index settings to specify a default for the `time field` and your Elasticsearch index's name. You can use a time pattern, such as `YYYY.MM.DD`, or a wildcard for the index name.
|
||||
|
||||
- **Pattern** - Select the matching pattern if using one in your index name. Options include:
|
||||
|
||||
- no pattern
|
||||
- hourly
|
||||
- daily
|
||||
- weekly
|
||||
- monthly
|
||||
- yearly
|
||||
|
||||
- **Time field name** - Name of the time field. The default value is @timestamp. You can enter a different name.
|
||||
|
||||
- **Max concurrent shard requests** - Sets the number of shards being queried at the same time. The default is `5`. For more information on shards see [Elasticsearch's documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/scalability.html#scalability).
|
||||
|
||||
- **Min time interval** - Defines a lower limit for the auto group-by time interval. This value **must** be formatted as a number followed by a valid time identifier:
|
||||
|
||||
| Identifier | Description |
|
||||
| ---------- | ----------- |
|
||||
| `y` | year |
|
||||
| `M` | month |
|
||||
| `w` | week |
|
||||
| `d` | day |
|
||||
| `h` | hour |
|
||||
| `m` | minute |
|
||||
| `s` | second |
|
||||
| `ms` | millisecond |
|
||||
|
||||
We recommend setting this value to match your Elasticsearch write frequency.
|
||||
For example, set this to `1m` if Elasticsearch writes data every minute.
|
||||
|
||||
You can also override this setting in a dashboard panel under its data source options. The default is `10s`.
|
||||
|
||||
- **X-Pack enabled** - Toggle to enable `X-Pack`-specific features and options, which provide the [query editor]({{< relref "./query-editor/" >}}) with additional aggregations, such as `Rate` and `Top Metrics`.
|
||||
|
||||
- **Include frozen indices** - Toggle on when the `X-Pack enabled` setting is active. You can configure Grafana to include [frozen indices](https://www.elastic.co/guide/en/elasticsearch/reference/7.13/frozen-indices.html) when performing search requests.
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
Frozen indices are [deprecated in Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/frozen-indices.html) since v7.14.
|
||||
{{% /admonition %}}
|
||||
|
||||
### Logs
|
||||
|
||||
In this section you can configure which fields the data source uses for log messages and log levels.
|
||||
|
||||
- **Message field name:** - Grabs the actual log message from the default source.
|
||||
|
||||
- **Level field name:** - Name of the field with log level/severity information. When a level label is specified, the value of this label is used to determine the log level and update the color of each log line accordingly. If the log doesn’t have a specified level label, we try to determine if its content matches any of the [supported expressions](/docs/grafana/latest/explore/logs-integration/#log-level). The first match always determines the log level. If Grafana cannot infer a log-level field, it will be visualized with an unknown log level.
|
||||
|
||||
### Data links
|
||||
|
||||
Data links create a link from a specified field that can be accessed in Explore's logs view. You can add multiple data links
|
||||
|
||||
Each data link configuration consists of:
|
||||
|
||||
- **Field** - Sets the name of the field used by the data link.
|
||||
|
||||
- **URL/query** - Sets the full link URL if the link is external. If the link is internal, this input serves as a query for the target data source.<br/>In both cases, you can interpolate the value from the field with the `${__value.raw }` macro.
|
||||
|
||||
- **URL Label** (Optional) - Sets a custom display label for the link. The link label defaults to the full external URL or name of the linked internal data source and is overridden by this setting.
|
||||
|
||||
- **Internal link** - Toggle on to set an internal link. For an internal link, you can select the target data source with a data source selector. This supports only tracing data sources.
|
||||
@ -15,6 +15,7 @@ labels:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
- data source
|
||||
menuTitle: Query editor
|
||||
title: Elasticsearch query editor
|
||||
weight: 300
|
||||
@ -22,52 +23,128 @@ weight: 300
|
||||
|
||||
# Elasticsearch query editor
|
||||
|
||||
{{< figure src="/static/img/docs/elasticsearch/query-editor-7-4.png" max-width="500px" class="docs-image--no-shadow" caption="Elasticsearch Query Editor" >}}
|
||||
Grafana provides a query editor for Elasticsearch. Elasticsearch queries are in Lucene format. See [Query string syntax](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/query-dsl-query-string-query.html#query-string-syntax) if you are new to working with Elasticsearch.
|
||||
|
||||
This topic explains querying specific to the Elasticsearch data source.
|
||||
For general documentation on querying data sources in Grafana, see [Query and transform data][query-transform-data].
|
||||
{{< figure src="/static/img/docs/elasticsearch/elastic-query-editor-10.1.png" max-width="800px" class="docs-image--no-shadow" caption="Elasticsearch query editor" >}}
|
||||
|
||||
## Select and edit metrics
|
||||
For general documentation on querying data sources in Grafana, including options and functions common to all query editors, see [Query and transform data]({{< relref "../../../panels-visualizations/query-transform-data" >}}).
|
||||
|
||||
## Aggregation types
|
||||
|
||||
Elasticsearch groups aggregations into three categories:
|
||||
|
||||
- **Bucket** - Bucket aggregations don't calculate metrics, they create buckets of documents based on field values, ranges and a variety of other criteria. See [Bucket aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket.html) for additional information. Use bucket aggregations under `Group by` when creating a metrics query in the query builder.
|
||||
|
||||
- **Metrics** - Metrics aggregations perform calculations such as sum, average, min, etc. They can be single-value or multi-value. See [Metrics aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics.html) for additional information. Use metrics aggregations in the metrics query type in the query builder.
|
||||
|
||||
- **Pipeline** - Elasticsearch pipeline aggregations work with inputs or metrics created from other aggregations (not documents or fields). There are parent and sibling and sibling pipeline aggregations. See [Pipeline aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/search-aggregations-pipeline.html) for additional information.
|
||||
|
||||
## Common options
|
||||
|
||||
There are several different types of queries you can create using the Elasticsearch query editor. The following options are available for all query types.
|
||||
|
||||
### Add query
|
||||
|
||||
Regardless of query type, you can create multiple queries by clicking **+ Add query**.
|
||||
|
||||
### Query inspector
|
||||
|
||||
Click **Query inspector** to get detailed statistics regarding your query. Query inspector functions as a kind of debugging tool that "inspects" your query. It provides query statistics under **Stats**, request response time under **Query**, data frame details under **{} JSON**, and the shape of your data under **Data**.
|
||||
|
||||
## Select a query type
|
||||
|
||||
There are three types of queries you can create with the Elasticsearch query builder. Each type is explained in detail below.
|
||||
|
||||
### Metrics query type
|
||||
|
||||
Metrics queries aggregate data and produce a variety of calculations such as count, min, max, etc. Click on the metric box to view a list of options in the dropdown menu. The default is `count`.
|
||||
|
||||
- **Alias** - Aliasing only applies to **time series queries**, where the last group is `date histogram`. This is ignored for any other type of query.
|
||||
|
||||
- **Metric** - Metrics aggregations include:
|
||||
|
||||
- count - see [Value count aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/search-aggregations-metrics-valuecount-aggregation.html)
|
||||
- average - see [Avg aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/search-aggregations-metrics-rate-aggregation.html)
|
||||
- sum - see [Sum aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-sum-aggregation.html)
|
||||
- max - see [Max aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/search-aggregations-metrics-max-aggregation.html)
|
||||
- min - see [Min aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/search-aggregations-metrics-min-aggregation.html)
|
||||
- extended stats - see [Extended stats aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-extendedstats-aggregation.html)
|
||||
- percentiles - see [Percentiles aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/search-aggregations-metrics-percentile-aggregation.html)
|
||||
- unique count - see [Cardinlaity aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/search-aggregations-metrics-cardinality-aggregation.html)
|
||||
- top metrics - see [Top metrics aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/search-aggregations-metrics-top-metrics.html)
|
||||
- rate - see [Rate aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/search-aggregations-metrics-rate-aggregation.html)
|
||||
|
||||
You can select multiple metrics and group by multiple terms or filters when using the Elasticsearch query editor.
|
||||
|
||||
Use the plus and minus icons to the right to add and remove metrics or group by clauses.
|
||||
To expand the row to view and edit any available metric or group-by options, click the option text.
|
||||
Use the **plus icon** to the right to add multiple metrics to your query. Click on the **eye icon** next to "Metric" to hide metrics, and the **garbage can icon** to remove metrics.
|
||||
|
||||
- **Group by options** - Create multiple group by options when constructing your Elasticsearch query. Date histogram is the default option. Below is a list of options in the dropdown menu.
|
||||
|
||||
- terms - see [Terms aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html).
|
||||
- filter - see [Filter aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filter-aggregation.html).
|
||||
- geo hash grid - see [Geohash grid aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-geohashgrid-aggregation.html).
|
||||
- date histogram - for time series queries. See [Date histogram aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html).
|
||||
- histogram - Depicts frequency distributions. See [Histogram aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-histogram-aggregation.html).
|
||||
- nested (experimental) - See [Nested aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-nested-aggregation.html).
|
||||
|
||||
Each group by option will have a different subset of options to further narrow your query.
|
||||
|
||||
The following options are specific to the **date histogram** bucket aggregation option.
|
||||
|
||||
- **Time field** - Depicts date data options. The default option can be specified when configuring the Elasticsearch data source in the **Time field name** under the [**Elasticsearch details**](/docs/grafana/latest/datasources/elasticsearch/configure-elasticsearch-data-source/#elasticsearch-details) section. Otherwise **@timestamp** field will be used as a default option.
|
||||
- **Interval** - Group by a type of interval. There are option to choose from the dropdown menu to select seconds, minutes, hours or day. You can also add a custom interval such as `30d` (30 days). `Auto` is the default option.
|
||||
- **Min doc count** - The minimum amount of data to include in your query. The default is `0`.
|
||||
- **Thin edges** - Select to trim edges on the time series data points. The default is `0`.
|
||||
- **Offset** - Changes the start value of each bucket by the specified positive(+) or negative (-) offset duration. Examples include `1h` for 1 hour, `5s` for 5 seconds or `1d` for 1 day.
|
||||
- **Timezone** - Select a timezone from the dropdown menu. The default is `Coordinated universal time`.
|
||||
|
||||
Configure the following options for the **terms** bucket aggregation option:
|
||||
|
||||
- **Order** - Sets the order of data. Options are `top` or `bottom.`
|
||||
- **Size** - Limits the number of documents, or size of the data set. You can set a custom number or `no limit`.
|
||||
- **Min doc count** - The minimum amount of data to include in your query. The default is `0`.
|
||||
- **Order by** - Order terms by `term value`, `doc count` or `count`.
|
||||
- **Missing** - Defines how documents missing a value should be treated. Missing values are ignored by default, but they can be treated as if they had a value. See [Missing value](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_missing_value_5) in Elasticsearch's documentation for more information.
|
||||
|
||||
Configure the following options for the **filters** bucket aggregation option:
|
||||
|
||||
- **Query** - Specify the query to create a bucket of documents (data). Examples are `hostname:"hostname1"`, `product:"widget5"`. Use the \* wildcard to match any number of characters.
|
||||
- **Label** - Add a label or name to the bucket.
|
||||
|
||||
Configure the following options for the **geo hash grid** bucket aggregation option:
|
||||
|
||||
- **Precision** - Specifies the number of characters of the geo hash.
|
||||
|
||||
Configure the following options for the **histogram** bucket aggregation option:
|
||||
|
||||
- **Interval** - Group by a type of interval. There are option to choose from the dropdown menu to select seconds, minutes, hours or day. You can also add a custom interval such as `30d` (30 days). `Auto` is the default option.
|
||||
- **Min doc count** - The minimum amount of data to include in your query. The default is `0`
|
||||
|
||||
The **nested** group by option is currently experimental, you can select a field and then settings specific to that field.
|
||||
|
||||
Click the **+ sign** to add multiple group by options. The data will grouped in order (first by, then by).
|
||||
|
||||
{{< figure src="/static/img/docs/elasticsearch/group-by-then-by-10.2.png" max-width="850px" class="docs-image--no-shadow" caption="Group by options" >}}
|
||||
|
||||
### Logs query type
|
||||
|
||||
Logs queries analyze Elasticsearch log data. You can configure the following options:
|
||||
|
||||
- **Logs Options/Limit** - Limits the number of logs to analyze. The default is `500`.
|
||||
|
||||
### Raw data query type
|
||||
|
||||
Run a raw data query to retrieve a table of all fields that are associated with each log line.
|
||||
|
||||
- **Raw data size** - Number of raw data documents. You can specify a different amount. The default is `500`.
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
The option to run a **raw document query** is deprecated as of Grafana v10.1.
|
||||
{{% /admonition %}}
|
||||
|
||||
## Use template variables
|
||||
|
||||
You can also augment queries by using [template variables]({{< relref "./template-variables" >}}).
|
||||
|
||||
## Name a time series
|
||||
|
||||
You can control the name for time series via the `Alias` input field.
|
||||
|
||||
| Pattern | Replacement value |
|
||||
| -------------------- | -------------------------------------- |
|
||||
| `{{term fieldname}}` | Value of a term group-by |
|
||||
| `{{metric}}` | Metric name, such as Average, Min, Max |
|
||||
| `{{field}}` | Metric field name |
|
||||
|
||||
## Control pipeline metrics visibility
|
||||
|
||||
Some metric aggregations, such as _Moving Average_ and _Derivative_, are called **Pipeline** aggregations.
|
||||
Elasticsearch pipeline metrics must be based on another metric.
|
||||
Use the eye icon next to the metric to prevent metrics from appearing in the graph.
|
||||
This is useful for metrics you only have in the query for use in a pipeline metric.
|
||||
|
||||
{{< figure src="/static/img/docs/elasticsearch/pipeline-aggregation-editor-7-4.png" max-width="500px" class="docs-image--no-shadow" caption="Pipeline aggregation editor" >}}
|
||||
|
||||
## Create a query
|
||||
|
||||
Write the query using a custom JSON string, with the field mapped as a [keyword](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#keyword) in the Elasticsearch index mapping.
|
||||
|
||||
If the query is [multi-field](https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html) with both a `text` and `keyword` type, use `"field":"fieldname.keyword"` (sometimes `fieldname.raw`) to specify the keyword field in your query.
|
||||
|
||||
| Query | Description |
|
||||
| ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `{"find": "fields", "type": "keyword"}` | Returns a list of field names with the index type `keyword`. |
|
||||
| `{"find": "terms", "field": "hostname.keyword", "size": 1000}` | Returns a list of values for a keyword using term aggregation. Query will use current dashboard time range as time range query. |
|
||||
| `{"find": "terms", "field": "hostname", "query": '<Lucene query>'}` | Returns a list of values for a keyword field using term aggregation and a specified Lucene query filter. Query will use current dashboard time range as time range for query. |
|
||||
You can also augment queries by using [template variables]({{< relref "./template-variables/" >}}).
|
||||
|
||||
Queries of `terms` have a 500-result limit by default.
|
||||
To set a custom limit, set the `size` property in your query.
|
||||
|
||||
@ -72,6 +72,21 @@ The example also uses a variable in the _Terms_ group by field input box, which
|
||||
|
||||
To view an example dashboard on Grafana Play, see the [Elasticsearch Templated Dashboard](https://play.grafana.org/d/z8OZC66nk/elasticsearch-8-2-0-sample-flight-data?orgId=1).
|
||||
|
||||
## Create a query
|
||||
|
||||
Write the query using a custom JSON string, with the field mapped as a [keyword](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#keyword) in the Elasticsearch index mapping.
|
||||
|
||||
If the query is [multi-field](https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html) with both a `text` and `keyword` type, use `"field":"fieldname.keyword"` (sometimes `fieldname.raw`) to specify the keyword field in your query.
|
||||
|
||||
| Query | Description |
|
||||
| ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `{"find": "fields", "type": "keyword"}` | Returns a list of field names with the index type `keyword`. |
|
||||
| `{"find": "terms", "field": "hostname.keyword", "size": 1000}` | Returns a list of values for a keyword using term aggregation. Query will use current dashboard time range as time range query. |
|
||||
| `{"find": "terms", "field": "hostname", "query": '<Lucene query>'}` | Returns a list of values for a keyword field using term aggregation and a specified Lucene query filter. Query will use current dashboard time range as time range for query. |
|
||||
|
||||
Queries of `terms` have a 500-result limit by default.
|
||||
To set a custom limit, set the `size` property in your query.
|
||||
|
||||
{{% docs/reference %}}
|
||||
[add-template-variables-multi-value-variables]: "/docs/grafana/ -> /docs/grafana/<GRAFANA VERSION>/dashboards/variables/add-template-variables#multi-value-variables"
|
||||
[add-template-variables-multi-value-variables]: "/docs/grafana-cloud/ -> /docs/grafana/<GRAFANA VERSION>/dashboards/variables/add-template-variables#multi-value-variables"
|
||||
|
||||
Reference in New Issue
Block a user