57 Commits

Author SHA1 Message Date
9fb8339f87 Elastic: Allow using long/int as date field for alerts (#44027)
* Use integers for time range filter

Previously it was passed as a string which is automatically converted by Elastic to a number only if the field type is "date". For other types (e.g. "long") such conversion doesn't work. In theory "date" could be passed as a formatted string but we don't use it this way and always pass it as a number so it is safe to always pass numbers, not strings.

* Fix time_series_query_test

* Retrigger build
2022-01-17 15:45:09 +01:00
1b99d88337 Elasticsearch: Add support for Elasticsearch 8.0 (Beta) (#41729)
* use fixed_interval in date_histogram

* Add 8.0 to available versions in datasource settings

* Remove moving_avg from available metric aggregations

* Add ES8 devenv

* Update public/app/plugins/datasource/elasticsearch/components/QueryEditor/MetricAggregationsEditor/utils.ts

Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>

* Add FE tests

* Add BE test

* fix FE test

* fix BE test

Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>
2021-11-17 20:30:53 +01:00
7b7c193551 Elasticsearch: Add time zone setting to Date Histogram aggregation (#40882)
* TimeZonePicker: Allow specifying internal timezones to display

* Elasticsearch: Add time zone setting to Date Histogram aggregation

* rename time_zone to timeZone

* Add tests

* fix typo

* Update packages/grafana-data/src/datetime/timezones.ts

Co-authored-by: Chris Cowan <chris@chriscowan.us>

Co-authored-by: Chris Cowan <chris@chriscowan.us>
2021-11-02 16:18:39 +00:00
76d88a6cec change log level to debug (#41102) 2021-11-01 09:00:00 +00:00
a45861298f Chore: Refactor GoConvey in tsdb/es/client (#40846) 2021-10-26 13:15:47 +02:00
c989f0fbbe SQL: Migrate to use SDK contracts (#36635)
* convert SQLs to use sdk contracts

* make draft

* postgres

* intermedia

* get datasourceinfo filled at the beginning of the service

* move the interval into package because of cyclict  import and fix all postgres tests

* fix mysql test

* fix mssql

* fix the test for pr https://github.com/grafana/grafana/issues/35839

* fix some issue about intervalv2 package

* update sql test

* wire migration for SQLs

* add sqls to the background process

* make it register instead of register and start

* revert formatting

* fix tests

* fix linter

* remove integration test

* Postgres test fix

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2021-09-07 09:35:37 +02:00
0df1b33d71 Plugins: Migrate Elasticsearch to backend plugin SDK (#36132)
* Migrate Elasticsearch to backend plugin SDK

* Fix linting

* Move away from Convey!

* Rebase commit

* Small logger fix

* Fixes according to reviewer's comments

* Fixes according to reviewer's comments

* Fixes according to reviewer's comments

* More cleanup

* Move things around - small refactoring

* Fix typo

* Update calculator - add tests

* Fixes according to reviewer's comments
2021-07-15 16:45:59 +02:00
f6b83a4f47 Elasticsearch: Add frozen indices search support (#36018)
* Revert "Revert "Elasticsearch: add frozen indices search support (#27472)" (#27726)"

This reverts commit 4c7131425ba048c648435410f2153d4d85f20304.

* Make label width a bit more consistent

* Add documentation for X-Pack & Frozen Indices support in Elasticsearch

* Change UI & docs casing

* create default empty dataframe

* Remove backticks and simplify regex

* small doc improvement
2021-07-15 15:52:02 +02:00
f580c9149c Elasticsearch: Add Top Metrics Aggregation and X-Pack support (#33041)
* Elasticsearch: Add Top Metrics Aggregation

* Adding support for non-timeseries visualizations

* removing console.logs

* restoring loadOptions type

* Honor xpack setting

* Adding test for elastic_response

* adding test for query builder

* Adding support of alerting

* Fixing separator spelling

* Fixing linting issues

* attempting to reduce cyclomatic complexity

* Adding elastic77 Docker block

* Update public/app/plugins/datasource/elasticsearch/components/QueryEditor/MetricAggregationsEditor/MetricEditor.test.tsx

Co-authored-by: Giordano Ricci <grdnricci@gmail.com>

* refactoring MetricsEditor tests

* Fixing typo

* Change getFields type & move TopMetrics to a separate component

* Fix SegmentAsync styles in TopMetrics Settings

* Fix field types for TopMetrics

* WIP

* Refactoring client side to support multiple top metrics

* Adding tests and finishing go implimentation

* removing fmt lib from debugging

* fixing tests

* reducing the cyclomatic complexity

* Update public/app/plugins/datasource/elasticsearch/elastic_response.ts

Co-authored-by: Giordano Ricci <grdnricci@gmail.com>

* Update public/app/plugins/datasource/elasticsearch/hooks/useFields.ts

Co-authored-by: Giordano Ricci <grdnricci@gmail.com>

* Checking for possible nil value

* Fixing types

* fix fake-data-gen param

* fix useFields hook

* Removing aggregateBy and size

* Fixing go tests

* Fixing TS tests

* fixing tests

* Fixes

* Remove date from top_metrics fields

* Restore previous formatting

* Update pkg/tsdb/elasticsearch/client/models.go

Co-authored-by: Dimitris Sotirakis <dimitrios.sotirakis@grafana.com>

* Update pkg/tsdb/elasticsearch/client/models.go

Co-authored-by: Dimitris Sotirakis <dimitrios.sotirakis@grafana.com>

* Fix code review comments on processTopMetricValue

* Remove underscore from variable names

* Remove intermediate array definition

* Refactor test to use testify

Co-authored-by: Giordano Ricci <grdnricci@gmail.com>
Co-authored-by: Elfo404 <me@giordanoricci.com>
Co-authored-by: Dimitris Sotirakis <dimitrios.sotirakis@grafana.com>
2021-06-04 11:07:59 +01:00
530072fce7 Elasticsearch: fix NewClient not passing httpClientProvider to client impl (#34539)
* Update client.go

fix missing "httpClientProvider" in NewClient

* Update client.go
2021-05-24 14:45:29 +01:00
348e76fc8e Datasource: Shared HTTP client provider for core backend data sources and any data source using the data source proxy (#33439)
Uses new httpclient package from grafana-plugin-sdk-go introduced 
via grafana/grafana-plugin-sdk-go#328. 
Replaces the GetHTTPClient, GetTransport, GetTLSConfig methods defined 
on DataSource model.
Longer-term the goal is to migrate core HTTP backend data sources to use the 
SDK contracts and using httpclient.Provider for creating HTTP clients and such.

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2021-05-19 23:53:41 +02:00
e98a8bd11b Elasticsearch: use semver strings to identify ES version (#33646)
* Elasticsearch: use proper semver strings to identify ES version

* Update BE & tests

* refactor BE tests

* refactor isValidOption check

* update test

* Update pkg/tsdb/elasticsearch/client/client.go

Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>

* Update pkg/tsdb/elasticsearch/client/search_request_test.go

Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>

* Remove leftover FIXME comment

* Add new test cases for new version format

* Docs: add documentation about version dropdown

* Update docs/sources/datasources/elasticsearch.md

Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>

* Update docs/sources/datasources/elasticsearch.md

Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>

* Update docs/sources/datasources/elasticsearch.md

Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>

* Update provisioning documentation

Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
2021-05-11 09:44:00 +01:00
cdb4785496 Elasticsearch: Allow omitting field when metric supports inline script (#32839)
* Elasticsearch: Allow omitting field when metric supports inline script

* Add tests for MetricEditor to show a None option

* Add tests for useFields hook

* Alerting: allow elasticsearch metrics without field
2021-04-14 15:18:06 +01:00
f7b408f99f Elasticsearch: use application/x-ndjson content type for multisearch requests (#32282) 2021-03-29 23:41:45 +01:00
b79e61656a Introduce TSDB service (#31520)
* Introduce TSDB service

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

Co-authored-by: Erik Sundell <erik.sundell87@gmail.com>
Co-authored-by: Will Browne <will.browne@grafana.com>
Co-authored-by: Torkel Ödegaard <torkel@grafana.org>
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
Co-authored-by: Zoltán Bedi <zoltan.bedi@gmail.com>
2021-03-08 07:02:49 +01:00
b2b3a603e8 Chore: Rewrite elasticsearch client test to standard library (#30093) 2021-01-07 09:35:56 +01:00
c2cad26ca9 Chore: Disable default golangci-lint filter (#29751)
* Disable default golangci-lint filter

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Chore: Fix linter warnings

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-12-15 09:32:06 +01:00
ac09baae7d Chore: Use Header.Set method instead of Header.Add (#29804)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-12-14 15:13:01 +01:00
f55818ca70 Chore: Enable exhaustive linter (#29458)
* Chore: Enable exhaustive linter

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-12-01 09:53:27 +01:00
ee3475d900 Elasticsearch: Fix index pattern not working with multiple base sections (#28348)
* Add logic to allow for multiple [] sections in pattern

* Small refactor

* Fix goimport error

* Refactor formateDate
2020-11-27 10:50:35 +01:00
4dd7b7a82d Chore: Remove unused Go code (#28852)
* Chore: Remove more unused Go code

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-11-17 11:51:31 +01:00
4c7131425b Revert "Elasticsearch: add frozen indices search support (#27472)" (#27726)
This reverts commit 61a3160c34f659be848d67a10bd9e0e5be6e1b58.
2020-09-23 12:24:46 +01:00
a5d9196a53 Chore/fix lint issues (#27704)
* Chore: Fix linting issues

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-09-22 16:22:19 +02:00
61a3160c34 Elasticsearch: add frozen indices search support (#27472)
* Elasticsearch: Add frozen indices support for alerting
2020-09-18 18:27:25 +01:00
5070f7a75b Chore: Start harmonizing linting with plugin SDK (#25854)
* Chore: Harmonize linting with plugin SDK

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Chore: Fix linting issues

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-06-29 14:08:32 +02:00
58cefe73ee Datasources: Handle URL parsing error (#25742)
Adds handling of error returned from URL parsing.

Fixes #25714
2020-06-22 16:34:40 +02:00
bb50fc5965 pkg/tsdb: Check errors (#19837)
Check errors in pkg/tsdb.
2019-10-22 16:16:53 +02:00
caa1314f44 Build: use golangci-lint as a make command (#17739)
* Build: use golangci-lint as a make command

* Since gometalinter was deprecated in favor of golangci-lint so it was
  replaced by it. Responsibilities held by the gometalinter was moved to
  golangci-lint

* There was some changes in implementation (that was also mentioned in
  the code comment) between the tools, which uncovered couple errors
  in the code. Those issues were either solved or disabled by
  the inline comments

* Introduce the golangci-lint config, to make their
  configuration more manageable

* Build: replace backend-lint.sh script with make
2019-07-02 16:06:59 +03:00
5713048f48 Alerting: Improve alert rule testing (#16286)
* tsdb: add support for setting debug flag of tsdb query

* alerting: adds debug flag in eval context

Debug flag is set when testing an alert rule and this debug
flag is used to return more debug information in test aler rule
response. This debug flag is also provided to tsdb queries so
datasources can optionally add support for returning additional
debug data

* alerting: improve test alert rule ui

Adds buttons for expand/collapse json and copy json to clipboard,
very similar to how the query inspector works.

* elasticsearch: implement support for tsdb query debug flag

* elasticsearch: embedding client response in struct

* alerting: return proper query model when testing rule
2019-06-25 08:52:17 +02:00
2fff8f77dc move log package to /infra (#17023)
ref #14679

Signed-off-by: zhulongcheng <zhulongcheng.me@gmail.com>
2019-05-13 08:45:54 +02:00
32a45077e7 Elasticsearch: Fix pre-v7.0 and alerting error (#16904)
This fixes a regression introduced in #16646 where using Elasticsearch 
pre-v7.0 and alerting resulted in an error when trying to deserialize the 
response of total number of hits.
Total number of hits is not in use by the backend so we're removing it 
for now to make ES 6 and 7 being able to deserialize search responses 
without errors.

Closes #15622
2019-05-06 15:12:18 +02:00
cff2be0d66 elasticsearch: add 7.x version support (#16646)
Adds a new version option 7.0+ (70 internally).
Version 7.0+ doesn't include document types in index mappings 
so support for handling this have been added.
Version 7.0+ returns number of hits in a different way so 
support for handling this have been added.
Version 7.0+ doesn't support sending 
max_concurrent_shard_requests in multisearch header so 
support for sending this in query string have been added.
Update elastic6 docker block and dashboards (devenv) to use 
6.7.1 images, filebeat index name is now filebeat-YYYY.MM.DD 
and dashboard include correct tags and links.
Add elastic7 docker block and provisioning (devenv).
Updates documentation regarding new version.

Closes #15622
2019-04-25 09:41:13 +02:00
66f6e16916 Security: Store datasource passwords encrypted in secureJsonData (#16175)
* Store passwords in secureJsonData

* Revert unnecessary refactors

* Fix for nil jsonSecureData value

* Remove copied encryption code from migration

* Fix wrong field reference

* Remove migration and provisioning changes

* Use password getters in datasource proxy

* Refactor password handling in datasource configs

* Add provisioning warnings

* Update documentation

* Remove migration command, moved to separate PR

* Remove unused code

* Set the upgrade version

* Remove unused code

* Remove double reference
2019-04-15 11:11:17 +02:00
e04218e550 Fix: #14706 Incorrect index pattern padding in alerting queries
When weekly index pattern is used, indices names contain single digit week number for week 1-9
This fix makes sure indice names always contain 2 digit week number for weekly pattern
2019-03-03 18:10:38 +05:30
b45f72a140 elasticsearch: support bucket script pipeline aggregations 2018-12-20 01:24:36 +01:00
f6fc7f7b64 Requested Backend changes, added details to popover description for the offset field 2018-11-26 08:58:25 -05:00
a022284cb0 fix handle of elasticsearch 6.0+ version 2018-11-23 15:24:16 +01:00
5803bfd2c7 fix terms agg order deprecation warning on es 6+ 2018-11-02 17:53:47 +01:00
88efb87b0d pkg/tsdb/elasticsearch/client/client_test.go: pass context.Background() instead of nil
See,
$ gometalinter --vendor --disable-all --enable=megacheck --disable=gotype --deadline 6m ./...
pkg/tsdb/elasticsearch/client/client_test.go:28:25⚠️ do not pass a nil Context, even if a function permits it; pass context.TODO if you are unsure about which Context to use (SA1012) (megacheck)
pkg/tsdb/elasticsearch/client/client_test.go:39:25⚠️ do not pass a nil Context, even if a function permits it; pass context.TODO if you are unsure about which Context to use (SA1012) (megacheck)
pkg/tsdb/elasticsearch/client/client_test.go:51:25⚠️ do not pass a nil Context, even if a function permits it; pass context.TODO if you are unsure about which Context to use (SA1012) (megacheck)
pkg/tsdb/elasticsearch/client/client_test.go:63:25⚠️ do not pass a nil Context, even if a function permits it; pass context.TODO if you are unsure about which Context to use (SA1012) (megacheck)
pkg/tsdb/elasticsearch/client/client_test.go:76:25⚠️ do not pass a nil Context, even if a function permits it; pass context.TODO if you are unsure about which Context to use (SA1012) (megacheck)
pkg/tsdb/elasticsearch/client/client_test.go:89:25⚠️ do not pass a nil Context, even if a function permits it; pass context.TODO if you are unsure about which Context to use (SA1012) (megacheck)
2018-10-03 09:59:10 +02:00
80fa66fcb0 Fix some typos found by codespell
See,
$ codespell -S "./.git*,./vendor*,./public*"
2018-09-21 11:54:52 +02:00
3689bb778c Fix misspell issues
See,
$ gometalinter --disable-all --enable misspell --deadline 10m --vendor ./...
pkg/api/dtos/alerting_test.go:32:13⚠️ "expectes" is a misspelling of "expects" (misspell)
pkg/api/static/static.go:2:18⚠️ "Unknwon" is a misspelling of "Unknown" (misspell)
pkg/components/imguploader/azureblobuploader.go:55:48⚠️ "conatiner" is a misspelling of "container" (misspell)
pkg/login/ldap_settings.go:51:115⚠️ "compatability" is a misspelling of "compatibility" (misspell)
pkg/middleware/auth_proxy_test.go:122:22⚠️ "Destory" is a misspelling of "Destroy" (misspell)
pkg/middleware/logger.go:2:18⚠️ "Unknwon" is a misspelling of "Unknown" (misspell)
pkg/services/notifications/codes.go:9:13⚠️ "Unknwon" is a misspelling of "Unknown" (misspell)
pkg/services/session/mysql.go:170:3⚠️ "Destory" is a misspelling of "Destroy" (misspell)
pkg/services/session/mysql.go:171:24⚠️ "Destory" is a misspelling of "Destroy" (misspell)
pkg/services/session/session.go:95:4⚠️ "Destory" is a misspelling of "Destroy" (misspell)
pkg/services/session/session.go:96:1⚠️ "Destory" is a misspelling of "Destroy" (misspell)
pkg/services/session/session.go:167:25⚠️ "Destory" is a misspelling of "Destroy" (misspell)
pkg/setting/setting.go:1:18⚠️ "Unknwon" is a misspelling of "Unknown" (misspell)
pkg/tsdb/cloudwatch/cloudwatch.go:199:14⚠️ "resolutin" is a misspelling of "resolutions" (misspell)
pkg/tsdb/cloudwatch/cloudwatch.go:270:15⚠️ "resolutin" is a misspelling of "resolutions" (misspell)
pkg/tsdb/elasticsearch/response_parser.go:531:24⚠️ "Unkown" is a misspelling of "Unknown" (misspell)
pkg/tsdb/elasticsearch/client/search_request.go:113:7⚠️ "initaite" is a misspelling of "initiate" (misspell)

Note: Unknwon is a library name, and Destory a mysql typo.
2018-09-21 11:54:36 +02:00
b609d81194 pkg/tsdb/elasticsearch/client/search_request.go: simplify loop with
append.

$ gometalinter --vendor --disable-all --enable=megacheck --disable=gotype --deadline=6m ./...
pkg/tsdb/elasticsearch/client/search_request.go:59:4⚠️ should replace loop with sr.Aggs = append(sr.Aggs, aggArray...) (S1011) (megacheck)
pkg/tsdb/elasticsearch/client/search_request.go:303:4⚠️ should replace loop with agg.Aggregation.Aggs = append(agg.Aggregation.Aggs, childAggs...) (S1011) (megacheck)
2018-09-20 20:00:02 +02:00
13a1d0a026 pkg/tsdb/elasticsearch/client/client.go: use time.Since instead of time.Now().Sub
$ gometalinter --vendor --disable=all --enable=megacheck --deadline=10m ./...
pkg/tsdb/elasticsearch/client/client.go:147:13⚠️ should use time.Since instead of time.Now().Sub (S1012) (megacheck)
pkg/tsdb/elasticsearch/client/client.go:190:14⚠️ should use time.Since instead of time.Now().Sub (S1012) (megacheck)
pkg/tsdb/elasticsearch/client/client.go:218:13⚠️ should use time.Since instead of time.Now().Sub (S1012) (megacheck)
2018-09-18 23:42:25 +02:00
0254a29e35 Interpolate $__interval in backend for alerting with sql datasources (#13156)
add support for interpolate $__interval and  $__interval_ms in sql datasources
2018-09-13 16:51:00 +02:00
ab8fa0de74 elasticsearch: support reversed index patterns
Now both [index-]pattern and pattern[-index] are supported
2018-07-26 21:39:02 +02:00
8b32dc5847 move go vet out of scripts and fixing warning (#12552) 2018-07-09 23:25:32 -07:00
d2f31a716f remove unnecessary conversions 2018-07-04 12:16:39 +02:00
9db964bf35 Fix typo: eleasticsearch -> elasticsearch (#12184) 2018-06-06 13:09:16 -07:00
b8ff3b1e3f remove dead code 2018-05-31 19:05:32 +02:00
dcac63936b elasticsearch: minor refactor
Handle all replacements if interval template variables in the client.
Fix issue with client and different versions.
Adds better tests of the client
2018-05-31 19:02:20 +02:00