87 Commits

Author SHA1 Message Date
7a304223e6 Elasticsearch: Fix min_doc_count value when alerting (#35254)
* Elasticsearch: Fix min_doc_count value when alerting

* Add tests
2021-06-04 16:42:00 +02:00
f580c9149c Elasticsearch: Add Top Metrics Aggregation and X-Pack support (#33041)
* Elasticsearch: Add Top Metrics Aggregation

* Adding support for non-timeseries visualizations

* removing console.logs

* restoring loadOptions type

* Honor xpack setting

* Adding test for elastic_response

* adding test for query builder

* Adding support of alerting

* Fixing separator spelling

* Fixing linting issues

* attempting to reduce cyclomatic complexity

* Adding elastic77 Docker block

* Update public/app/plugins/datasource/elasticsearch/components/QueryEditor/MetricAggregationsEditor/MetricEditor.test.tsx

Co-authored-by: Giordano Ricci <grdnricci@gmail.com>

* refactoring MetricsEditor tests

* Fixing typo

* Change getFields type & move TopMetrics to a separate component

* Fix SegmentAsync styles in TopMetrics Settings

* Fix field types for TopMetrics

* WIP

* Refactoring client side to support multiple top metrics

* Adding tests and finishing go implimentation

* removing fmt lib from debugging

* fixing tests

* reducing the cyclomatic complexity

* Update public/app/plugins/datasource/elasticsearch/elastic_response.ts

Co-authored-by: Giordano Ricci <grdnricci@gmail.com>

* Update public/app/plugins/datasource/elasticsearch/hooks/useFields.ts

Co-authored-by: Giordano Ricci <grdnricci@gmail.com>

* Checking for possible nil value

* Fixing types

* fix fake-data-gen param

* fix useFields hook

* Removing aggregateBy and size

* Fixing go tests

* Fixing TS tests

* fixing tests

* Fixes

* Remove date from top_metrics fields

* Restore previous formatting

* Update pkg/tsdb/elasticsearch/client/models.go

Co-authored-by: Dimitris Sotirakis <dimitrios.sotirakis@grafana.com>

* Update pkg/tsdb/elasticsearch/client/models.go

Co-authored-by: Dimitris Sotirakis <dimitrios.sotirakis@grafana.com>

* Fix code review comments on processTopMetricValue

* Remove underscore from variable names

* Remove intermediate array definition

* Refactor test to use testify

Co-authored-by: Giordano Ricci <grdnricci@gmail.com>
Co-authored-by: Elfo404 <me@giordanoricci.com>
Co-authored-by: Dimitris Sotirakis <dimitrios.sotirakis@grafana.com>
2021-06-04 11:07:59 +01:00
530072fce7 Elasticsearch: fix NewClient not passing httpClientProvider to client impl (#34539)
* Update client.go

fix missing "httpClientProvider" in NewClient

* Update client.go
2021-05-24 14:45:29 +01:00
e2e78f14d2 Elasticsearch: fix flaky test (#34517) 2021-05-20 17:13:48 +02:00
348e76fc8e Datasource: Shared HTTP client provider for core backend data sources and any data source using the data source proxy (#33439)
Uses new httpclient package from grafana-plugin-sdk-go introduced 
via grafana/grafana-plugin-sdk-go#328. 
Replaces the GetHTTPClient, GetTransport, GetTLSConfig methods defined 
on DataSource model.
Longer-term the goal is to migrate core HTTP backend data sources to use the 
SDK contracts and using httpclient.Provider for creating HTTP clients and such.

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2021-05-19 23:53:41 +02:00
8ec87250c1 Elasticsearch: Fix handling of inline scripts in different ES versions (#34070)
* Devenv: add block for es 5.0, provisioned datasource & dashboard

* Trasnsform script property based on running ES version

* Handle different scripts format in BE
2021-05-14 11:50:15 +01:00
e98a8bd11b Elasticsearch: use semver strings to identify ES version (#33646)
* Elasticsearch: use proper semver strings to identify ES version

* Update BE & tests

* refactor BE tests

* refactor isValidOption check

* update test

* Update pkg/tsdb/elasticsearch/client/client.go

Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>

* Update pkg/tsdb/elasticsearch/client/search_request_test.go

Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>

* Remove leftover FIXME comment

* Add new test cases for new version format

* Docs: add documentation about version dropdown

* Update docs/sources/datasources/elasticsearch.md

Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>

* Update docs/sources/datasources/elasticsearch.md

Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>

* Update docs/sources/datasources/elasticsearch.md

Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>

* Update provisioning documentation

Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
2021-05-11 09:44:00 +01:00
c88af6e221 Elasticsearch: Add generic support for template variables (#32762)
* Elasticsearch: Add generic support for template variables

* format MovingAverage settings as numbers

* Move formatting logic to query builder & forma serial_diff settings as numbers

* modify presence check

* add todo

* minor fixes

* transform string values to numbers

* Move casting logic

* Slightly cleaner implementation

* Add BE tests

* Leverage elastic validation when string doesn't resolve to a numeric value

* move newly introduced test to testify

* add FE query_builder tests

* check error

* Parse values to float instead of int

* Fix tests & ParseFloat bit size
2021-04-26 16:54:23 +01:00
7627b55ef4 TSDB: add deprecation comments to many tsdb structs (#33281) 2021-04-23 03:03:11 +02:00
cdb4785496 Elasticsearch: Allow omitting field when metric supports inline script (#32839)
* Elasticsearch: Allow omitting field when metric supports inline script

* Add tests for MetricEditor to show a None option

* Add tests for useFields hook

* Alerting: allow elasticsearch metrics without field
2021-04-14 15:18:06 +01:00
f7b408f99f Elasticsearch: use application/x-ndjson content type for multisearch requests (#32282) 2021-03-29 23:41:45 +01:00
b79e61656a Introduce TSDB service (#31520)
* Introduce TSDB service

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

Co-authored-by: Erik Sundell <erik.sundell87@gmail.com>
Co-authored-by: Will Browne <will.browne@grafana.com>
Co-authored-by: Torkel Ödegaard <torkel@grafana.org>
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
Co-authored-by: Zoltán Bedi <zoltan.bedi@gmail.com>
2021-03-08 07:02:49 +01:00
5088e2044a Elasticsearch: Support extended stats and percentiles in terms order by (#28910)
Adds support to the terms aggregation for ordering by percentiles and extended stats. 

Closes #5148

Co-authored-by: Giordano Ricci <grdnricci@gmail.com>
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2021-01-15 12:10:16 +01:00
3d7748d9f8 Elasticsearch: Use minimum interval for alerts (#30049)
The current backend code doesn't honor the minimum interval set in the UI for alerts using 
the Elasticsearch data source. This means that the data the alerts are triggering against 
will never match the data in the visualization if auto is used in the date histogram as interval. 
This fixes the problem to make sure that date histogram auto interval is set according to 
min interval set in UI for the query or fallback to data source min interval setting.

Fixes #22082

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2021-01-14 10:48:59 +01:00
cf1c01dd8b Elasticsearch: Add Support for Serial Differencing Pipeline Aggregation (#28618)
* Elasticsearch: Add support for serial diff pipeline aggregation

* Removing settings transsforms

* Removing unused deps

* removing unused dep

* Fixing type in test

* Adding backend support for serial_diff
2021-01-08 11:52:23 +00:00
b2b3a603e8 Chore: Rewrite elasticsearch client test to standard library (#30093) 2021-01-07 09:35:56 +01:00
c2cad26ca9 Chore: Disable default golangci-lint filter (#29751)
* Disable default golangci-lint filter

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Chore: Fix linter warnings

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-12-15 09:32:06 +01:00
ac09baae7d Chore: Use Header.Set method instead of Header.Add (#29804)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-12-14 15:13:01 +01:00
f55818ca70 Chore: Enable exhaustive linter (#29458)
* Chore: Enable exhaustive linter

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-12-01 09:53:27 +01:00
ee3475d900 Elasticsearch: Fix index pattern not working with multiple base sections (#28348)
* Add logic to allow for multiple [] sections in pattern

* Small refactor

* Fix goimport error

* Refactor formateDate
2020-11-27 10:50:35 +01:00
4dd7b7a82d Chore: Remove unused Go code (#28852)
* Chore: Remove more unused Go code

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-11-17 11:51:31 +01:00
4ed7612e57 Elasticsearch: Add Moving Function Pipeline Aggregation (#28131)
* Elasticsearch: Add Moving Function Pipeline Aggregation

* Removing support for moving average in 7.x

* Fixing getMetricAggTypes to handle undefined esVersion

* Adding moving_fn to go code

* Update public/app/plugins/datasource/elasticsearch/metric_agg.ts

Co-authored-by: Giordano Ricci <grdnricci@gmail.com>

* Adding test for esversion 70

* Removing default value for script, adding placeholder instead

* Fixing formatting

* Adding code for handling missing or obsolete aggregations

Co-authored-by: Giordano Ricci <grdnricci@gmail.com>
2020-11-11 17:42:28 +00:00
4c7131425b Revert "Elasticsearch: add frozen indices search support (#27472)" (#27726)
This reverts commit 61a3160c34f659be848d67a10bd9e0e5be6e1b58.
2020-09-23 12:24:46 +01:00
a5d9196a53 Chore/fix lint issues (#27704)
* Chore: Fix linting issues

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-09-22 16:22:19 +02:00
61a3160c34 Elasticsearch: add frozen indices search support (#27472)
* Elasticsearch: Add frozen indices support for alerting
2020-09-18 18:27:25 +01:00
7896836deb Chore: Reduce TSDB Go code complexity (#26401)
* tsdb: Make code less complex
2020-08-18 14:43:18 +02:00
d4e4cb4c71 Chore: Enable Go linter gocritic (#26224)
* Chore: Enable gocritic linter

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-07-16 14:39:01 +02:00
be961c5466 Elasticsearch: Fix using multiple bucket script aggregations when only grouping by terms (#24064)
* In the parser when it was configurate more bucket it put the formula in name column

* Tests

* Tests

* Tests

* Tests

* Tests

* Tests

* Process/Tests - backend (Go)

* Update pkg/tsdb/elasticsearch/response_parser.go

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* reverse

* Update pkg/tsdb/elasticsearch/response_parser.go

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2020-07-09 15:21:19 +02:00
41d432b5ae Chore: Enable whitespace linter (#25903)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-07-06 20:17:28 +02:00
d1e6214a4a Chore: Enable scopelint Go linter (#25896)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-06-29 17:04:38 +02:00
5070f7a75b Chore: Start harmonizing linting with plugin SDK (#25854)
* Chore: Harmonize linting with plugin SDK

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Chore: Fix linting issues

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-06-29 14:08:32 +02:00
58cefe73ee Datasources: Handle URL parsing error (#25742)
Adds handling of error returned from URL parsing.

Fixes #25714
2020-06-22 16:34:40 +02:00
51d1261a4b Elasticsearch: Adds cumulative sum aggregation support (#24820)
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2020-06-01 23:55:15 +02:00
bb50fc5965 pkg/tsdb: Check errors (#19837)
Check errors in pkg/tsdb.
2019-10-22 16:16:53 +02:00
39d8e73412 Chore: Upgrade to Go 1.13 (#19502)
* Chore: Upgrade to Go 1.13

Fixes: #18878

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Chore: Upgrade lint tools in order to work with Go 1.13

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Chore: Fix Go linting issues

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2019-10-02 13:59:05 +02:00
caa1314f44 Build: use golangci-lint as a make command (#17739)
* Build: use golangci-lint as a make command

* Since gometalinter was deprecated in favor of golangci-lint so it was
  replaced by it. Responsibilities held by the gometalinter was moved to
  golangci-lint

* There was some changes in implementation (that was also mentioned in
  the code comment) between the tools, which uncovered couple errors
  in the code. Those issues were either solved or disabled by
  the inline comments

* Introduce the golangci-lint config, to make their
  configuration more manageable

* Build: replace backend-lint.sh script with make
2019-07-02 16:06:59 +03:00
5713048f48 Alerting: Improve alert rule testing (#16286)
* tsdb: add support for setting debug flag of tsdb query

* alerting: adds debug flag in eval context

Debug flag is set when testing an alert rule and this debug
flag is used to return more debug information in test aler rule
response. This debug flag is also provided to tsdb queries so
datasources can optionally add support for returning additional
debug data

* alerting: improve test alert rule ui

Adds buttons for expand/collapse json and copy json to clipboard,
very similar to how the query inspector works.

* elasticsearch: implement support for tsdb query debug flag

* elasticsearch: embedding client response in struct

* alerting: return proper query model when testing rule
2019-06-25 08:52:17 +02:00
2fff8f77dc move log package to /infra (#17023)
ref #14679

Signed-off-by: zhulongcheng <zhulongcheng.me@gmail.com>
2019-05-13 08:45:54 +02:00
32a45077e7 Elasticsearch: Fix pre-v7.0 and alerting error (#16904)
This fixes a regression introduced in #16646 where using Elasticsearch 
pre-v7.0 and alerting resulted in an error when trying to deserialize the 
response of total number of hits.
Total number of hits is not in use by the backend so we're removing it 
for now to make ES 6 and 7 being able to deserialize search responses 
without errors.

Closes #15622
2019-05-06 15:12:18 +02:00
cff2be0d66 elasticsearch: add 7.x version support (#16646)
Adds a new version option 7.0+ (70 internally).
Version 7.0+ doesn't include document types in index mappings 
so support for handling this have been added.
Version 7.0+ returns number of hits in a different way so 
support for handling this have been added.
Version 7.0+ doesn't support sending 
max_concurrent_shard_requests in multisearch header so 
support for sending this in query string have been added.
Update elastic6 docker block and dashboards (devenv) to use 
6.7.1 images, filebeat index name is now filebeat-YYYY.MM.DD 
and dashboard include correct tags and links.
Add elastic7 docker block and provisioning (devenv).
Updates documentation regarding new version.

Closes #15622
2019-04-25 09:41:13 +02:00
66f6e16916 Security: Store datasource passwords encrypted in secureJsonData (#16175)
* Store passwords in secureJsonData

* Revert unnecessary refactors

* Fix for nil jsonSecureData value

* Remove copied encryption code from migration

* Fix wrong field reference

* Remove migration and provisioning changes

* Use password getters in datasource proxy

* Refactor password handling in datasource configs

* Add provisioning warnings

* Update documentation

* Remove migration command, moved to separate PR

* Remove unused code

* Set the upgrade version

* Remove unused code

* Remove double reference
2019-04-15 11:11:17 +02:00
e04218e550 Fix: #14706 Incorrect index pattern padding in alerting queries
When weekly index pattern is used, indices names contain single digit week number for week 1-9
This fix makes sure indice names always contain 2 digit week number for weekly pattern
2019-03-03 18:10:38 +05:30
b45f72a140 elasticsearch: support bucket script pipeline aggregations 2018-12-20 01:24:36 +01:00
c46d01dc83 Merge pull request #14090 from marefr/8843_pipeline_doc_count
Fix pipeline aggregations on doc count
2018-12-12 13:21:13 +01:00
fadd816d89 pkg/*: Fix misspell issues
See,
$ find . -type f | xargs misspell -locale US | grep -vi -e vendor -e devenv -e Unknwon -e Destory | grep pkg
./pkg/services/sqlstore/org_test.go:190:15: "frome" is a misspelling of "from"
./pkg/tsdb/elasticsearch/response_parser.go:544:24: "Unkown" is a misspelling of "Unknown"
./pkg/tsdb/opentsdb/opentsdb.go:87:20: "marshalling" is a misspelling of "marshaling"
./pkg/components/dynmap/dynmap_test.go:2:7: "Licence" is a misspelling of "License"
./pkg/components/dynmap/dynmap.go:2:7: "Licence" is a misspelling of "License"
2018-12-04 23:29:34 +01:00
ce74b1ddc2 Requested Backend changes, removed link in popover description for the offset field 2018-11-27 03:42:20 -05:00
f6fc7f7b64 Requested Backend changes, added details to popover description for the offset field 2018-11-26 08:58:25 -05:00
a022284cb0 fix handle of elasticsearch 6.0+ version 2018-11-23 15:24:16 +01:00
9a8ad70013 fix pipeline aggregations on doc count 2018-11-15 19:06:47 +01:00
5803bfd2c7 fix terms agg order deprecation warning on es 6+ 2018-11-02 17:53:47 +01:00