144 Commits

Author SHA1 Message Date
bedd0b311a elastic: improve error-messages, first step, tests only (#61847) 2023-01-23 12:46:52 +01:00
b44df6c8d1 ElasticSearch: Improve ES error handling message (#61471)
Look for 'caused_by.reason' in ES error response

When the ES response does not contain `reason` or `root_cause[0].reason`
is empty, there is no information for the user to know what is going
wrong.

An example of the error message after this change:
```
Failed to evaluate queries and expressions: failed to execute query A: Trying to create too many buckets. Must be less than or equal to: [65536] but this number of buckets was exceeded. This limit can be set by changing the [search.max_buckets] cluster level setting.
```

Related to https://github.com/grafana/grafana/issues/61246
2023-01-20 11:46:31 +01:00
57425234c9 elastic: backend: removed unused time-field (#61362)
* elastic: backend: removed unused time-field

* improve comment formatting

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>
2023-01-12 15:52:32 +00:00
256f640e19 Elasticsearch: Refactor processQuery to make it more readable (#61145)
* WIP: Simplify process query logic

* WIP: Simplify process query logic

* Simplify

* fix lint
2023-01-10 10:49:43 +01:00
30424a8618 Elasticsearch: Return error if invalid query (#61044)
* Elasticsearch: Return error if invalid query

* Add test

* Fix query in test
2023-01-06 11:14:17 +01:00
772e8cbf60 Elasticsearch: Use interval provided by data request in backend (#60480)
* Elasticsearch: Remove interval caulation and use interval provises by grafana

* Remove redundant code

* Adjust snapshot tests

* Update test

* Fix lint
2023-01-05 12:26:27 +01:00
12c4098bcd elastic: backend: handle naming bucket_scripts (#60460) 2023-01-04 15:26:57 +01:00
a1ef3d77f2 Elasticsearch: Use constants for default values and types (#60673) 2022-12-22 15:08:15 +01:00
09bb4423d2 Elasticsearch: Fix pipeline aggregation queries in backend to pass frontend tests (#60337)
* Elasticsearch: Fix pipeline aggregation queries in backend

* Update

* Update lint

* Update pkg/tsdb/elasticsearch/time_series_query.go

* Fix lint

* Fix merge
2022-12-16 17:45:43 +01:00
d332dab3ec Elasticsearch: Refactor parse query (#60440)
* Refactor parse query to functions

* Move parsing to new file

* Create empty result variable and use it when returning early

* Fix linting

* Revert "Create empty result variable and use it when returning early"

This reverts commit 36a503f66e52f8213c673972774329a963a78100.
2022-12-16 16:30:36 +01:00
3cfeac9c5a elastic: enabled more tests (#60331) 2022-12-15 07:40:41 -05:00
70e34e7f32 Elasticsearch: Fix defaulting to time field if no field is specified in aggregation (#60238)
* Elasticsearch: Add test for using timeField from datasource when not specified in query

* Add default to timefield

* Merge fix
2022-12-14 14:19:03 +01:00
d3ef86bd90 Elasticsearch: Add query building for log queries (#60182)
* Elasticsearch: Fix ordering in raw_document and add logic for raw_data

* Add comments

* Fix raw data request to use correct timefield

* Fix linting

* Add raw data as metric type

* Fix linting

* Elasticsearch: Add defaults for log query

* Add higlight

* Fix lint

* Add snapshot test

* Implement correct query for logs

* Update

* Adjust naming and comments

* Fix lint

* Remove ifs
2022-12-14 13:56:09 +01:00
5633843f76 Elasticsearch: backend: do not set name for time-fields (#60169)
elastic: backend: improve field-naming
2022-12-13 13:19:03 +01:00
0973501233 Elasticsearch: Add query building for raw data and raw document queries to backend (#59741)
* Elasticsearch: Fix ordering in raw_document and add logic for raw_data

* Add comments

* Fix raw data request to use correct timefield

* Fix linting

* Add raw data as metric type

* Fix linting

* Hopefully fix lint
2022-12-13 13:11:44 +01:00
3172d8dd9b elastic: response parser: simplify code (#59782)
elastic: simplify code
2022-12-12 16:00:15 +01:00
13acddd605 Elasticsearch: Fix adding correct min_doc_count to all aggregation types in backend implementation (#59713)
* Elasticsearch: Fix adding correct min_doc_count to all aggregation types in backend implementation

* Update snapshot tests

* Fix linting
2022-12-07 09:47:31 +01:00
e356a56741 Elasticsearch: Fix removing of empty settings from query in backend implementation (#59704)
* Elasticsearch: Fix removing of empty settings from query in backend implementation

* Update

* Update

* Update pkg/tsdb/elasticsearch/time_series_query.go

Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>

Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>
2022-12-05 16:06:55 +01:00
b8b2de1ac4 elastic: remove unused debug-code (#59712)
* elastic: remove unused debug-code

* removed unused structs
2022-12-05 04:21:15 -05:00
a2ed586576 Elasticsearch: Migrate frontend query builder tests to backend (#59578)
* WIP: Migrated first batch of tests

* Migrate more tests

* Migrate rest of the tests

* Update comments

* Update

* Update

* Remove newline
2022-12-01 12:09:12 +01:00
51d7231595 elastic: added more backend tests, copied from the frontend code (#59423) 2022-12-01 04:06:39 -05:00
f1dfaa784a elastic: add more backend tests (#59312)
* elastic: added backend snapshot tests

* linter fixes
2022-11-28 14:59:57 +01:00
fa5a304de8 Elasticsearch: fix typo (#58519) 2022-11-09 14:10:47 +00:00
7996a1c8ad Chore: Update grafana-plugin-sdk-go to v0.142.0 (#58115)
* Chore: Update grafana-plugin-sdk-go to v0.142.0

* Update tests and golden files for 207 status code

* Chore: Move update flag definition at the top in response_parser_test.go

* retrigger

Co-authored-by: Will Browne <will.browne@grafana.com>
2022-11-03 18:34:27 +01:00
d9c40ca41e Chore: Update sqleng, elasticsearch, tempo and opentsdb plugins to support contextual logs. (#57777)
* make sql engine use pick log context for logs
* update tempo to get log context
* update opentsdb to use log context
* update es client to use log context
2022-11-02 10:03:50 -04:00
0491c19712 elastic: backend-mode: fix trimEdges functionality (#56985) 2022-10-19 10:40:42 +02:00
e4f2006cce Elasticsearch: Fix calculation of trimEdges (#56148)
* Elasticsearch: Fix calculation of trimEdges

When a value of trimEdges is set greater than 1 we need to drop both the
first and last sample of the data from the response.

* Elasticsearch: Fix reading trimEdges from the query settings

Currently the trimEdges property in the JSON panel is stored as a string
and not directly as a number.

This caused that the reading of the value failed in the go backend
because the simplejson.Int() method doesn't properly handle this case.
This failure when decoding the value goes unnoticed because of the early
return causing the trimEdges configuration to be ignored.

* Refactor castToInt to also return an error

Add a new test case that sets the `trimEdges` property as a quoted
number.
2022-10-14 12:53:12 +02:00
2efd7fa481 Elasticsearch: Reuse http client in the backend (#55172)
* elastic: backend: reuse http client

* fixed lint error
2022-09-26 14:27:46 +02:00
6dcc8534f7 elastic: simplify unit test (#55168) 2022-09-14 08:59:35 -04:00
a58edc9f5e elastic: always use millisecond-intervals for alerting (#54157) 2022-08-26 14:40:28 +02:00
35253a909d Elasticsearch: remove already-disabled code (#52932)
* elastic: removed unused code from go-part

* elastic: removed unused code from js-part
2022-08-22 16:25:20 +02:00
Jo
062d255124 Handle ioutil deprecations (#53526)
* replace ioutil.ReadFile -> os.ReadFile

* replace ioutil.ReadAll -> io.ReadAll

* replace ioutil.TempFile -> os.CreateTemp

* replace ioutil.NopCloser -> io.NopCloser

* replace ioutil.WriteFile -> os.WriteFile

* replace ioutil.TempDir -> os.MkdirTemp

* replace ioutil.Discard -> io.Discard
2022-08-10 15:37:51 +02:00
a2eb4e85e5 elasticsearch: always use fixed_interval (#50297) 2022-06-13 10:28:29 +02:00
250b72cc1b Elasticsearch: Remove support for versions after their end of the life (<7.10.0) (#48715)
* Elasticsearch: Remove support for versions after their EOL

* Update docs

* Remove old versions from config

* Update pkg/tsdb/elasticsearch/elasticsearch.go

Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>

* Fix tests

* Fix typecheck errors

Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>
2022-05-05 16:16:34 +02:00
8f6877e12a chore: remove all remaining uses of golang.org/x/net/context (#47564)
* chore: remove all remaining uses of  golang.org/x/net/context

This PR finishes the work started in #47532, replacing all calls to that package with the stdlib context and using http.NewRequestWithContext to include the context where necessary.

Bonus: small formatting fixes to goimports in these files.

closes #44178

* tweak: use context.Background in favor of TODO for tests
2022-04-11 14:20:10 -04:00
c0104db72d bugfix: fix proper type of setting maxConcurrentShardRequests (#47120) 2022-04-11 10:29:49 +02:00
7fbc7d019a Plugins: Core plugins register via backend factory provider (#43171)
* refactoring store interface and init flow

* fix import

* fix linter

* refactor resource calling

* load with class

* re-order args

* fix tests

* fix linter

* remove old creator

* add custom config struct

* fix some tests

* cleanup

* fix

* tackle plugins

* fix linter

* refactor and fix test

* add connect failure error

* add fix for azure, cloud monitoring and test data

* restructure

* remove unused err

* add fake tracer for test

* fix grafana ds plugin
2022-01-20 18:16:22 +01:00
9fb8339f87 Elastic: Allow using long/int as date field for alerts (#44027)
* Use integers for time range filter

Previously it was passed as a string which is automatically converted by Elastic to a number only if the field type is "date". For other types (e.g. "long") such conversion doesn't work. In theory "date" could be passed as a formatted string but we don't use it this way and always pass it as a number so it is safe to always pass numbers, not strings.

* Fix time_series_query_test

* Retrigger build
2022-01-17 15:45:09 +01:00
e4ba5f17dd add core plugins thru store (#43085) 2021-12-14 15:22:40 +01:00
1b99d88337 Elasticsearch: Add support for Elasticsearch 8.0 (Beta) (#41729)
* use fixed_interval in date_histogram

* Add 8.0 to available versions in datasource settings

* Remove moving_avg from available metric aggregations

* Add ES8 devenv

* Update public/app/plugins/datasource/elasticsearch/components/QueryEditor/MetricAggregationsEditor/utils.ts

Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>

* Add FE tests

* Add BE test

* fix FE test

* fix BE test

Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>
2021-11-17 20:30:53 +01:00
baab021fec Chore: Refactor usage of legacy data contracts (#41218)
Refactor usage of legacy data contracts. Moves legacy data contracts 
to pkg/tsdb/legacydata package.
Refactor pkg/expr to be a proper service/dependency that can be provided 
to wire to remove some unneeded dependencies to SSE in ngalert and other places.
Refactor pkg/expr to not use the legacydata,RequestHandler and use 
backend.QueryDataHandler instead.
2021-11-10 11:52:16 +01:00
7b7c193551 Elasticsearch: Add time zone setting to Date Histogram aggregation (#40882)
* TimeZonePicker: Allow specifying internal timezones to display

* Elasticsearch: Add time zone setting to Date Histogram aggregation

* rename time_zone to timeZone

* Add tests

* fix typo

* Update packages/grafana-data/src/datetime/timezones.ts

Co-authored-by: Chris Cowan <chris@chriscowan.us>

Co-authored-by: Chris Cowan <chris@chriscowan.us>
2021-11-02 16:18:39 +00:00
b80fbe03f0 Plugins: Refactor Plugin Management (#40477)
* add core plugin flow

* add instrumentation

* move func

* remove cruft

* support external backend plugins

* refactor + clean up

* remove comments

* refactor loader

* simplify core plugin path arg

* cleanup loggers

* move signature validator to plugins package

* fix sig packaging

* cleanup plugin model

* remove unnecessary plugin field

* add start+stop for pm

* fix failures

* add decommissioned state

* export fields just to get things flowing

* fix comments

* set static routes

* make image loading idempotent

* merge with backend plugin manager

* re-use funcs

* reorder imports + remove unnecessary interface

* add some TODOs + remove unused func

* remove unused instrumentation func

* simplify client usage

* remove import alias

* re-use backendplugin.Plugin interface

* re order funcs

* improve var name

* fix log statements

* refactor data model

* add logic for dupe check during loading

* cleanup state setting

* refactor loader

* cleanup manager interface

* add rendering flow

* refactor loading + init

* add renderer support

* fix renderer plugin

* reformat imports

* track errors

* fix plugin signature inheritance

* name param in interface

* update func comment

* fix func arg name

* introduce class concept

* remove func

* fix external plugin check

* apply changes from pm-experiment

* fix core plugins

* fix imports

* rename interface

* comment API interface

* add support for testdata plugin

* enable alerting + use correct core plugin contracts

* slim manager API

* fix param name

* fix filter

* support static routes

* fix rendering

* tidy rendering

* get tests compiling

* fix install+uninstall

* start finder test

* add finder test coverage

* start loader tests

* add test for core plugins

* load core + bundled test

* add test for nested plugin loading

* add test files

* clean interface + fix registering some core plugins

* refactoring

* reformat and create sub packages

* simplify core plugin init

* fix ctx cancel scenario

* migrate initializer

* remove Init() funcs

* add test starter

* new logger

* flesh out initializer tests

* refactoring

* remove unused svc

* refactor rendering flow

* fixup loader tests

* add enabled helper func

* fix logger name

* fix data fetchers

* fix case where plugin dir doesn't exist

* improve coverage + move dupe checking to loader

* remove noisy debug logs

* register core plugins automagically

* add support for renderer in catalog

* make private func + fix req validation

* use interface

* re-add check for renderer in catalog

* tidy up from moving to auto reg core plugins

* core plugin registrar

* guards

* copy over core plugins for test infra

* all tests green

* renames

* propagate new interfaces

* kill old manager

* get compiling

* tidy up

* update naming

* refactor manager test + cleanup

* add more cases to finder test

* migrate validator to field

* more coverage

* refactor dupe checking

* add test for plugin class

* add coverage for initializer

* split out rendering

* move

* fixup tests

* fix uss test

* fix frontend settings

* fix grafanads test

* add check when checking sig errors

* fix enabled map

* fixup

* allow manual setup of CM

* rename to cloud-monitoring

* remove TODO

* add installer interface for testing

* loader interface returns

* tests passing

* refactor + add more coverage

* support 'stackdriver'

* fix frontend settings loading

* improve naming based on package name

* small tidy

* refactor test

* fix renderer start

* make cloud-monitoring plugin ID clearer

* add plugin update test

* add integration tests

* don't break all if sig can't be calculated

* add root URL check test

* add more signature verification tests

* update DTO name

* update enabled plugins comment

* update comments

* fix linter

* revert fe naming change

* fix errors endpoint

* reset error code field name

* re-order test to help verify

* assert -> require

* pm check

* add missing entry + re-order

* re-check

* dump icon log

* verify manager contents first

* reformat

* apply PR feedback

* apply style changes

* fix one vs all loading err

* improve log output

* only start when no signature error

* move log

* rework plugin update check

* fix test

* fix multi loading from cfg.PluginSettings

* improve log output #2

* add error abstraction to capture errors without registering a plugin

* add debug log

* add unsigned warning

* e2e test attempt

* fix logger

* set home path

* prevent panic

* alternate

* ugh.. fix home path

* return renderer even if not started

* make renderer plugin managed

* add fallback renderer icon, update renderer badge + prevent changes when renderer is installed

* fix icon loading

* rollback renderer changes

* use correct field

* remove unneccessary block

* remove newline

* remove unused func

* fix bundled plugins base + module fields

* remove unused field since refactor

* add authorizer abstraction

* loader only returns plugins expected to run

* fix multi log output
2021-11-01 10:53:33 +01:00
76d88a6cec change log level to debug (#41102) 2021-11-01 09:00:00 +00:00
a45861298f Chore: Refactor GoConvey in tsdb/es/client (#40846) 2021-10-26 13:15:47 +02:00
45a844e77e Elasticsearch/Prometheus: Set correct service namespace for SigV4 (#39439)
Makes sure to set the SigV4 service namespace for Elasticsearch and Prometheus.

Fixes #38440
2021-09-27 14:32:19 +02:00
f172701043 IntervalV2: Use maxDataPoints to calculate correct interval (#39036)
* Use max data points to calculate interval

* Fix mockCalculator to mach Calculator

* Fix incorrect merge, replace queryModel with query
2021-09-13 14:58:32 +02:00
dc36f15fbb Prometheus: Revert stepMode functionality (#38982)
* Revert "Prometheus: add functionality to specify desired step interval in dashboards panels (#36422)"

This reverts commit ddf5b65c51a36168e803d471f65c9f9e81abbf57.
Co-authored-by: Ivana Huckova <ivana.huckova@gmail.com>

* Revert "Explore: add functionality for supporting different step modes in prometheus (#37829)"

This reverts commit f433cfd8d9978ee37b0ceeb0b679b335a0acf193.
Co-authored-by: Ivana Huckova <ivana.huckova@gmail.com>

* Revert stepMode BE implementation from #36796
Co-authored-by: "Ivana Huckova" <ivana.huckova@gmail.com>
2021-09-09 14:05:08 +02:00
c989f0fbbe SQL: Migrate to use SDK contracts (#36635)
* convert SQLs to use sdk contracts

* make draft

* postgres

* intermedia

* get datasourceinfo filled at the beginning of the service

* move the interval into package because of cyclict  import and fix all postgres tests

* fix mysql test

* fix mssql

* fix the test for pr https://github.com/grafana/grafana/issues/35839

* fix some issue about intervalv2 package

* update sql test

* wire migration for SQLs

* add sqls to the background process

* make it register instead of register and start

* revert formatting

* fix tests

* fix linter

* remove integration test

* Postgres test fix

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2021-09-07 09:35:37 +02:00
12320dda3c Elasticsearch: Fix metric names for alert queries (#38546)
* Export fieldName to var

* Add empty string as frame name
2021-08-26 09:20:38 +02:00