70 Commits

Author SHA1 Message Date
722c414fef Encryption: Refactor securejsondata.SecureJsonData to stop relying on global functions (#38865)
* Encryption: Add support to encrypt/decrypt sjd

* Add datasources.Service as a proxy to datasources db operations

* Encrypt ds.SecureJsonData before calling SQLStore

* Move ds cache code into ds service

* Fix tlsmanager tests

* Fix pluginproxy tests

* Remove some securejsondata.GetEncryptedJsonData usages

* Add pluginsettings.Service as a proxy for plugin settings db operations

* Add AlertNotificationService as a proxy for alert notification db operations

* Remove some securejsondata.GetEncryptedJsonData usages

* Remove more securejsondata.GetEncryptedJsonData usages

* Fix lint errors

* Minor fixes

* Remove encryption global functions usages from ngalert

* Fix lint errors

* Minor fixes

* Minor fixes

* Remove securejsondata.DecryptedValue usage

* Refactor the refactor

* Remove securejsondata.DecryptedValue usage

* Move securejsondata to migrations package

* Move securejsondata to migrations package

* Minor fix

* Fix integration test

* Fix integration tests

* Undo undesired changes

* Fix tests

* Add context.Context into encryption methods

* Fix tests

* Fix tests

* Fix tests

* Trigger CI

* Fix test

* Add names to params of encryption service interface

* Remove bus from CacheServiceImpl

* Add logging

* Add keys to logger

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>

* Add missing key to logger

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>

* Undo changes in markdown files

* Fix formatting

* Add context to secrets service

* Rename decryptSecureJsonData to decryptSecureJsonDataFn

* Name args in GetDecryptedValueFn

* Add template back to NewAlertmanagerNotifier

* Copy GetDecryptedValueFn to ngalert

* Add logging to pluginsettings

* Fix pluginsettings test

Co-authored-by: Tania B <yalyna.ts@gmail.com>
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
2021-10-07 17:33:50 +03:00
1c892a2fc4 Postgres/MySQL/MSSQL: Add setting to limit maximum amount of rows processed (#38986)
Adds a new setting dataproxy.row_limit that allows an operator to limit the 
amount of rows being processed/accepted in response to database queries 
originating from SQL data sources.

Closes #38975
Ref #39095

Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
2021-09-13 15:27:51 +02:00
c989f0fbbe SQL: Migrate to use SDK contracts (#36635)
* convert SQLs to use sdk contracts

* make draft

* postgres

* intermedia

* get datasourceinfo filled at the beginning of the service

* move the interval into package because of cyclict  import and fix all postgres tests

* fix mysql test

* fix mssql

* fix the test for pr https://github.com/grafana/grafana/issues/35839

* fix some issue about intervalv2 package

* update sql test

* wire migration for SQLs

* add sqls to the background process

* make it register instead of register and start

* revert formatting

* fix tests

* fix linter

* remove integration test

* Postgres test fix

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2021-09-07 09:35:37 +02:00
73fdd9dc22 [MYSQL] convert explicitely smallint and tinyint (#35897)
* [MYSQL] convert explicitely smallint and tinyint

* clean up code

* fix comments
2021-06-22 00:10:45 +08:00
29be741f26 MySQL Datasource: Add timezone parameter (#27535)
* Add MySQL timezone parameter

* Reword docs/sources/features/datasources/mysql.md

Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>

* update documentation

* fix formatting mistake from merge

* enhance documentation with reason for parameter usage

* enhance documentation with reason for parameter usage

Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>
2021-05-27 10:03:03 +02:00
348e76fc8e Datasource: Shared HTTP client provider for core backend data sources and any data source using the data source proxy (#33439)
Uses new httpclient package from grafana-plugin-sdk-go introduced 
via grafana/grafana-plugin-sdk-go#328. 
Replaces the GetHTTPClient, GetTransport, GetTLSConfig methods defined 
on DataSource model.
Longer-term the goal is to migrate core HTTP backend data sources to use the 
SDK contracts and using httpclient.Provider for creating HTTP clients and such.

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2021-05-19 23:53:41 +02:00
bd66c8dde3 SQL data sources: Convert to return data frames (#32257)
Convert SQL data sources to return data frames.

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Will Browne <will.browne@grafana.com>
Co-authored-by: Hugo Häggmark <hugo.haggmark@gmail.com>
2021-05-05 16:46:07 +02:00
7627b55ef4 TSDB: add deprecation comments to many tsdb structs (#33281) 2021-04-23 03:03:11 +02:00
b79e61656a Introduce TSDB service (#31520)
* Introduce TSDB service

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

Co-authored-by: Erik Sundell <erik.sundell87@gmail.com>
Co-authored-by: Will Browne <will.browne@grafana.com>
Co-authored-by: Torkel Ödegaard <torkel@grafana.org>
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
Co-authored-by: Zoltán Bedi <zoltan.bedi@gmail.com>
2021-03-08 07:02:49 +01:00
9593d57914 Chore: Enable errorlint linter (#29227)
* Enable errorlint linter
* Handle wrapped errors

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
2020-11-19 14:47:17 +01:00
574553ec7b Chore: Fix issues found by staticcheck (#28802)
* Fix linting issues

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-11-05 11:29:39 +01:00
a078e40238 Settings: Rename constants/variables to follow Go naming standards (#28002)
* settings: Rename constants/variables to follow Go naming standards
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-10-02 15:45:45 +02:00
a5d9196a53 Chore/fix lint issues (#27704)
* Chore: Fix linting issues

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-09-22 16:22:19 +02:00
ea35ae4d1f Chore: Update the xorm dependency (#22376)
* Upgrade xorm

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-04-01 15:57:21 +02:00
2ca1cc5645 MySql: Fix password regression in MySQL datasource (#20376) 2019-11-14 12:16:38 +02:00
19dbd27c5c mysql: fix encoding in connection string (#20192) 2019-11-06 10:54:53 +02:00
bb50fc5965 pkg/tsdb: Check errors (#19837)
Check errors in pkg/tsdb.
2019-10-22 16:16:53 +02:00
3de693af49 MySQL: Limit datasource error details returned from the backend (#19373)
Only return certain mysql errors from backend.
The following errors is returned as is from backend:
error code 1064 (parse error)
error code 1054 (bad column/field selected)
error code 1146 (table not exists)
Any other errors is logged and returned as a generic
error.
Restrict use of certain functions:
Do not allow usage of the following in query:
system_user()
session_user()
current_user() or current_user
user()
show grants

Fixes #19360
2019-09-24 20:50:49 +02:00
e5bf3027fc Refactor: Move sql_engine to sub package of tsdb (#18991)
this way importing the tsdb package does not come with xorm dependencies
2019-09-10 15:50:04 -04:00
2fff8f77dc move log package to /infra (#17023)
ref #14679

Signed-off-by: zhulongcheng <zhulongcheng.me@gmail.com>
2019-05-13 08:45:54 +02:00
66f6e16916 Security: Store datasource passwords encrypted in secureJsonData (#16175)
* Store passwords in secureJsonData

* Revert unnecessary refactors

* Fix for nil jsonSecureData value

* Remove copied encryption code from migration

* Fix wrong field reference

* Remove migration and provisioning changes

* Use password getters in datasource proxy

* Refactor password handling in datasource configs

* Add provisioning warnings

* Update documentation

* Remove migration command, moved to separate PR

* Remove unused code

* Set the upgrade version

* Remove unused code

* Remove double reference
2019-04-15 11:11:17 +02:00
09b9b595b2 Add check for Env before log 2019-03-15 11:53:30 +01:00
1bc2a0af70 use unique datasource id when registering mysql tls config 2019-02-08 18:08:07 +01:00
f157c19e16 extract parsing of datasource tls config to method 2019-01-28 19:38:56 +01:00
f31fe495e9 fix go fmt 2019-01-15 13:54:25 +01:00
7db848f153 [Feature request] MySQL SSL CA in datasource connector
https://github.com/grafana/grafana/issues/8570
2019-01-15 13:29:56 +01:00
f640df7a2c simplify code 2018-10-18 01:45:14 +08:00
36dbc5f51c Add socket support for mysql data source 2018-10-17 17:45:06 +08:00
27db454012 mysql: use new sql engine 2018-07-26 18:10:45 +02:00
de0d409a23 Revert "Opportunities to unindent code (unindent)" 2018-05-02 14:06:46 +02:00
893a91af3a Use opportunities to unindent code (unindent)
This commit fixes the following unindent findings:
pkg/api/common.go:102:2: "if x { if y" should be "if x && y"
pkg/components/dynmap/dynmap.go:642:2: invert condition and early return
pkg/components/dynmap/dynmap.go:681:2: invert condition and early return
pkg/components/simplejson/simplejson.go:171:2: "if x { if y" should be "if x && y"
pkg/middleware/dashboard_redirect.go:42:3: invert condition and early return
pkg/tsdb/mssql/mssql.go:301:3: invert condition and early break
pkg/tsdb/mysql/mysql.go:312:3: invert condition and early break
pkg/tsdb/postgres/postgres.go:292:3: invert condition and early break
pkg/tsdb/sql_engine.go:144:2: invert condition and early return
2018-04-28 10:53:16 +02:00
346577b664 mysql: fix value columns conversion to float when using timeseries query 2018-04-24 19:53:06 +02:00
f61e69ce75 Simplify comparison to bool constant (gosimple)
This fixes:
build.go:553:6: should omit comparison to bool constant, can be simplified to !strings.Contains(path, ".sha256") (S1002)
pkg/cmd/grafana-cli/commands/ls_command.go:27:5: should omit comparison to bool constant, can be simplified to !pluginDirInfo.IsDir() (S1002)
pkg/components/dynmap/dynmap_test.go:24:5: should omit comparison to bool constant, can be simplified to !value (S1002)
pkg/components/dynmap/dynmap_test.go:122:14: should omit comparison to bool constant, can be simplified to b (S1002)
pkg/components/dynmap/dynmap_test.go:125:14: should omit comparison to bool constant, can be simplified to !b (S1002)
pkg/components/dynmap/dynmap_test.go:128:14: should omit comparison to bool constant, can be simplified to !b (S1002)
pkg/models/org_user.go:51:5: should omit comparison to bool constant, can be simplified to !(*r).IsValid() (S1002)
pkg/plugins/datasource/wrapper/datasource_plugin_wrapper_test.go:77:12: should omit comparison to bool constant, can be simplified to !haveBool (S1002)
pkg/services/alerting/conditions/evaluator.go:23:9: should omit comparison to bool constant, can be simplified to !reducedValue.Valid (S1002)
pkg/services/alerting/conditions/evaluator.go:48:5: should omit comparison to bool constant, can be simplified to !reducedValue.Valid (S1002)
pkg/services/alerting/conditions/evaluator.go:91:5: should omit comparison to bool constant, can be simplified to !reducedValue.Valid (S1002)
pkg/services/alerting/conditions/query.go:56:6: should omit comparison to bool constant, can be simplified to !reducedValue.Valid (S1002)
pkg/services/alerting/extractor.go:107:20: should omit comparison to bool constant, can be simplified to !enabled.MustBool() (S1002)
pkg/services/alerting/notifiers/telegram.go:222:41: should omit comparison to bool constant, can be simplified to this.UploadImage (S1002)
pkg/services/sqlstore/apikey.go:58:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/apikey.go:72:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/dashboard.go:66:33: should omit comparison to bool constant, can be simplified to !cmd.Overwrite (S1002)
pkg/services/sqlstore/dashboard.go:175:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/dashboard.go:311:13: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/dashboard.go:444:12: should omit comparison to bool constant, can be simplified to !exists (S1002)
pkg/services/sqlstore/dashboard.go:472:12: should omit comparison to bool constant, can be simplified to !exists (S1002)
pkg/services/sqlstore/dashboard.go:554:32: should omit comparison to bool constant, can be simplified to !cmd.Overwrite (S1002)
pkg/services/sqlstore/dashboard_snapshot.go:83:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/plugin_setting.go:39:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/quota.go:34:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/quota.go:111:6: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/quota.go:136:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/quota.go:213:6: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/temp_user.go:129:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/user.go:157:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/user.go:182:5: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/user.go:191:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/user.go:212:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/user.go:307:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/social/generic_oauth.go:185:5: should omit comparison to bool constant, can be simplified to !s.extractToken(&data, token) (S1002)
pkg/tsdb/mssql/mssql.go:148:39: should omit comparison to bool constant, can be simplified to ok (S1002)
pkg/tsdb/mssql/mssql.go:212:6: should omit comparison to bool constant, can be simplified to !query.Model.Get("fillNull").MustBool(false) (S1002)
pkg/tsdb/mssql/mssql.go:247:56: should omit comparison to bool constant, can be simplified to ok (S1002)
pkg/tsdb/mssql/mssql.go:274:7: should omit comparison to bool constant, can be simplified to !exist (S1002)
pkg/tsdb/mssql/mssql.go:282:8: should omit comparison to bool constant, can be simplified to !exist (S1002)
pkg/tsdb/mysql/mysql.go:221:6: should omit comparison to bool constant, can be simplified to !query.Model.Get("fillNull").MustBool(false) (S1002)
pkg/tsdb/mysql/mysql.go:256:56: should omit comparison to bool constant, can be simplified to ok (S1002)
pkg/tsdb/mysql/mysql.go:283:7: should omit comparison to bool constant, can be simplified to !exist (S1002)
pkg/tsdb/mysql/mysql.go:291:8: should omit comparison to bool constant, can be simplified to !exist (S1002)
pkg/tsdb/postgres/postgres.go:134:39: should omit comparison to bool constant, can be simplified to ok (S1002)
pkg/tsdb/postgres/postgres.go:201:6: should omit comparison to bool constant, can be simplified to !query.Model.Get("fillNull").MustBool(false) (S1002)
pkg/tsdb/postgres/postgres.go:236:56: should omit comparison to bool constant, can be simplified to ok (S1002)
pkg/tsdb/postgres/postgres.go:263:7: should omit comparison to bool constant, can be simplified to !exist (S1002)
pkg/tsdb/postgres/postgres.go:271:8: should omit comparison to bool constant, can be simplified to !exist (S1002)
2018-04-16 21:04:57 +02:00
af62646624 mysql: fix precision for time columns in time series query mode 2018-04-10 11:01:43 +02:00
f5654f88e2 mysql: fix precision for the time column in table/annotation query mode
Use the ConvertSqlTimeColumnToEpochMs function to convert any native
datetime data type or epoch time (millisecond precision).
Refactored mysql implementation to make it more similar to postgres
and mssql implementations.
Added $__timeEpoch macro function with same implementation as $__time.
Added possibility to use a time column named time in addition to
the currectly supported time_sec.
Additional tests and update of existing.
Added test dashboard.
2018-03-22 15:40:46 +01:00
0f931f9e5e Merge remote-tracking branch 'upstream/master' into macroengine-interpolate 2018-03-01 18:01:54 +01:00
6d13645206 mysql: update to use ColumnTypes interface in new version
The custom code in vendor is not needed anymore and most of the
mapping code can be replaced.
2018-01-18 11:50:16 +01:00
cb15d9cbdd add missing value fill code to mysql datasource 2017-12-10 19:50:01 +01:00
b86a42fffe pass tsdbQuery to transformToTimeSeries and transformToTable to get
access to selected frontend timerange
2017-12-09 20:35:00 +01:00
5dd9582520 mysql: add data source support for Azure MySql
Fixes #9649
2017-11-21 13:21:19 +01:00
d1c9760fa8 Postgres Data Source (#9475)
* add postgresql datasource

* add rest of files for postgres datasource

* fix timeseries query, remove unused code

* consistent naming, refactoring

* s/mysql/postgres/

* s/mysql/postgres/

* couple more tests

* tests for more datatypes

* fix macros for postgres

* add __timeSec macro

* add frontend for postgres datasource

* adjust documentation

* fix formatting

* add proper plugin description

* merge editor changes from mysql

* port changes from mysql datasource

* set proper defaultQuery for postgres

* add time_sec to timeseries query
accept int for value for timeseries query

* revert allowing time_sec and handle int or float values as unix
timestamp for "time" column

* fix tslint error

* handle decimal values in timeseries query

* allow setting sslmode for postgres datasource

* use type switch for handling data types

* fix value for timeseries query

* refactor timeseries queries to make them more flexible

* remove debug statement from inner loop in type conversion

* use plain for loop in getTypedRowData

* fix timeseries queries

* adjust postgres datasource to tsdb refactoring

* adjust postgres datasource to frontend changes

* update lib/pq to latest version

* move type conversion to getTypedRowData

* handle address types cidr, inet and macaddr

* adjust response parser and docs for annotations

* convert unknown types to string

* add documentation for postgres datasource

* add another example query with metric column

* set more helpful default query

* update help text in query editor

* handle NULL in value column of timeseries query

* add __timeGroup macro

* add test for __timeGroup macro

* document __timeGroup and set proper default query for annotations

* fix typos in docs

* add postgres to list of datasources

* add postgres to builtInPlugins

* mysql: refactoring as prep for merging postgres

Refactors out the initialization of the xorm engine and the query logic
for an sql data source.

* mysql: rename refactoring + test update

* postgres:refactor to use SqlEngine(same as mysql)

Refactored to use a common base class with the MySql data source.

Other changes from the original PR:
- Changed time column to be time_sec to allow other time units in the
future and to be the same as MySQL
- Changed integration test to test the main Query method rather than
the private transformToTable method
- Changed the __timeSec macro name to __timeEpoch
- Renamed PostgresExecutor to PostgresQueryEndpoint

Fixes #9209 (the original PR)

* postgres: encrypt password on config page

With some other cosmetic changes to the config page:
- placeholder texts
- reset button for the password after it has been encrypted.
- default value for the sslmode field.

* postgres: change back col name to time from time_sec

* postgres mysql: remove annotation title

Title has been removed from annotations

* postgres: fix images for docs page

* postgres mysql: fix specs
2017-10-10 15:19:14 +02:00
ed661767f8 follow go idiom and return error as second param 2017-09-21 18:04:16 +02:00
a45e2ec9c3 context is reserved for go's context 2017-09-21 15:23:34 +02:00
16b5b9f6be make ds a param for Query 2017-09-21 15:09:14 +02:00
7f9f388293 rename executor into tsdbqueryendpoint 2017-09-21 15:02:17 +02:00
0229d28d64 remove unused structs 2017-09-21 15:02:17 +02:00
55f1b36e31 refactor response flow 2017-09-21 15:02:17 +02:00
c5400ffe76 mysql: use collation instead of charset (#9156)
- use charset would issue additional 'SET NAMES <VALUE>' queries, set
collation would be better. see: https://github.com/go-sql-driver/mysql#charset
2017-09-04 12:32:07 +02:00
c92317bafa fix: fixed gofmt formating for #9091 2017-08-28 13:29:24 +02:00
3eea5d235d Added import for container/list 2017-08-24 15:42:54 -04:00