39 Commits

Author SHA1 Message Date
5d01add7da Fix Postgres query handling null values for smallint (#36648)
* Fix Postgres query handling null values for smallint

* Fix converting to int16
2021-07-13 09:36:24 +02:00
bd66c8dde3 SQL data sources: Convert to return data frames (#32257)
Convert SQL data sources to return data frames.

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Will Browne <will.browne@grafana.com>
Co-authored-by: Hugo Häggmark <hugo.haggmark@gmail.com>
2021-05-05 16:46:07 +02:00
7627b55ef4 TSDB: add deprecation comments to many tsdb structs (#33281) 2021-04-23 03:03:11 +02:00
b79e61656a Introduce TSDB service (#31520)
* Introduce TSDB service

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

Co-authored-by: Erik Sundell <erik.sundell87@gmail.com>
Co-authored-by: Will Browne <will.browne@grafana.com>
Co-authored-by: Torkel Ödegaard <torkel@grafana.org>
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
Co-authored-by: Zoltán Bedi <zoltan.bedi@gmail.com>
2021-03-08 07:02:49 +01:00
c57047a420 Postgres: allow providing TLS/SSL certificates as text in addition to file paths (#30353)
* postgres SSL certification

* add back the UI to configure SSL Authentication files by file path

* add backend logic

* correct unittest

* mini changes

* Update public/app/plugins/datasource/postgres/config_ctrl.ts

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* Update public/app/plugins/datasource/postgres/partials/config.html

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* mutex

* check file exist before remove

* change permission

* change default configuremethod to file-path

* Update public/app/plugins/datasource/postgres/partials/config.html

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* Update public/app/plugins/datasource/postgres/partials/config.html

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* Update public/app/plugins/datasource/postgres/partials/config.html

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* Update public/app/plugins/datasource/postgres/partials/config.html

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* rename sslconfiguremethod to sslconfigurationmethod

* frontend update

* solve comments

* Postgres: Convert tests to stdlib

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Postgres: Be consistent about TLS/SSL terminology

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* fix init inconsistancy

* Fix tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* naming convention

* Fix tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Undo change

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix TLS issue

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* change permissions

* Fix data source field names

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Clean up HTML

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Improve popover text

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix SSL input bug

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Undo unnecessary change

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Clean up backend code

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix build

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* More consistent naming

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Clean up code

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Enforce certificate file permissions

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* add settings

* Undo changes

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* fix windows file path

* PostgresDataSource: Fix mutex usage

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Update pkg/tsdb/postgres/postgres.go

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* Apply suggestions from code review

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* fix compilation

* fix unittest

* Apply suggestions from code review

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* Apply suggestions from code review

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* mock function

* change kmutex package

* add kmutex into middleware

* lock connection file per datasource

* add unittest regarding concurrency

* version should be equal

* adding unittest

* fix the loop

* fix unitest

* fix postgres unittst

* remove comments

* move dataPath from arg to tlsManager struct field

* Use DecryptedValues method

Use cached decrypted values instead of
using secure json data decrypt which will
decrypt unchanged values over and over again.

* remove unneeded mutex in tests and cleanup tests

* fix the lint

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2021-02-23 22:10:55 +01:00
7126a91901 Postgres: Be consistent about TLS/SSL terminology (#30532)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2021-01-22 14:53:37 +01:00
a078e40238 Settings: Rename constants/variables to follow Go naming standards (#28002)
* settings: Rename constants/variables to follow Go naming standards
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-10-02 15:45:45 +02:00
a5d9196a53 Chore/fix lint issues (#27704)
* Chore: Fix linting issues

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-09-22 16:22:19 +02:00
9285595c50 tsdb/postgres: Support Unix socket for host (#25778)
* tsdb/postgres: Support Unix socket for host

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix host regex

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix host regex

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Remove panic

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix host parsing; add tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Add test

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Use strings.HasPrefix to detect Unix socket host

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Remove debug logging

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Escape single quotes and backslashes

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Test generateConnectionString as requested in review

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Add test case for SSL mode

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-07-09 14:57:09 +02:00
1f0e1b33bc Postgres: Add SSL support for datasource (#21341)
* Add SSL certificate/key fields to Postgres data source configuration

Co-authored-by: Edwin Cordeiro <edwin@scordeiro.net>
Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-04-09 12:48:29 +02:00
ea35ae4d1f Chore: Update the xorm dependency (#22376)
* Upgrade xorm

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-04-01 15:57:21 +02:00
3de693af49 MySQL: Limit datasource error details returned from the backend (#19373)
Only return certain mysql errors from backend.
The following errors is returned as is from backend:
error code 1064 (parse error)
error code 1054 (bad column/field selected)
error code 1146 (table not exists)
Any other errors is logged and returned as a generic
error.
Restrict use of certain functions:
Do not allow usage of the following in query:
system_user()
session_user()
current_user() or current_user
user()
show grants

Fixes #19360
2019-09-24 20:50:49 +02:00
e5bf3027fc Refactor: Move sql_engine to sub package of tsdb (#18991)
this way importing the tsdb package does not come with xorm dependencies
2019-09-10 15:50:04 -04:00
2fff8f77dc move log package to /infra (#17023)
ref #14679

Signed-off-by: zhulongcheng <zhulongcheng.me@gmail.com>
2019-05-13 08:45:54 +02:00
66f6e16916 Security: Store datasource passwords encrypted in secureJsonData (#16175)
* Store passwords in secureJsonData

* Revert unnecessary refactors

* Fix for nil jsonSecureData value

* Remove copied encryption code from migration

* Fix wrong field reference

* Remove migration and provisioning changes

* Use password getters in datasource proxy

* Refactor password handling in datasource configs

* Add provisioning warnings

* Update documentation

* Remove migration command, moved to separate PR

* Remove unused code

* Set the upgrade version

* Remove unused code

* Remove double reference
2019-04-15 11:11:17 +02:00
09b9b595b2 Add check for Env before log 2019-03-15 11:53:30 +01:00
3955133f7e Don't pass datasource to newPostgresMacroEngine 2018-08-14 09:15:24 +02:00
3552a4cb86 refactor timescaledb handling in MacroEngine 2018-08-14 08:47:33 +02:00
2f3851b915 postgres: use new sql engine 2018-07-26 18:10:17 +02:00
93e73919e8 fix code style 2018-07-24 11:03:46 +02:00
26aa575cb4 escaping ssl mode on postgres param 2018-07-22 08:04:57 +07:00
de0d409a23 Revert "Opportunities to unindent code (unindent)" 2018-05-02 14:06:46 +02:00
893a91af3a Use opportunities to unindent code (unindent)
This commit fixes the following unindent findings:
pkg/api/common.go:102:2: "if x { if y" should be "if x && y"
pkg/components/dynmap/dynmap.go:642:2: invert condition and early return
pkg/components/dynmap/dynmap.go:681:2: invert condition and early return
pkg/components/simplejson/simplejson.go:171:2: "if x { if y" should be "if x && y"
pkg/middleware/dashboard_redirect.go:42:3: invert condition and early return
pkg/tsdb/mssql/mssql.go:301:3: invert condition and early break
pkg/tsdb/mysql/mysql.go:312:3: invert condition and early break
pkg/tsdb/postgres/postgres.go:292:3: invert condition and early break
pkg/tsdb/sql_engine.go:144:2: invert condition and early return
2018-04-28 10:53:16 +02:00
cf43007531 postgres: fix value columns conversion to float when using timeseries query 2018-04-24 19:53:36 +02:00
f61e69ce75 Simplify comparison to bool constant (gosimple)
This fixes:
build.go:553:6: should omit comparison to bool constant, can be simplified to !strings.Contains(path, ".sha256") (S1002)
pkg/cmd/grafana-cli/commands/ls_command.go:27:5: should omit comparison to bool constant, can be simplified to !pluginDirInfo.IsDir() (S1002)
pkg/components/dynmap/dynmap_test.go:24:5: should omit comparison to bool constant, can be simplified to !value (S1002)
pkg/components/dynmap/dynmap_test.go:122:14: should omit comparison to bool constant, can be simplified to b (S1002)
pkg/components/dynmap/dynmap_test.go:125:14: should omit comparison to bool constant, can be simplified to !b (S1002)
pkg/components/dynmap/dynmap_test.go:128:14: should omit comparison to bool constant, can be simplified to !b (S1002)
pkg/models/org_user.go:51:5: should omit comparison to bool constant, can be simplified to !(*r).IsValid() (S1002)
pkg/plugins/datasource/wrapper/datasource_plugin_wrapper_test.go:77:12: should omit comparison to bool constant, can be simplified to !haveBool (S1002)
pkg/services/alerting/conditions/evaluator.go:23:9: should omit comparison to bool constant, can be simplified to !reducedValue.Valid (S1002)
pkg/services/alerting/conditions/evaluator.go:48:5: should omit comparison to bool constant, can be simplified to !reducedValue.Valid (S1002)
pkg/services/alerting/conditions/evaluator.go:91:5: should omit comparison to bool constant, can be simplified to !reducedValue.Valid (S1002)
pkg/services/alerting/conditions/query.go:56:6: should omit comparison to bool constant, can be simplified to !reducedValue.Valid (S1002)
pkg/services/alerting/extractor.go:107:20: should omit comparison to bool constant, can be simplified to !enabled.MustBool() (S1002)
pkg/services/alerting/notifiers/telegram.go:222:41: should omit comparison to bool constant, can be simplified to this.UploadImage (S1002)
pkg/services/sqlstore/apikey.go:58:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/apikey.go:72:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/dashboard.go:66:33: should omit comparison to bool constant, can be simplified to !cmd.Overwrite (S1002)
pkg/services/sqlstore/dashboard.go:175:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/dashboard.go:311:13: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/dashboard.go:444:12: should omit comparison to bool constant, can be simplified to !exists (S1002)
pkg/services/sqlstore/dashboard.go:472:12: should omit comparison to bool constant, can be simplified to !exists (S1002)
pkg/services/sqlstore/dashboard.go:554:32: should omit comparison to bool constant, can be simplified to !cmd.Overwrite (S1002)
pkg/services/sqlstore/dashboard_snapshot.go:83:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/plugin_setting.go:39:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/quota.go:34:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/quota.go:111:6: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/quota.go:136:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/quota.go:213:6: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/temp_user.go:129:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/user.go:157:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/user.go:182:5: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/user.go:191:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/user.go:212:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/user.go:307:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/social/generic_oauth.go:185:5: should omit comparison to bool constant, can be simplified to !s.extractToken(&data, token) (S1002)
pkg/tsdb/mssql/mssql.go:148:39: should omit comparison to bool constant, can be simplified to ok (S1002)
pkg/tsdb/mssql/mssql.go:212:6: should omit comparison to bool constant, can be simplified to !query.Model.Get("fillNull").MustBool(false) (S1002)
pkg/tsdb/mssql/mssql.go:247:56: should omit comparison to bool constant, can be simplified to ok (S1002)
pkg/tsdb/mssql/mssql.go:274:7: should omit comparison to bool constant, can be simplified to !exist (S1002)
pkg/tsdb/mssql/mssql.go:282:8: should omit comparison to bool constant, can be simplified to !exist (S1002)
pkg/tsdb/mysql/mysql.go:221:6: should omit comparison to bool constant, can be simplified to !query.Model.Get("fillNull").MustBool(false) (S1002)
pkg/tsdb/mysql/mysql.go:256:56: should omit comparison to bool constant, can be simplified to ok (S1002)
pkg/tsdb/mysql/mysql.go:283:7: should omit comparison to bool constant, can be simplified to !exist (S1002)
pkg/tsdb/mysql/mysql.go:291:8: should omit comparison to bool constant, can be simplified to !exist (S1002)
pkg/tsdb/postgres/postgres.go:134:39: should omit comparison to bool constant, can be simplified to ok (S1002)
pkg/tsdb/postgres/postgres.go:201:6: should omit comparison to bool constant, can be simplified to !query.Model.Get("fillNull").MustBool(false) (S1002)
pkg/tsdb/postgres/postgres.go:236:56: should omit comparison to bool constant, can be simplified to ok (S1002)
pkg/tsdb/postgres/postgres.go:263:7: should omit comparison to bool constant, can be simplified to !exist (S1002)
pkg/tsdb/postgres/postgres.go:271:8: should omit comparison to bool constant, can be simplified to !exist (S1002)
2018-04-16 21:04:57 +02:00
1783c534fd postgres: fix precision for time columns in time series query mode 2018-04-10 11:07:01 +02:00
66c03f84f5 postgres: fix precision for the time column in table/annotation query mode
Use the ConvertSqlTimeColumnToEpochMs function to convert any native
datetime data type or epoch time (millisecond precision).
Additional tests and update of existing due to timezone issues
running postgres on UTC and dev environment on non-utc.
Added test dashboard.
2018-03-22 15:32:33 +01:00
4904a051cf use net/url to generate postgres connection url 2018-03-05 16:06:04 +01:00
0f931f9e5e Merge remote-tracking branch 'upstream/master' into macroengine-interpolate 2018-03-01 18:01:54 +01:00
ebbc079853 improve error message for invalid/unknown datatypes (#10834) 2018-02-14 15:21:00 +01:00
6cdd901ec6 fix typo 2017-12-10 12:40:09 +01:00
c1282e8ea8 implement missing value fill functionality for postgres 2017-12-10 11:42:55 +01:00
b86a42fffe pass tsdbQuery to transformToTimeSeries and transformToTable to get
access to selected frontend timerange
2017-12-09 20:35:00 +01:00
c80eadcdf4 handle native postgres datetime types in annotation queries (#9986) 2017-12-06 18:12:24 +01:00
373389c920 treat any text column in timeseries query as metric name unless column (#9985)
named metric is present
2017-12-06 18:04:33 +01:00
26ab25b7c0 dont loose subsecond precision when dealing with timestamp or (#9851)
timestamptz
2017-11-15 14:50:32 +01:00
9e6a7dcb90 properly escape components of connection string (#9850) 2017-11-15 10:55:08 +01:00
a503c1d39c change default sslmode for postgres to verify-full (#9736) 2017-10-31 13:55:32 +01:00
d1c9760fa8 Postgres Data Source (#9475)
* add postgresql datasource

* add rest of files for postgres datasource

* fix timeseries query, remove unused code

* consistent naming, refactoring

* s/mysql/postgres/

* s/mysql/postgres/

* couple more tests

* tests for more datatypes

* fix macros for postgres

* add __timeSec macro

* add frontend for postgres datasource

* adjust documentation

* fix formatting

* add proper plugin description

* merge editor changes from mysql

* port changes from mysql datasource

* set proper defaultQuery for postgres

* add time_sec to timeseries query
accept int for value for timeseries query

* revert allowing time_sec and handle int or float values as unix
timestamp for "time" column

* fix tslint error

* handle decimal values in timeseries query

* allow setting sslmode for postgres datasource

* use type switch for handling data types

* fix value for timeseries query

* refactor timeseries queries to make them more flexible

* remove debug statement from inner loop in type conversion

* use plain for loop in getTypedRowData

* fix timeseries queries

* adjust postgres datasource to tsdb refactoring

* adjust postgres datasource to frontend changes

* update lib/pq to latest version

* move type conversion to getTypedRowData

* handle address types cidr, inet and macaddr

* adjust response parser and docs for annotations

* convert unknown types to string

* add documentation for postgres datasource

* add another example query with metric column

* set more helpful default query

* update help text in query editor

* handle NULL in value column of timeseries query

* add __timeGroup macro

* add test for __timeGroup macro

* document __timeGroup and set proper default query for annotations

* fix typos in docs

* add postgres to list of datasources

* add postgres to builtInPlugins

* mysql: refactoring as prep for merging postgres

Refactors out the initialization of the xorm engine and the query logic
for an sql data source.

* mysql: rename refactoring + test update

* postgres:refactor to use SqlEngine(same as mysql)

Refactored to use a common base class with the MySql data source.

Other changes from the original PR:
- Changed time column to be time_sec to allow other time units in the
future and to be the same as MySQL
- Changed integration test to test the main Query method rather than
the private transformToTable method
- Changed the __timeSec macro name to __timeEpoch
- Renamed PostgresExecutor to PostgresQueryEndpoint

Fixes #9209 (the original PR)

* postgres: encrypt password on config page

With some other cosmetic changes to the config page:
- placeholder texts
- reset button for the password after it has been encrypted.
- default value for the sslmode field.

* postgres: change back col name to time from time_sec

* postgres mysql: remove annotation title

Title has been removed from annotations

* postgres: fix images for docs page

* postgres mysql: fix specs
2017-10-10 15:19:14 +02:00