68 Commits

Author SHA1 Message Date
5ce4baf6f5 Correlations: Add CreateCorrelation HTTP API (#51630)
* Correlations: add migration

* Correlations: Add CreateCorrelation API

* Correlations: Make correlations work with provisioning

* Handle version changes

* Fix lining error

* lint fixes

* rebuild betterer results

* add a UID to each correlation

* Fix lint errors

* add docs

* better wording in API docs

* remove leftover comment

* handle ds updates

* Fix error message typo

* add bad data test

* make correlations a separate table

* skip readonly check when provisioning correlations

* delete stale correlations when datasources are deleted

* restore provisioned readonly ds

* publish deletion event with full data

* generate swagger and HTTP API docs

* apply source datasource permission to create correlation API

* Fix tests & lint errors

* ignore empty deletion events

* fix last lint errors

* fix more lint error

* Only publish deletion event if datasource was actually deleted

* delete DS provisioning deletes correlations, added & fixed tests

* Fix unmarshalling tests

* Fix linting errors

* Fix deltion event tests

* fix small linting error

* fix lint errors

* update betterer

* fix test

* make path singular

* Revert "make path singular"

This reverts commit 420c3d315e83ef9a7154766c812a24c7b02fc104.

* add integration tests

* remove unneeded id from correlations table

* update spec

* update leftover references to CorrelationDTO

* fix tests

* cleanup tests

* fix lint error
2022-07-25 15:19:07 +01:00
50ae42130b Alerting: take datasources as external alertmanagers into consideration (#52534) 2022-07-20 16:50:49 +02:00
2d8a91a846 Secrets: Improve unified secrets migration and implement compatibility flag (#50463)
* Implement disableSecretsCompatibility flag

* Allow secret deletion right after migration

* Use dialect.Quote for secure_json_data on secret deletion

Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>

* Set secure_json_data to NULL instead of empty json

* Run toggles_gen_test and use generated flag variable

* Add ID to delete data source secrets command on function call

Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>

* Remove extra query to get datasource on secret deletion

* Fix linting issues with CHANGELOG.md

* Use empty json string when deleting secure json data

* Implement secret migration as a background process

* Refactor secret migration as a background service

* Refactor migration to be inside secret store

* Re-add secret deletion function removed on merge

* Try using transaction to fix db lock during tests

* Disable migration for pipeline debugging

* Try adding sleep to fix database lock

* Remove unecessary time sleep from migration

* Fix merge issue, replace models with datasources

* Try event listener approach

* Fix merge issue, replace models with datasources

* Fix linting issues with unchecked error

* Remove unecessary trainling new line

* Increase wait interval on background secret migration

* Rename secret store migration folder for consistency

* Convert background migration to blocking

* Fix number of arguments on server tests

* Check error value of secret migration provider

* Fix linting issue with method varaible

* Revert unintended change on background services

* Move secret migration service provider to wire.go

* Remove unecessary else from datasource service

* Move transaction inside loop on secret migration

* Remove unecessary GetServices function

* Remove unecessary interface after method removal

* Rename Run to Migrate on secret migration interface

* Rename secret migrations service variable on server

* Use MustBool on datasource secret migration

* Revert changes to GetDataSources

* Implement GetAllDataSources function

* Remove DeleteDataSourceSecrets function

* Move datasource secret migration to datasource service

* Remove unecessary properties from datasource secret migration

* Make DecryptLegacySecrets a private method

* Remove context canceled check on secret migrator

* Log error when fail to unmarshal datasource secret

* Add necessary fields to update command on migration

* Handle high availability on secret migration

* Use kvstore for datasource secret migration status

* Add error check for migration status set on kvstore

* Remove NewSecretMigrationService from server tests

* Use const for strings on datasource secrets migration

* Test all cases for datasources secret migrations

Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2022-07-12 17:27:37 -03:00
945f015770 backend/datasources: move datasources models into the datasources service package (#51267)
* backend/datasources: move datasources models into the datasources service pkg
2022-06-27 12:23:15 -04:00
c043a8818a Secrets: add better error handling for secret plugin failures when updating datasources (#50542)
* Add protobuf config and generated code, and client wrapper

* wire up loading of secretsmanager plugin, using renderer plugin as a model

* update kvstore provider to check if we should use the grpc plugin. return false always in OSS

* add OSS remote plugin check

* refactor wire gen file

* log which secrets manager is being used

* Fix argument types for remote checker

* Turns out if err != nil, then the result is always nil. Return empty values if there is an error.

* remove duplicate import

* ensure atomicity by adding secret management as a step to sql operations and rolling back if necessary

* Update pkg/services/secrets/kvstore/kvstore.go

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* Update pkg/services/secrets/kvstore/kvstore.go

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* refactor RemotePluginCheck interface to just return the Plugin client directly

* rename struct to something less silly

* add special error handling for remote secrets management

* switch to errors.as instead of type inference

* remove unnecessary rollback call

* just declare error once

* refactor .proto file according to prior PR suggestions

* re-generate protobuf files and fix compilation errors

* only wrap (ergo display in the front end) errors that are user friendly from the plugin

* rename error type to suggest user friendly only

* rename plugin functions to be more descriptive

* change delete message name

* Revert "change delete message name"

This reverts commit 8ca978301eea45d3cc6a22cda15b9d099c533955.

* Revert "rename plugin functions to be more descriptive"

This reverts commit 4355c9b9ff95443f74c0b588d1ffeabb544f1a34.

* fix pointer to pointer problem

* change plugin user error to just hold a string

* fix sequencing problem with datasource updates

* clean up some return statements

* need to wrap multiple transactions with the InTransaction() func in order to keep the lock

* make linter happy

* revert input var name

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2022-06-16 12:26:57 -04:00
31630edf0c Chore: Remove Wrapf (#50128)
* Chore: Remove Wrapf

* Remove all Wrapf refs

* Remove last Wrapf ref

* Fix lint errors

* Remove Wrap and Wrapf definitions

* Remove unnecessary colon
2022-06-06 22:30:31 +02:00
36c3398c6d Datasource: Remove support for unencrypted passwords (#49987)
* Datasource: Remove support for unencrypted passwords

* regenerate swagger

* [WIP] Remove references to datasource password and basic auth password fields (#50015)

* try delete moar tings

* delete provisioning stuff

* remove from yaml

* update snapshots

* remove lingering snapshot fields

* fix ds http settings

* Re-generate swagger and fix swagger-api-spec make target

Co-authored-by: Will Browne <will.browne@grafana.com>
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
2022-06-03 17:38:22 +02:00
0e991461b0 backend/sqlstore: move GetDashboardTags to dashboard service (#50091)
+ removing some functions that were already copied to the dashboard service in a previous pr.
2022-06-02 10:00:47 -04:00
fc514738b2 Chore: fix comment (#45593) 2022-02-18 11:46:52 +01:00
47e248ceab AccessControl: Remove permissions on data source delete (#45504)
* AccessControl: Remove permissions on datasource delete

* Ensure legacy behavior is preserved
2022-02-17 14:06:34 +01:00
725dbf8d95 Chore: Add context to datasource service (#42294)
* Add context to datasource service

* Adjust wire for ShouldBeReported method

* Replace inTransactionCtx
2021-11-26 18:10:36 +01:00
e4297006f6 Chore: Propagate context for data source provisioning (#40235)
* context all the things

* apply feedback

* rollback some alerting changes

* rollback some alerting changes #2

* more rollbacks

* more rollbacks #2

* more rollbacks #3

* more rollbacks #4

* fix integration test

* add missing context

* add missing and remove incorrect dispatch
2021-10-18 17:06:19 +02:00
722c414fef Encryption: Refactor securejsondata.SecureJsonData to stop relying on global functions (#38865)
* Encryption: Add support to encrypt/decrypt sjd

* Add datasources.Service as a proxy to datasources db operations

* Encrypt ds.SecureJsonData before calling SQLStore

* Move ds cache code into ds service

* Fix tlsmanager tests

* Fix pluginproxy tests

* Remove some securejsondata.GetEncryptedJsonData usages

* Add pluginsettings.Service as a proxy for plugin settings db operations

* Add AlertNotificationService as a proxy for alert notification db operations

* Remove some securejsondata.GetEncryptedJsonData usages

* Remove more securejsondata.GetEncryptedJsonData usages

* Fix lint errors

* Minor fixes

* Remove encryption global functions usages from ngalert

* Fix lint errors

* Minor fixes

* Minor fixes

* Remove securejsondata.DecryptedValue usage

* Refactor the refactor

* Remove securejsondata.DecryptedValue usage

* Move securejsondata to migrations package

* Move securejsondata to migrations package

* Minor fix

* Fix integration test

* Fix integration tests

* Undo undesired changes

* Fix tests

* Add context.Context into encryption methods

* Fix tests

* Fix tests

* Fix tests

* Trigger CI

* Fix test

* Add names to params of encryption service interface

* Remove bus from CacheServiceImpl

* Add logging

* Add keys to logger

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>

* Add missing key to logger

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>

* Undo changes in markdown files

* Fix formatting

* Add context to secrets service

* Rename decryptSecureJsonData to decryptSecureJsonDataFn

* Name args in GetDecryptedValueFn

* Add template back to NewAlertmanagerNotifier

* Copy GetDecryptedValueFn to ngalert

* Add logging to pluginsettings

* Fix pluginsettings test

Co-authored-by: Tania B <yalyna.ts@gmail.com>
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
2021-10-07 17:33:50 +03:00
785072086c Add event after datasource create (#38467)
* Add event after datasource create
2021-08-26 08:50:05 -05:00
32e11434da Fire event after datasource delete (#38090) 2021-08-20 14:51:31 -06:00
0c2045109e Chore: Collect elasticsearch version usage stats (#31787)
* Chore: Collect elasticsearch version usage stats

* Fix lint error

* use GetDataSources from sqlstore

* Apply review suggestions

* Return error if datasource type is not specified

* Update pkg/services/sqlstore/datasource.go

* fix undefined var
2021-03-17 09:14:53 +00:00
b39df22574 Mssql integrated security (#30369)
* update go-mssqldb package

* mssql: support for Windows integrated security

Adds a dropdown to the config page to choose between
SQL Server auth and Windows auth. If the login/username
is empty then the go-mssqldb driver will use single sign on
(Windows integrated security) if on Windows.

* mssql: update ds - don't ignore the user column
2021-01-20 08:23:09 +01:00
1c1a800bc0 API: Add by UID routes for data sources (#29884)
- also add Get by UID+OrgID to datasource cache
- Refactor backend commands for Delete and Get to be unified
2021-01-13 13:16:27 -05:00
2a9c625c9f Alerting: improve alerting default datasource search when extracting alerts (#29993)
* improve alerting search datasource

* correct the xorm get usage

* adding default datasource unittest
2021-01-07 21:33:17 +01:00
375e8e4fd0 SQLStore: customise the limit of retrieved datasources per organisation (#29358)
* SQLStore: customise the limit of retrieved datasources per organisation

* update all suggestions regarding nil or 0 as default

* Apply suggestions from code review

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>

* correct default.ini description + adding unittest

* Apply suggestions from code review

Co-authored-by: Sofia Papagiannaki <papagian@users.noreply.github.com>

* modify unittest name

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
Co-authored-by: Sofia Papagiannaki <papagian@users.noreply.github.com>
2020-12-28 12:24:42 +01:00
ab078133bf Backend: Remove unused code (#28933)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-11-11 11:38:04 +01:00
b5379c5335 Chore: Fix SQL related Go variable naming (#28887)
* Chore: Fix variable naming

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-11-11 06:21:08 +01:00
45adfe7732 Data sources: Use SQL store directly, not via bus (#27000)
* Server: Make it possible to avoid getting SQL store via bus

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
2020-08-21 09:30:06 +02:00
a2fbffe48a Server: Add health check route (#26999)
* Server: Add health check route

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Server: Remove health check middleware

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-08-18 14:58:08 +02:00
41d432b5ae Chore: Enable whitespace linter (#25903)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-07-06 20:17:28 +02:00
5070f7a75b Chore: Start harmonizing linting with plugin SDK (#25854)
* Chore: Harmonize linting with plugin SDK

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Chore: Fix linting issues

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-06-29 14:08:32 +02:00
e5dd7efdee Provisioning: Allows specifying uid for datasource and use that in derived fields (#23585)
* Add uid to datasource

* Fix uid passing when provisioning

* Better error handling and Uid column type change

* Fix test and strict null error counts

* Add backend tests

* Add tests

* Fix strict null checks

* Update test

* Improve tests

* Update pkg/services/sqlstore/datasource.go

Co-Authored-By: Arve Knudsen <arve.knudsen@gmail.com>

* Variable rename

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-04-20 15:48:38 +02:00
ea35ae4d1f Chore: Update the xorm dependency (#22376)
* Upgrade xorm

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-04-01 15:57:21 +02:00
f9962eabff chore: avoid aliasing imports in services (#22499) 2020-02-29 13:35:15 +01:00
75fa1f0207 Metrics: use consistent naming for exported variables (#18134)
* Metrics: remove unused metrics

Metric `M_Grafana_Version` is not used anywhere, nor the mentioned
`M_Grafana_Build_Version`. Seems to be an artefact?

* Metrics: make the naming consistent

* Metrics: add comments to exported vars

* Metrics: use proper naming

Fixes #18110
2019-07-16 16:58:46 +02:00
66f6e16916 Security: Store datasource passwords encrypted in secureJsonData (#16175)
* Store passwords in secureJsonData

* Revert unnecessary refactors

* Fix for nil jsonSecureData value

* Remove copied encryption code from migration

* Fix wrong field reference

* Remove migration and provisioning changes

* Use password getters in datasource proxy

* Refactor password handling in datasource configs

* Add provisioning warnings

* Update documentation

* Remove migration command, moved to separate PR

* Remove unused code

* Set the upgrade version

* Remove unused code

* Remove double reference
2019-04-15 11:11:17 +02:00
f3753712e0 Revert "Fix Datasource Update to no User/Password" 2019-03-11 07:22:54 +01:00
71bfefa8fb Merge pull request #15608 from Maddin-619/fix-datasource-update
Fix Datasource Update to no User/Password
2019-03-05 14:38:32 +01:00
ae9327ff3a remove UseBool since we use AllCols 2019-03-05 13:41:40 +01:00
d49f0bedd3 fix: prevent datasource json data stored as nil (#15508)
prevent datasource json data stored as nil

closes #14239
2019-03-02 21:18:26 +01:00
8a3a3cccc3 moves metric package to /infra
ref #14679
2019-02-24 21:12:51 +01:00
09cd173e92 updates all cols except created so user and password
of the database can be chaned to no user and password
2019-02-22 14:19:15 +01:00
f51222027d Raise datasources number to 5000 2018-12-18 23:59:14 +08:00
b3c78f1265 wip: data source permissions hooks 2018-10-01 15:38:55 +02:00
1a041a2250 bug: return correct err message
if the sql query failed has is false and the method will return
m.ErrDataSourceNotFound which is incorrect. We now return the
correct error message from the query

ref #10843
2018-02-14 13:32:52 +01:00
ebbfc529e0 datasource as cfg: support globbing 2017-11-15 14:30:32 +01:00
0f136a940c datasource as cfg: enable editable ds's 2017-11-15 14:30:32 +01:00
36676e23c1 datasource as cfg: adds readonly datasources 2017-11-15 14:30:32 +01:00
0f29b8ac83 datasources as cfg: tests for insert/updating datasources 2017-11-15 14:30:32 +01:00
d68bfaa8be datasource as cfg: typo 2017-10-25 12:21:06 +02:00
da6430ac01 datasource as cfg: explain why cmd.version can be higher 2017-10-25 09:39:27 +02:00
89923bf77a datasources: change to optimisic concurrency
prerequisite for #9504
2017-10-23 07:55:16 +02:00
2de94d6548 convert old metrics to prom metrics 2017-09-14 14:26:32 +02:00
4f9fbcc211 dataproxy: added caching of datasources when doing data proxy requests, #9078 2017-08-23 13:31:26 +02:00
359421b5e8 refactor: sqlstore inTransaction consolidated to one version, fixes #8376 2017-05-23 10:56:23 +02:00