86 Commits

Author SHA1 Message Date
b80fbe03f0 Plugins: Refactor Plugin Management (#40477)
* add core plugin flow

* add instrumentation

* move func

* remove cruft

* support external backend plugins

* refactor + clean up

* remove comments

* refactor loader

* simplify core plugin path arg

* cleanup loggers

* move signature validator to plugins package

* fix sig packaging

* cleanup plugin model

* remove unnecessary plugin field

* add start+stop for pm

* fix failures

* add decommissioned state

* export fields just to get things flowing

* fix comments

* set static routes

* make image loading idempotent

* merge with backend plugin manager

* re-use funcs

* reorder imports + remove unnecessary interface

* add some TODOs + remove unused func

* remove unused instrumentation func

* simplify client usage

* remove import alias

* re-use backendplugin.Plugin interface

* re order funcs

* improve var name

* fix log statements

* refactor data model

* add logic for dupe check during loading

* cleanup state setting

* refactor loader

* cleanup manager interface

* add rendering flow

* refactor loading + init

* add renderer support

* fix renderer plugin

* reformat imports

* track errors

* fix plugin signature inheritance

* name param in interface

* update func comment

* fix func arg name

* introduce class concept

* remove func

* fix external plugin check

* apply changes from pm-experiment

* fix core plugins

* fix imports

* rename interface

* comment API interface

* add support for testdata plugin

* enable alerting + use correct core plugin contracts

* slim manager API

* fix param name

* fix filter

* support static routes

* fix rendering

* tidy rendering

* get tests compiling

* fix install+uninstall

* start finder test

* add finder test coverage

* start loader tests

* add test for core plugins

* load core + bundled test

* add test for nested plugin loading

* add test files

* clean interface + fix registering some core plugins

* refactoring

* reformat and create sub packages

* simplify core plugin init

* fix ctx cancel scenario

* migrate initializer

* remove Init() funcs

* add test starter

* new logger

* flesh out initializer tests

* refactoring

* remove unused svc

* refactor rendering flow

* fixup loader tests

* add enabled helper func

* fix logger name

* fix data fetchers

* fix case where plugin dir doesn't exist

* improve coverage + move dupe checking to loader

* remove noisy debug logs

* register core plugins automagically

* add support for renderer in catalog

* make private func + fix req validation

* use interface

* re-add check for renderer in catalog

* tidy up from moving to auto reg core plugins

* core plugin registrar

* guards

* copy over core plugins for test infra

* all tests green

* renames

* propagate new interfaces

* kill old manager

* get compiling

* tidy up

* update naming

* refactor manager test + cleanup

* add more cases to finder test

* migrate validator to field

* more coverage

* refactor dupe checking

* add test for plugin class

* add coverage for initializer

* split out rendering

* move

* fixup tests

* fix uss test

* fix frontend settings

* fix grafanads test

* add check when checking sig errors

* fix enabled map

* fixup

* allow manual setup of CM

* rename to cloud-monitoring

* remove TODO

* add installer interface for testing

* loader interface returns

* tests passing

* refactor + add more coverage

* support 'stackdriver'

* fix frontend settings loading

* improve naming based on package name

* small tidy

* refactor test

* fix renderer start

* make cloud-monitoring plugin ID clearer

* add plugin update test

* add integration tests

* don't break all if sig can't be calculated

* add root URL check test

* add more signature verification tests

* update DTO name

* update enabled plugins comment

* update comments

* fix linter

* revert fe naming change

* fix errors endpoint

* reset error code field name

* re-order test to help verify

* assert -> require

* pm check

* add missing entry + re-order

* re-check

* dump icon log

* verify manager contents first

* reformat

* apply PR feedback

* apply style changes

* fix one vs all loading err

* improve log output

* only start when no signature error

* move log

* rework plugin update check

* fix test

* fix multi loading from cfg.PluginSettings

* improve log output #2

* add error abstraction to capture errors without registering a plugin

* add debug log

* add unsigned warning

* e2e test attempt

* fix logger

* set home path

* prevent panic

* alternate

* ugh.. fix home path

* return renderer even if not started

* make renderer plugin managed

* add fallback renderer icon, update renderer badge + prevent changes when renderer is installed

* fix icon loading

* rollback renderer changes

* use correct field

* remove unneccessary block

* remove newline

* remove unused func

* fix bundled plugins base + module fields

* remove unused field since refactor

* add authorizer abstraction

* loader only returns plugins expected to run

* fix multi log output
2021-11-01 10:53:33 +01:00
e822c8a24d CloudMonitoring: Migrate to use backend plugin SDK contracts (#38650)
* Use SDK contracts for cloudmonitoring

* Get build running, tests passing and do some refactoring (#38754)

* fix build+tests and refactor

* remove alerting stuff

* remove unused field

* fix plugin fetch

* end to end

* resp rename

* tidy annotations

* reformatting

* update refID

* reformat imports

* fix styling

* clean up unmarshalling

* uncomment + fix tests

* appease linter

* remove spaces

* remove old cruft

* add check for empty queries

* update tests

* remove pm as dep

* adjust proxy route contract

* fix service loading

* use UNIX val

* fix endpoint + resp

* h@ckz for frontend

* fix resp

* fix interval

* always set custom meta

* remove unused param

* fix labels fetch

* fix linter

* fix test + remove unused field

* apply pr feedback

* fix grafana-auto intervals

* fix tests

* resolve conflicts

* fix bad merge

* fix conflicts

* remove bad logger import

Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
Co-authored-by: Will Browne <will.browne@grafana.com>
2021-10-08 14:46:35 +02:00
722c414fef Encryption: Refactor securejsondata.SecureJsonData to stop relying on global functions (#38865)
* Encryption: Add support to encrypt/decrypt sjd

* Add datasources.Service as a proxy to datasources db operations

* Encrypt ds.SecureJsonData before calling SQLStore

* Move ds cache code into ds service

* Fix tlsmanager tests

* Fix pluginproxy tests

* Remove some securejsondata.GetEncryptedJsonData usages

* Add pluginsettings.Service as a proxy for plugin settings db operations

* Add AlertNotificationService as a proxy for alert notification db operations

* Remove some securejsondata.GetEncryptedJsonData usages

* Remove more securejsondata.GetEncryptedJsonData usages

* Fix lint errors

* Minor fixes

* Remove encryption global functions usages from ngalert

* Fix lint errors

* Minor fixes

* Minor fixes

* Remove securejsondata.DecryptedValue usage

* Refactor the refactor

* Remove securejsondata.DecryptedValue usage

* Move securejsondata to migrations package

* Move securejsondata to migrations package

* Minor fix

* Fix integration test

* Fix integration tests

* Undo undesired changes

* Fix tests

* Add context.Context into encryption methods

* Fix tests

* Fix tests

* Fix tests

* Trigger CI

* Fix test

* Add names to params of encryption service interface

* Remove bus from CacheServiceImpl

* Add logging

* Add keys to logger

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>

* Add missing key to logger

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>

* Undo changes in markdown files

* Fix formatting

* Add context to secrets service

* Rename decryptSecureJsonData to decryptSecureJsonDataFn

* Name args in GetDecryptedValueFn

* Add template back to NewAlertmanagerNotifier

* Copy GetDecryptedValueFn to ngalert

* Add logging to pluginsettings

* Fix pluginsettings test

Co-authored-by: Tania B <yalyna.ts@gmail.com>
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
2021-10-07 17:33:50 +03:00
c3ab2fdeb7 Macaron: remove custom Request type (#37874)
* remove macaron.Request, use http.Request instead

* remove com dependency from bindings module

* fix another c.Req.Request
2021-09-01 11:18:30 +02:00
a6b2e1865c Datasource: Improve default timeout settings for HTTP client provider (#36621)
Make sure that default timeout settings are based on configuration
parameters. This now applies for core data sources using old TSDB
contracts and new SDK contracts. Before it was only applied for old TSDB
contracts.
Also moves global setting variables to non-global (setting.Cfg).
2021-07-15 14:30:06 +02:00
60ac54d969 Chore: Refactor OAuth/social package to service (#35403)
* Creating SocialService

* Add GetOAuthProviders as socialService method

* Add OAuthTokenService

* Add GetOAuthHttpClient method to SocialService

* Rename services, access socialMap from GetConnector

* Fix tests by mocking oauthtoken methods

* Move NewAuthService into Init

* Move OAuthService to social pkg

* Refactor OAuthService to OAuthProvider

* Fix nil map error, rename file, simplify tests

* Fix bug for Forward OAuth Identify

* Remove file after rebase
2021-07-07 08:54:17 +02:00
348e76fc8e Datasource: Shared HTTP client provider for core backend data sources and any data source using the data source proxy (#33439)
Uses new httpclient package from grafana-plugin-sdk-go introduced 
via grafana/grafana-plugin-sdk-go#328. 
Replaces the GetHTTPClient, GetTransport, GetTLSConfig methods defined 
on DataSource model.
Longer-term the goal is to migrate core HTTP backend data sources to use the 
SDK contracts and using httpclient.Provider for creating HTTP clients and such.

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2021-05-19 23:53:41 +02:00
81f6c806e1 Azure token provider with support for Managed Identities (#33807)
* Azure token provider

* Configuration for Azure token provider

* Authentication via Azure SDK for Go

* Fix typo

* ConcurrentTokenCache for Azure credentials

* Resolve AAD authority for selected Azure cloud

* Fixes

* Generic AccessToken and fixes

* Tests and wordings

* Tests for getAccessToken

* Tests for getClientSecretCredential

* Tests for token cache
2021-05-14 13:59:07 +02:00
fcd674ec58 Fix deleting prom rules endpoints via ds_proxy (#33491) 2021-04-29 10:20:51 +03:00
dadccdda06 Allows posting to prom rules endpoints via ds_proxy (#32946)
* allows posting to prom rules endpoints via ds_proxy

* prom proxy routes via plugin and fix proxy route matching bug

* bump ci
2021-04-14 13:06:20 -04:00
7a2ab93e7a Prometheus: Allow exemplars endpoint in data source proxy (#32802) 2021-04-08 18:10:59 +02:00
c0edf88f9f Data proxy: Fix encoded characters in URL path should be proxied encoded (#30597)
Fix encoded characters in URL path should be proxied as encoded in the data proxy.

Fixes #26870 
Fixes #31438

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2021-03-17 12:17:41 +01:00
dce67db6ba Prometheus: Use configured HTTP method for /series and /labels endpoints (#31401)
* Run post-friendly request with set method first

* Improve messaging, retry only when post and specific status code

* Add comments

* Fix backend

* Update public/app/plugins/datasource/prometheus/datasource.ts
2021-02-23 16:31:03 +01:00
15a6508d4a trace user login and datasource name instead of id (#29183)
id's are not very helpful when debugging a system.

Signed-off-by: bergquist <carl.bergquist@gmail.com>
2020-11-18 09:21:45 +01:00
e503188b6f Data source proxy: Convert 401 from data source to 400 (#28962)
* Data source proxy: Convert 401 from data source to 400

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-11-13 13:21:43 +01:00
3d3a7cbba8 Chore: Fix staticcheck issues (#28860)
* Chore: Fix issues reported by staticcheck

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Undo changes

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Chore: Fix issues reported by staticcheck

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix test

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix test

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-11-05 13:07:06 +01:00
b3a868169b OAuth: Support Forward OAuth Identity for backend data source plugins (#27055)
Adds support for the Forward OAuth Identity feature in backend data source plugins. 
Earlier this feature has only been supported for non-backend data source plugins.

Fixes #26023

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2020-10-24 01:34:38 +02:00
19caa100dc OAuth: Fix token refresh failure when custom SSL settings are configured for OAuth provider (#27523)
OAuth token refresh fails when custom SSL settings are configured for 
oauth provider. These changes makes sure that custom SSL settings 
are applied for HTTP client before refreshing token.

Fixes #27514
2020-09-11 17:25:03 +02:00
900eb8070e Proxy: Fix updating refresh token in OAuth pass-thru (#26885)
* Handle updating refresh token in oauth pass-thru

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-08-17 18:38:32 +02:00
d4e4cb4c71 Chore: Enable Go linter gocritic (#26224)
* Chore: Enable gocritic linter

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-07-16 14:39:01 +02:00
d352c213b3 API: Recognize MSSQL data source URLs (#25629)
* API: Recognize MSSQL URLs

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Move MSSQL URL validation into mssql package

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-06-17 11:17:11 +02:00
034abaa73a Security: Use Header.Set and Header.Del for X-Grafana-User header (#25495)
This ensures that the X-Grafana-User header can be trusted.
If the configuration enabled the setting of this header, the
server can now trust that X-Grafana-User is set/unset by Grafana.
Before this, an anonymous user could simply set the X-Grafana-User
header themselves (using the developer tool for example)
2020-06-11 21:15:17 +02:00
164242f5fd Data sources: Don't fail if URL doesn't specify protocol (#24497) 2020-05-12 13:04:18 +02:00
52154b465b dsproxy: adds support for url params for plugin routes (#23503)
* dsproxy: adds support for url params for plugin routes

* docs: fixes after review

* pluginproxy: rename Params to URLParams

* Update pkg/plugins/app_plugin.go

Co-Authored-By: Arve Knudsen <arve.knudsen@gmail.com>

* Apply suggestions from code review

Co-Authored-By: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>
Co-Authored-By: Arve Knudsen <arve.knudsen@gmail.com>

* pluginproxy: rename struct

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>
2020-04-24 10:32:13 +02:00
7d88018531 DataSourceProxy: Handle URL parsing error (#23731)
* pluginproxy: Handle URL parsing error
* pkg/api: Validate data source URLs
* pkg/api: Return 400 for URL validation error
2020-04-22 10:30:06 +02:00
b017e437d8 Fix typo in oauth error (#23270)
fixes: #23269
2020-04-02 08:24:59 +02:00
3fdd2648b1 Chore: Avoid aliasing importing models in api package (#22492) 2020-03-04 12:57:20 +01:00
e6cec8dbdc Backend plugins: Prepare and clean request headers before resource calls (#22321)
Moves common request proxy utilities to proxyutil package with
support for removing X-Forwarded-Host, X-Forwarded-Port,
X-Forwarded-Proto headers, setting X-Forwarded-For header
and cleaning Cookie header.
Using the proxyutil package to prepare and clean request
headers before resource calls.

Closes #21512
2020-03-03 11:45:16 +01:00
ff9556229a Data proxy: Log proxy errors using Grafana logger (#22174)
Now any errors logged by http.ReverseProxy are forwarded to 
Grafana's logger and includes more contextual information like
level (error), user id, org id, username, proxy path, referer and 
IP address.

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2020-02-18 13:26:01 +01:00
f56f54b1a3 Auth: Rotate auth tokens at the end of requests (#21347)
By rotating the auth tokens at the end of the request we ensure
that there is minimum delay between a new token being generated
and the client receiving it.
Adds auth token slow load test which uses random latency for all 
tsdb queries..
Cleans up datasource proxy response handling.
DefaultHandler in middleware tests should write a response, the 
responseWriter BeforeFuncs wont get executed unless a response
is written.

Fixes #18644 

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2020-01-15 13:03:12 +01:00
dfa2f3d2fb Chore: Log actual error when oauth pass thru fails (#20419)
Logs the actual error when failing to retreive access token
when OAuth pass true is enabled for a datasource.

Ref: #20407
2019-11-15 17:14:53 +01:00
8cd93f0b88 Datasource: Add custom headers on alerting queries (#19508)
* Add custom headers on alerting queries

Reference issue #15381

Signed-off-by: Martin Schneppenheim <martin.schneppenheim@rewe-digital.com>

* Fix datasource transport tests

* Migrate decrypting header test to models pkg

* Check correct header

* Add HTTP transport test

Fixes #15381
2019-10-11 14:28:52 +02:00
0a2d5e16dd pkg/api: Check errors (#19657)
* pkg/api: Check errors
* pkg/api: Remove unused function HashEmail
2019-10-08 18:57:53 +02:00
151fe240fc OAuth: Fix for wrong user token updated on OAuth refresh in DS proxy (#17541) 2019-06-12 13:47:38 +02:00
2fff8f77dc move log package to /infra (#17023)
ref #14679

Signed-off-by: zhulongcheng <zhulongcheng.me@gmail.com>
2019-05-13 08:45:54 +02:00
e210725d3d DataProxy: Restore Set-Cookie header after proxy request (#16838)
If Grafana rotates the user's auth token during a request to the data 
source proxy it will set the Set-Cookie header with new auth token in 
response before proxying the request to the datasource.
Before this fix the Set-Cookie response header was cleared after the 
proxied request was finished to make sure that proxied datasources 
cannot affect cookies in users browsers. This had the consequence 
of accidentally also clearing the new auth token set in Set-Cookie 
header.
With this fix the original Set-Cookie value in response header is now 
restored after the proxied datasource request is finished. The existing
logic of clearing Set-Cookie response header from proxied request 
have been left intact.

Fixes #16757
2019-05-01 16:32:03 +02:00
66f6e16916 Security: Store datasource passwords encrypted in secureJsonData (#16175)
* Store passwords in secureJsonData

* Revert unnecessary refactors

* Fix for nil jsonSecureData value

* Remove copied encryption code from migration

* Fix wrong field reference

* Remove migration and provisioning changes

* Use password getters in datasource proxy

* Refactor password handling in datasource configs

* Add provisioning warnings

* Update documentation

* Remove migration command, moved to separate PR

* Remove unused code

* Set the upgrade version

* Remove unused code

* Remove double reference
2019-04-15 11:11:17 +02:00
b696492891 Rename dispatched commands to make them easy to grok 2019-03-20 14:32:41 -04:00
3f9a19dcd5 Merge branch 'master' into 12556-oauth-pass-thru 2019-03-14 13:18:00 -04:00
697a87b7b2 Add check so that header is not sent for anonymous users 2019-03-14 16:33:21 +01:00
bbdc1c0e64 Add custom header with grafana user and a config switch for it 2019-03-14 16:33:19 +01:00
8d8119aa98 Change import path for social since it has moved 2019-03-13 14:34:55 -04:00
8d19ca005f Merge branch 'master' into 12556-oauth-pass-thru 2019-03-13 14:25:43 -04:00
7e62394d01 Add function in ds_proxy to handle oauthPassThru headers 2019-03-13 13:45:32 -04:00
3b15e110a5 Get most recent oauth token from db, rather than lookup by auth_module 2019-03-13 13:22:22 -04:00
a1cd550df4 revert ds_proxy timeout and implement dataproxy timeout correctly 2019-02-11 13:42:05 +01:00
5a59cdf0ef Add oauth pass-thru option for datasources 2019-02-01 19:40:57 -05:00
8261613b51 pkg/util/{ip.go,url.go}: Fix some golint issues
See,
$ gometalinter --vendor --deadline 10m --disable-all --enable=golint  ./...
ip.go:8:6⚠️ func SplitIpPort should be SplitIPPort (golint)
url.go:14:6⚠️ func NewUrlQueryReader should be NewURLQueryReader (golint)
url.go:9:6⚠️ type UrlQueryReader should be URLQueryReader (golint)
url.go:37:6⚠️ func JoinUrlFragments should be JoinURLFragments (golint)
2019-01-28 22:29:20 +01:00
9108fd1b9d add global datasource proxy timeout setting
closes grafana#5699
2019-01-24 14:06:48 -05:00
62417ca69f Remove Origin and Referer headers while proxying requests
Fix #13949
Fix #13328

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2018-11-08 14:57:47 +01:00