283 Commits

Author SHA1 Message Date
e19f36649f Storage: Content root storage (#54929)
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2022-09-12 18:34:46 -04:00
b287047052 Chore: Upgrade Go to 1.19.1 (#54902)
* WIP

* Set public_suffix to a pre Ruby 2.6 version

* we don't need to install python

* Stretch->Buster

* Bump versions in lib.star

* Manually update linter

Sort of messy, but the .mod-file need to contain all dependencies that
use 1.16+ features, otherwise they're assumed to be compiled with
-lang=go1.16 and cannot access generics et al.

Bingo doesn't seem to understand that, but it's possible to manually
update things to get Bingo happy.

* undo reformatting

* Various lint improvements

* More from the linter

* goimports -w ./pkg/

* Disable gocritic

* Add/modify linter exceptions

* lint + flatten nested list

Go 1.19 doesn't support nested lists, and there wasn't an obvious workaround.
https://go.dev/doc/comment#lists
2022-09-12 12:03:49 +02:00
78978048c3 Instrumentation: log the total number of db queries per request (#54647)
Signed-off-by: bergquist <carl.bergquist@gmail.com>
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
Co-authored-by: Dave Henderson <dave.henderson@grafana.com>
2022-09-05 07:39:22 +02:00
7a340f486b Storage: add WithContents option to storage.Get() (#53105)
* Storage: add `WithContents` option to `storage.Get()`

* fix tests

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2022-08-30 15:23:16 +02:00
fe062f2eaa Chore: Use db.DB interface instead of sqlstore (#54358)
* use db.DB interface instead of sqlstore

* make store service depends on db.DB instead of sqlstore
2022-08-26 19:07:58 -05:00
277ea836b6 Secrets: Implement migration of secrets from plugin back to unified secrets (#53561)
* initial cut at migration from plugin

* create new migration from plugin

* only migrate to or from, not both

* remove cfg check from plugin migration itself

* update comments, clean up secret after migration

* add better error handling

* hook up REST API with migrations

* Minor fixes

* fix wire injection issue

* modify migrator to access plugin calls directly. create unit tests

* change pre-migration checks in admin api

* stop plugin after migrating from it

* fix compile issues after merge

* add comment about migration

* fix linting issue

* bleh, fix unit test

* fix another unit test

* update plugin error fatal flag after a migration from the plugin

* add extra logging to migration

* make linter happy

Co-authored-by: Leandro Deveikis <leandro.deveikis@gmail.com>
2022-08-24 16:24:50 -04:00
4d4ecd7fec fix the event attributes (#54117) 2022-08-24 09:59:06 -05:00
806fb8ab7a Tracing: Fix OpenTelemetry Jaeger context propagation (#53269)
* fix otel jaeger context propagation

* add back launch.json

* add back launch.json
2022-08-11 16:11:34 +02:00
a14621fff6 Chore: Add user service method SetUsingOrg and GetSignedInUserWithCacheCtx (#53343)
* Chore: Add user service method SetUsingOrg

* Chore: Add user service method GetSignedInUserWithCacheCtx

* Use method GetSignedInUserWithCacheCtx from user service

* Fix lint after rebase

* Fix lint

* Fix lint error

* roll back some changes

* Roll back changes in api and middleware

* Add xorm tags to SignedInUser ID fields
2022-08-11 13:28:55 +02:00
Jo
ca72cd570e Remove ioutil.ReadDir from usage (#53550)
* add depguard rule for ioutil

* replace ioutil.ReadDir with os.ReadDir

* use legacy option in depguard supported in golangci-lint v1.40

* replace ioutil.ReadDir with os.ReadDir

* return error for file info
2022-08-11 07:21:12 -04:00
Jo
062d255124 Handle ioutil deprecations (#53526)
* replace ioutil.ReadFile -> os.ReadFile

* replace ioutil.ReadAll -> io.ReadAll

* replace ioutil.TempFile -> os.CreateTemp

* replace ioutil.NopCloser -> io.NopCloser

* replace ioutil.WriteFile -> os.WriteFile

* replace ioutil.TempDir -> os.MkdirTemp

* replace ioutil.Discard -> io.Discard
2022-08-10 15:37:51 +02:00
07681a737d Add LockExecuteAndRelease method to ServerLockService (#53417) 2022-08-10 10:28:25 -03:00
6e4b537ba6 Chore: Remove datasources from the Store interface (#53515) 2022-08-10 15:04:51 +02:00
6afad51761 Move SignedInUser to user service and RoleType and Roles to org (#53445)
* Move SignedInUser to user service and RoleType and Roles to org

* Use go naming convention for roles

* Fix some imports and leftovers

* Fix ldap debug test

* Fix lint

* Fix lint 2

* Fix lint 3

* Fix type and not needed conversion

* Clean up messages in api tests

* Clean up api tests 2
2022-08-10 11:56:48 +02:00
119ce90234 Allow empty value in kv_store (#53416) 2022-08-10 10:48:14 +02:00
046a2602ff Storage: optimize sql list operation (#53009) 2022-07-30 00:19:28 +04:00
b5d57c45e3 Plugins: Add support for HTTP logger (#46578) 2022-07-21 09:46:47 -04:00
e1785f4eb4 Secret migration from Sql KV Store to Secret Plugin (#52191)
* Created PluginSecretMigrationService to be able to migrate from the secrets table from the database to the secret plugin. Added migration which takes all the secrets at the sql store and stores it in the plugin. Then deletes all the secrets from the sql
* Added secretsKVStoreSQL.GetAll() method to return all the secrets at the sql table
* Renaming kvstore_test.go as sql_test.go, adding GetAll test case. Fixing decryption of keys
2022-07-19 14:42:23 -03:00
e0a58300ac Public Dashboards: count public dashboards view requests (#52419) 2022-07-19 11:50:37 -03:00
1d2aa7c69b Storage: add delete / deleteFolder / createFolder (#51887)
* delete / delete folder / create folder

* add backend tests

* implement force delete

* fix merge

* lint fix

* fix delete root folder

* fix folder name validation

* fix mysql path_hash issue

* Fix returning error
2022-07-08 22:23:16 +04:00
56f3f3fe69 Chore: Remove the old log format (#51526)
* remove the old log format

* fix CI

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
2022-07-08 09:56:30 -05:00
e96f67ae2e Renderer: Add sanitize API (#50936)
* svg fun

* #50597: add proto

* #50597: add sanitizer methods

* #50597: add provider

* #50597: use sanitizer

* #50597: use sanitizer

* update grafana to match new api

* add comments

* add capability check

* add timing

* update sanitize path

* improve log message

* strings.HasPrefix rather than filepath.IsAbs

* filepath.Clean + filepath.ToSlash for windows

* read 404

* remove `path.clean` from `getPathAndScope`

* add resp body close

* remove unneeded prop

* Update pkg/services/rendering/rendering.go

Co-authored-by: Agnès Toulet <35176601+AgnesToulet@users.noreply.github.com>

* remove test files

* filepath.ToSlash correct wrapping

* filepath.ToSlash correct wrapping

* filepath.ToSlash comment

* compilation error

* lint fix

* fix error message

* Update pkg/services/rendering/rendering.go

Co-authored-by: Agnès Toulet <35176601+AgnesToulet@users.noreply.github.com>

* add `image/svg+xml` mime type

* refactored log

* refactored log

Co-authored-by: Agnès Toulet <35176601+AgnesToulet@users.noreply.github.com>
2022-07-07 15:32:18 +04:00
b7e22c37a8 UsageStats: fixed elasticsearch version number to semver (#49054)
* UsageStats: fixed elasticsearch version number

- The version numbering was changed from plain numbers to a semver-ish approach

* added missing version assertion

* adapted tests
2022-07-04 11:08:01 +02:00
739d3469bc Chore: Update ConcreteLogger to implement gokit Logger interface (#51599) 2022-06-29 18:16:10 -04:00
945f015770 backend/datasources: move datasources models into the datasources service package (#51267)
* backend/datasources: move datasources models into the datasources service pkg
2022-06-27 12:23:15 -04:00
4c4d6fd425 PublicDashboards: collect stats for public dashboards (#50553)
* PublicDashboards: collect stats for public dashboards
2022-06-23 12:02:57 -03:00
84e2e1ec8e Log: Fix text logging for unsupported types (#51306)
* Fix text log for unsupported types

* Apply suggestions from code review

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
2022-06-23 13:24:26 +02:00
ecc15a2f71 KVStore: Extend kvstore to retrieve all items (#50848)
* Extend kvstore to retrieve all items

* Fix comment

* Fix tests

* Change test order

* Move test outside to avoid order conditions

* Update Items to GetAll function and return a map

* Add explanation of map result

* Add description comment

Co-authored-by: Tania B <yalyna.ts@gmail.com>
2022-06-23 11:12:07 +02:00
096330c53b datasource metrics: Correct help text for histograms (#50942)
In #50420 a few summary metrics were converted to histograms,
but the Help text in a couple of them still referred to summaries.
This fixes that help text.

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
2022-06-16 14:18:09 +02:00
68691d7775 Convert some metrics to Histograms (#50420)
Because Summary metrics can not be aggreated, convert them to histograms
so that users with HA deployments can use these metrics.
* Convert metrics registration to promauto.
* Improve help text style.

Signed-off-by: SuperQ <superq@gmail.com>
2022-06-15 13:19:43 +02:00
ae9491c3a7 Chore: Make test tracer noop and return no errors (#50797) 2022-06-15 12:40:41 +02:00
97baa6911d Metrics: Expose functions to initialize counters at zero (#50122) 2022-06-13 17:35:10 +02:00
bd35e6917a Chore: Exclude integration tests from running on test-backend step (#50359)
* Chore: Exclude integration tests from running on test-backend step

* Remove -v from go test command

* Add check to skip integration tests before each integration test

* Try to restart pipeline

* Retrying to make pipeline run
2022-06-10 11:46:21 -04:00
99db588919 Data Sources: Add QueryData OAuth & cookie forwarding middleware (#50466) 2022-06-10 08:41:29 -04:00
ef401f5d62 Storage: improve path validation, add tests (#50441)
* improve path validation

* add test case
2022-06-09 21:09:06 +04:00
cddab15769 Chore: Upgrade grafana-plugin-sdk-go to v0.137.0 (#50443)
Co-authored-by: Todd Treece <todd.treece@grafana.com>
2022-06-09 10:10:58 +02:00
31630edf0c Chore: Remove Wrapf (#50128)
* Chore: Remove Wrapf

* Remove all Wrapf refs

* Remove last Wrapf ref

* Fix lint errors

* Remove Wrap and Wrapf definitions

* Remove unnecessary colon
2022-06-06 22:30:31 +02:00
3c3039f5b3 Chore: Remove Wrap (#50048)
* Chore: Remove Wrap and Wrapf

* Fix: Add error check
2022-06-03 09:24:24 +02:00
24c6a73095 Tracing: Deprecate opentracing (#50058)
* Deprecate opentracing

* Fix comment

* Adjust comment

* Fix docs for opentelemetry

* Add deprecated to sample.ini
2022-06-02 14:13:00 +02:00
c63ebc887b Chore: Run integration tests without grabpl (#49448)
* Chore: Run integration tests without grabpl

* Add new step for integration tests in lib.star

* Remove old integration test step from lib.star

* Change drone signature

* Fix: Edit starlark integration step to not affect enterprise

* Remove all build tags & rename starlark integration test step

* Resync .drone.yml with .drone.star

* Fix lint errors

* Fix lint errors

* Fix lint errors

* Fix more lint errors

* Fix another lint error

* Rename integration test step

* Fix last lint error

* Recomment enterprise step

* Remove comment from Makefile

Co-authored-by: Ida Furjesova <ida.furjesova@grafana.com>
2022-06-01 14:55:22 -04:00
9562fb389f Metrics: Remove support for using summaries instead of histogram for HTTP instrumentation (#49985)
Signed-off-by: bergquist <carl.bergquist@gmail.com>
2022-06-01 14:33:59 +02:00
d3ffb9e245 Usage stats: Divide collection into multiple functions to isolate failures (#49928) 2022-06-01 10:27:06 +00:00
0d7a3209e7 etcd: Fix vuln CVE-2018-1098 (#49976)
* Update etcd

* Update go.sum

* Replace etcd with etcd/v3
2022-06-01 09:31:34 +02:00
3ca3a59079 pkg/web: remove dependency injection (#49123)
* pkg/web: store http.Handler internally

* pkg/web: remove injection

Removes any injection code from pkg/web.

It already was no longer functional, as we already only injected into
`http.Handler`, meaning we only inject ctx.Req and ctx.Resp.

Any other types (*Context, *ReqContext) were already accessed using the
http.Request.Context.Value() method.

* *: remove type mappings

Removes any call to the previously removed TypeMapper, as those were
non-functional already.

* pkg/web: remove Context.Invoke

was no longer used outside of pkg/web and also no longer functional
2022-05-24 15:35:08 -04:00
50c2b4682a Chore: Rename integration tests (#49438)
* Chore: Rename integration tests

* Remove one Integration

Co-authored-by: Ida Furjesova <ida.furjesova@grafana.com>
2022-05-24 11:04:03 +02:00
e43879e55d Encryption: Add support for multiple data keys per day (#47765)
* Add database migrations

* Use short uids as data key ids

* Add support for manual data key rotation

* Fix duplicated mutex unlocks

* Fix migration

* Manage current data keys per name

* Adjust key re-encryption and test

* Modify rename column migration for MySQL compatibility

* Refactor secrets manager and data keys cache

* Multiple o11y adjustments

* Fix stats query

* Apply suggestions from code review

Co-authored-by: Tania <yalyna.ts@gmail.com>

* Fix linter

* Docs: Rotate data encryption keys API endpoint

Co-authored-by: Tania <yalyna.ts@gmail.com>
2022-05-23 13:13:55 +02:00
3d5d8c785b pkg/web: restrict handler types (#48495)
Makes `pkg/web` only accept handles from the following set:

```go
	handlerStd       = func(http.ResponseWriter, *http.Request)
	handlerStdCtx    = func(http.ResponseWriter, *http.Request, *web.Context)
	handlerStdReqCtx = func(http.ResponseWriter, *http.Request, *models.ReqContext)
	handlerReqCtx    = func(*models.ReqContext)
	handlerReqCtxRes = func(*models.ReqContext) Response
	handlerCtx       = func(*web.Context)
```

This is a first step to reducing above set to only `http.Handler`.

---

Due to a cyclic import situation between `pkg/models` and `pkg/web`, parts of this PR were put into `pkg/api/response`, even though they definitely do not belong there. This however is _temporary_ until we untangle `models.ReqContext`.
2022-05-20 12:45:18 -04:00
aad2983350 Security: fixes CVE-2022-29170 (#49240)
* Request interceptor: block redirects

* handle location missing

* Update pkg/infra/httpclient/httpclientprovider/host_redirect_validation_middleware.go

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* Update pkg/infra/httpclient/httpclientprovider/host_redirect_validation_middleware.go

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* linter

* fixes tests

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2022-05-19 17:01:09 +02:00
4e3f53f106 Chore: Make logger mutable for regression test (#49146)
* Log: make logger mutable

* remove flag from public configuration
2022-05-18 17:04:52 +02:00
5c321599c8 Storage: enable SQL backend (#48095)
* #45498: add String util to ListResponse for better UX

* #45498: refactor db_filestorage FS API backend - use path_hash in DB schema

* #45498: enable DB backend fs api tests

* #45498: add comment

* #45498: enable Storage feature flag during integration tests

* remove fmt.println

* #45498: reduce sizes of hash columns

* separate conditions

* #45498: make it easy to ignore backends when running fs api integration tests

* #45498: quote `key` column name

* #45498: reduce path_hash size

* #45498: verify `/{orgId}/{storageName}/` prefix convention in integration tests

* #45498: add etag to the sql table

* #45498: add etag to the sql table

* remove feature flag check (storage isn't dev-mode only)

* add cacheControl and content disposition

* add comments

* add path_hash comment

* explicitly set `path` column collation in `file` table for postgres
2022-05-16 10:26:40 -07:00