47 Commits

Author SHA1 Message Date
84a4ed612b fix: re-order db initialization logic for resource server (#108059) 2025-07-14 10:25:42 +00:00
602b7826c4 refactor(unified-storage): remove resource_server config option (#107649) 2025-07-11 15:33:51 +02:00
3f502f305d Chore: Update mocks with recent mockery (#107816) 2025-07-09 09:15:34 +02:00
041c343a86 Unified storage: Respect GF_DATABASE_URL override (#105331)
* Database for unified storage resources now reuses DB code that respects URL override.

Access instrument_queries via section getter.
2025-07-03 09:57:40 +00:00
e076c74869 sqltemplate, dbimpl: Remove single-method function types (#107525)
* Remove dbProviderFunc function.

This removes one extra indirection that made the code bit more difficult to navigate.

* Remove indirection function types implementing single-method interfaces.

This streamlines the code and makes it bit easier to navigate.

* Update pkg/storage/unified/sql/sqltemplate/dialect_mysql.go

Co-authored-by: Mustafa Sencer Özcan <32759850+mustafasencer@users.noreply.github.com>

---------

Co-authored-by: Mustafa Sencer Özcan <32759850+mustafasencer@users.noreply.github.com>
2025-07-03 10:38:12 +02:00
3674d952a6 fix: use database section for resource server db engine configuration (#107495) 2025-07-02 09:00:51 +00:00
9a92900ef3 fix(unified-storage): remove 's' in config parameter (#107457) 2025-07-01 14:06:21 +02:00
d76e55371a fix(unified-storage): use the provided connection config parameters (#107455)
* fix(unified-storage): use the provided connection config parameters

* extend tests

* make update-workspace
2025-07-01 11:25:16 +00:00
392c1a71c9 Storage: Take migration_locking setting into account (#105938) 2025-05-23 14:11:21 +02:00
1ffe653156 Revert change of order of checks when setting up resource DB (#105514)
* Revert change of order in newResourceDBProvider

* Ignore error in provisioningSecrets when working with wrong DB.

* Update workspace.

* Add owner.
2025-05-16 11:35:36 +03:00
7268d6c1bc admin api: Fix reencryption of private_key in signing_key table, introduce test for reencryption (#105010)
* Add integration test for reencrypting secrets.
Rotate keys before reencryption.
Test for setting table from enterprise repo.
2025-05-15 09:51:25 +02:00
694b9dfe50 Chore: Replace xorm.io/xorm imports (#104458)
* replace xorm.io/xorm imports

* replace xorm from other go.mod files

* clean up workspace

* nolint does not make sense anymore as it is not a module

* try if nolint directive helps

* use nolint:all for xorm

* add more nolints

* try to skip xorm in linter config

* exclude xorm differently

* retrigger ci
2025-05-02 17:13:01 +02:00
c63cfc11e2 [Chore] Mark CodeQL warning as safe (#101365)
* Mark CodeQL warning as safe

* Temporarily ignore file
2025-04-07 11:52:09 +02:00
9c49c601f1 Unistore: Add generation column (#102222) 2025-04-01 20:38:23 +03:00
a4e8bd16de fix: add index for resource history get with trash on (#102894) 2025-03-26 15:52:13 +01:00
4927376f32 unistore: use the same connection string as grafana (#102387) 2025-03-19 13:56:54 +01:00
b7a0aeeb0d SCIM: Disable auto assign organization if the user has been provisioned (#101307)
* Add isProvisioned field to model

* Add new isProvisioned column to migration

* Disable auto assignment to organization if the user is provisioned

* add annotation to user model

* add annotation to user models

* Remove IsProvisioned field from Identity

* Move new field assignenment and add default value

* Update annotations for user query results

* Remove isProvisioned from identity

* Add new column to test

* Resolve user from identity at SyncOrgHook
2025-03-03 17:51:23 +01:00
c522a5b13b fix(unified-storage): proper setup TLS in new db_engine for MySQL (#100686) 2025-02-14 16:23:25 +01:00
9d30911107 SQL/Storage: Use xorm code migration for DeletionMarker (#99353) 2025-01-22 14:07:08 +03:00
429da7fd68 SQL/Blob: Add support for blob storage to SQL backend (#98192) 2025-01-08 22:08:10 +02:00
9642eedce6 Unified Storage: Use tls preferred when grafana db using ssl (#97378)
When unified storage is using the grafana db (default case for on-prem), if the db is using ssl then set tls=preferred. This fixes an on-prem bug in 11.3.1.
2024-12-03 23:56:13 +02:00
4f8ab73a8c unistore: Add support for verify-full in postgres (#96825)
Add support for verify-full in postgres
2024-11-21 12:31:33 +00:00
f4e91da6b1 Unistore: fix fodler migration (#96299)
configure default migration for resource folder
2024-11-12 16:06:42 +02:00
c3571752b6 Unitstore: Fix the Folder migration (#96078)
* UniStore: Add folder to the SQL backend

This reverts commit aaa4f4a23e3ba0e02526f8beb05321761382259f.

* fix folder migration
2024-11-12 12:52:04 +01:00
aaa4f4a23e Revert "UniStore: Add folder to the SQL backend" (#95966)
Revert "UniStore: Add folder to the SQL backend (#95671)"

This reverts commit f0756626969f1a2d345874f92463b275c721ab79.
2024-11-06 17:04:57 +02:00
f075662696 UniStore: Add folder to the SQL backend (#95671)
* Add folder to the SQL backend

* remove unused key

* fix tests

* Return folders from watch
2024-11-05 10:37:23 +01:00
76b43267c8 Unified Storage: Use ssl_mode instead of sslmode (#95658)
change unistore to use ssl_mode instead of sslmode
2024-10-31 16:10:47 +01:00
337f2a0a8b [search] add index; increase page size (#95515)
[search] add index; increase page size
2024-10-28 15:48:49 -04:00
2ffb88b0ee Unified Storage: Add index for poller query (#95293)
add index for poller query
2024-10-23 15:40:56 -06:00
3e1f5559a6 Unistore Chore: Add database-level observability (#92266)
* add testing harness

* fix mockery and linters

* WIP

* wip

* fix transactions

* fix transaction tracing; add tracing by default

* rename package

* move WithTx to simplify logic of DB implementations

* fix potential issue with context deadline

* add db instrumentation to dbutil

* add otel tests

* improve naming

* minor fix in semantics and add comprehensive OTel testing

* fix naming

* instrument resourceVersionAtomicInc

* provide a default testing tracer

* fix docs

* fix typo in docs

* add semconv for k8s
2024-10-18 00:32:08 -03:00
cf08f6762d Unistore Chore: Add OTEL testing harness (#94835)
* add testing harness

* fix mockery and linters

* fix data race in tests

* fix data race in tests

* reduce cardinality of data
2024-10-17 08:41:06 -03:00
44282134da Unistore Chore: Make it easier to implement DB interface (#94680)
make it easier to implement DB interface
2024-10-16 13:40:35 -03:00
5c03c14b25 resource-api: Loosen name validation to match K8s requirements (#93404)
* resource-api: Loosen name validation to match K8s requirements

This patch modifies some of the requirements for name validation of
objects in Resource API to match Kubernetes.

The limit we have on characters in name is 64, but some resources allow
upto 253 characters. Similarly we also include `:` in the regex, as many
objects in default K8s setup use it in the name (the group
`system:masters` for example)

Signed-off-by: Prem Kumar <prem.saraswat@grafana.com>

* Update the name column length in migrator and update e2e test to verify

---------

Signed-off-by: Prem Kumar <prem.saraswat@grafana.com>
2024-10-10 11:03:18 -04:00
4a800eda9f Unistore Chore: add inocuous preparative changes for otel-based db observability (#94473)
add inocuous preparative changes for otel tracing
2024-10-09 20:32:09 +00:00
03f55e5288 Unistore : Ensure Watch works in HA mode (#94120)
* Revert "Revert "Unistore : Ensure Watch works in HA mode." (#94097)"

This reverts commit 7c3fc2f2619a0eb2157ea27e7dfef91996a9de7c.

* make previous_resource_version nullable

* handle nil case
2024-10-07 10:01:53 +02:00
7d32d5eff4 Unistore: Reuse MySQL and Postgres Grafana core config instead of the object (#94223)
* Reuse MySQL and Postgres Grafana config instead of the object

- Only reuse the Grafana DB object for SQLite. Support for SQLite will be added in a different PR
- Fail when reusing the Grafana DB object if it is using DB instrumentation
- In the case that we have to reuse a Grafana DB with its instrumentation, fail with an error that describes a workaround
- Add regression tests to reproduce incident 2144

* remove temp file

* fix linter

* fix linter x2

* fix linter x3
2024-10-04 12:07:20 +00:00
7c3fc2f261 Revert "Unistore : Ensure Watch works in HA mode." (#94097)
Revert "Unistore : Ensure Watch works in HA mode. (#93428)"

This reverts commit 0a26c9e9aec15ba0d21a836e1cbc1b03bad4bc34.
2024-10-01 18:45:47 +00:00
0a26c9e9ae Unistore : Ensure Watch works in HA mode. (#93428)
* Replace Watch with WatchNext

* remove watchset

* fix previous page and closing the channel

* Remove the broadcaster cache to prevent dupplicated events

* add watch bookmark

* add watch bookmark

* cleanup comments

* disable the tests for bookmarks for now

* Ensure we send previosu events

* lint

* re-introduce the cache

* load from cache

* disabling legacy test

* disabling legacy test

* Update pkg/storage/unified/resource/server.go

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>

* Could not read previous events

* add proper migration

* Add previous_resource_version to both history and resource

* First event should have an RV of 2 and not 1

* Test both storage backends

* fix the inital RV for the sql backend

* ensure graceful stop of the stream decoder

* gocyclo

---------

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
2024-09-30 13:14:07 +02:00
c59dddf7af MySQL: Add parseTime=true to SQL connections (#92469) 2024-08-27 14:16:04 +03:00
2136fd9a92 Storage: Remove unified storage feature flag (#92192)
remove unified storage flag
2024-08-21 19:28:30 +03:00
9a65d0a977 Storage: remove sqlstore.WrapDatabaseDriverWithHooks (#92111)
remove sqlstore.WrapDatabaseDriverWithHooks from Unified Storage after related incident
2024-08-21 10:31:34 -03:00
1cb3470e3e Unified Storage: Wrapping the mysql driver with hooks causes issues (#92099)
dont wrap the mysql driver with hooks. Its causing the error 'begin tx: sql: driver does not support non-default isolation level'.
2024-08-19 15:09:25 -06:00
28f2a4078d Unified Storage: Records grafana database metrics (#91932)
* records grafana database metrics for unified storage

* update type to tracing.Tracer

* use nil for tracer in tests
2024-08-15 11:13:36 -06:00
160fe2a3a4 Storage: Remove entity store (#91233) 2024-07-31 09:25:39 +03:00
2362e79e4e Unified Storage: Configure client to use tls=preferred (#91162)
* optionally set tls=preferred in the mysql connection string from env var
2024-07-30 11:24:28 -06:00
399d77a0fd Resource server improvements and fixes (#90715)
* cleanup dependencies and improve list method
* Improve Resource Server API, remove unnecessary dependencies
* Reduce the API footprint of ResourceDBInterface and its implementation
* Improve LifecycleHooks to use context
* Improve testing
* reduce API size and improve code
* sqltemplate: add DialectForDriver func and improve naming
* improve lifecycle API
* many small fixes after adding more tests
2024-07-22 20:08:30 +03:00
08c611c68b ResourceServer: Resource store sql backend (#90170) 2024-07-18 08:03:18 -07:00