167 Commits

Author SHA1 Message Date
f4ee58db50 unified-storage: split resource index client (#106297)
* split resource server and index server grpc connection if defined in config
2025-06-27 08:15:52 -04:00
0982cfd9a0 Unified Storage/Search: Add max count config for indexing (#107255)
* Add max count config for indexing
* Build empty index when max count is exceeded
* Address linting
* Refactor buildIndexes
* Add test for max count threshold
* Update test doc comments
* Refactor TestBuildIndexes_MaxCountThreshold to not use mock framework
* Rename mocks used in TestBuildIndexes_MaxCountThreshold

* Refactor mockResourceIndex

* Test setting of indexing threshold configs

* Tweak comments, log

* Fix logging in buildEmptyIndex

* Export and reuse TestDocumentBuilderSupplier

* Reuse MockResourceIndex
2025-06-27 14:00:39 +02:00
5f21f320f7 fix(unified-storage): use continue token containing both formats for dualwriter (#106525) 2025-06-13 15:59:46 +02:00
7864c1660f K8s: Get trash fixes (#106411) 2025-06-11 13:54:38 -05:00
1196d66c5e feat(unified-storage): log continue token on error (#106475) 2025-06-10 16:41:39 +02:00
04d39cbbc6 unistore: split ListIterator and ListHistory in StorageBackend (#105654)
split listIterator from ListHistory
2025-05-23 15:00:18 +02:00
392c1a71c9 Storage: Take migration_locking setting into account (#105938) 2025-05-23 14:11:21 +02:00
c4d3eb1cd0 Remove support for Google Spanner database. (#105846)
* Remove support for Google Spanner database.
2025-05-23 11:35:59 +02:00
a13fe75de8 refactor(unified-storage): set the GUID in the resource server (#105683) 2025-05-21 09:49:49 +02:00
a7922912fe SecretsManager: Introduce secrets database wrapper (#105472)
SecretsManager: Introduce secret database wrapper

Co-authored-by: PoorlyDefinedBehaviour <brunotj2015@hotmail.com>
Co-authored-by: Leandro Deveikis <leandro.deveikis@gmail.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-05-20 11:48:47 +01:00
91882cf50e chore(unified-storage): add tracing for unified storage folder implementation (#105601) 2025-05-19 23:25:08 +03:00
bdae4424e8 unified-storage: setup distributor module option A (#104737)
* setup distributor module

* move lifecycler into resource server provider

* remove ring/client pool setup from distributor module and use the same ring/client pool between storage server module and distributor module

* implement resourcestore server methods

* make healthcheck fail if ring is not running
2025-05-19 07:46:03 -04:00
5d8f2e0ec3 refactor(unified-storage): extract list iterator and add tests (#105600) 2025-05-19 13:05:49 +02:00
1ffe653156 Revert change of order of checks when setting up resource DB (#105514)
* Revert change of order in newResourceDBProvider

* Ignore error in provisioningSecrets when working with wrong DB.

* Update workspace.

* Add owner.
2025-05-16 11:35:36 +03:00
002f46736a refactor(unified-storage): move generated protos to own pkg (#105356) 2025-05-15 21:36:52 +02:00
7268d6c1bc admin api: Fix reencryption of private_key in signing_key table, introduce test for reencryption (#105010)
* Add integration test for reencrypting secrets.
Rotate keys before reencryption.
Test for setting table from enterprise repo.
2025-05-15 09:51:25 +02:00
e162c69c34 search: fix document missing at startup (#105198)
* fix document missing at startup

* go-lint

* fix tests

* fix tests

* fix integration tests now that we are storing real values
2025-05-12 07:56:25 +02:00
15b3de5893 search: add index batching (#104163)
* add basic search backend integration tests

* add search backend benchmark

* add benchmark indexServer

* fix

* lint

* add more tests

* lint

* do not use the poller

* batch write

* refactor and add tests

* improvements

* improvements

* cleanup

* only observe index success

* add monitorIndexEvents method

* nit use switch instead of if

* make newIndexQueueProcessor private

* simplify runProcessor

* go lint
2025-05-09 15:36:21 +02:00
694b9dfe50 Chore: Replace xorm.io/xorm imports (#104458)
* replace xorm.io/xorm imports

* replace xorm from other go.mod files

* clean up workspace

* nolint does not make sense anymore as it is not a module

* try if nolint directive helps

* use nolint:all for xorm

* add more nolints

* try to skip xorm in linter config

* exclude xorm differently

* retrigger ci
2025-05-02 17:13:01 +02:00
3f2ad925fb chore(unified-storage): remove debug logging for poller (#104862) 2025-05-02 15:02:17 +03:00
188a02723f fix(unified-storage): only fetch from history table if rv changed (#104740) 2025-04-30 12:29:09 +02:00
4adebd6058 unified-storage: setup ring to shard requests (#103783)
* Updates the instrumentation_server service to use mux instead of the builtin router, and have it store the router in the module server: this is so we can register the /ring endpoint to check the status of the ring
* Create a new Ring service that depends on the instrumentation server and declares it as a dependency for the storage server
* Create standalone MemberlistKV service for Ring service to use
* Update the storage server Search and GetStats handler to distribute requests if applicable
2025-04-25 13:08:44 -04:00
fc9f32a9f6 SQLTemplates: Add helper to ensure all templates have a test-case (#103964)
* SQLTemplates: Add helper to ensure all templates have a test-case associated

* UnifiedStorage: Add missing sql template test case

* LegacyDashboards: Add sql templates fs to test cases for exhaustiveness check

* RBACStore: Add sql templates fs to test cases for exhaustiveness check

* LegacyIAM: Add missing sql template test cases
2025-04-22 11:21:51 +02:00
3380ea441a feat(unified-storage): return guid, group and resource on read/list (#104121) 2025-04-17 12:58:58 +02:00
e69052a417 unistore: add check when update the folder of a resource (#102699)
* Add check for move folder

* make the server test generic

* address comment
2025-04-14 16:57:40 +02:00
1125f36f4f fix(test): Attempt to make TestPollingNotifier/notify_returns_channel_and_starts_polling less flaky (#103947)
Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
2025-04-14 09:41:09 +02:00
757be6365a CI: Bump golangci-lint to 2.0.2 (#103572) 2025-04-10 14:42:23 +02:00
a9ef8bcced Use logging from app sdk in the resource mod (#103281)
* Use logging from app sdk in the resource mod

* make update-workspace

* Use app-sdk logging in SQL backend

* Use grafana-app logging in tests

* make update-workspace

* make update-workspace

* make update-workspace

* Use default logging

* Remove dependency on grafana/grafana

* Fix imports
2025-04-08 15:35:11 +02:00
56b4e5670d fix(unified-storage): fix graceful termination for grafana target servers (#103520) 2025-04-08 11:45:10 +02:00
e9ed7223a6 Use authlib repo. Use otel (#103178)
* Use authlib repo. Use otel

* Use interceptors on the provider level

* Create a new wire set with otel

* Lint

* Fix test

* make update-workflow

* make update-workspace

* make update-workspace. Try to add authlib as enterprise imports

* make update-workspace
2025-04-07 15:47:40 +02:00
c63cfc11e2 [Chore] Mark CodeQL warning as safe (#101365)
* Mark CodeQL warning as safe

* Temporarily ignore file
2025-04-07 11:52:09 +02:00
9c49c601f1 Unistore: Add generation column (#102222) 2025-04-01 20:38:23 +03:00
f0a6327edc Unified Storage: Don't read before create (#102906)
* Unified Storage: Don't read before create

* test: use the existing test infra

* fix: support pq

We use pgx, but it seems to be wrapped in a pq driver shim, causing the errors to be remapped to pq's type. Weird
situation.

* feat: support CDK backend

* revert: there is a postgres_tests block

* fix(CDK): only check existence on ADDED updates

* fix(CDK): use ReadResource to deal with deleted files
2025-03-31 15:06:31 +02:00
fe1a72e41b fix(unified-storage): unlist the deleted resource's versions (#102588) 2025-03-27 18:34:37 +01:00
a4e8bd16de fix: add index for resource history get with trash on (#102894) 2025-03-26 15:52:13 +01:00
f7b9f1ce69 Unified Storage: Return an already exists error (#102857)
* Unified Storage: Return an already exists error

When inserting a resource that already exists (i.e. race condition), we can safely catch UNIQUE constraint violations
and transform them into a `k8s.io/apimachinery/pkg/api/errors` error that stands the test of `errors.IsAlreadyExists`.

* feat: clarify existing conflict error

* chore: make update-workspace

* feat: make new package for backend error

* fix: assign dependency owner

* feat: use dialect for checking error type

* chore: go generate

* revert: to 5af369166d6
2025-03-26 14:44:44 +01:00
cc8b40585b feat(unified-storage): add traces for all the backend sub-paths (#102876) 2025-03-26 12:50:29 +01:00
65d474bfa4 chore(unified-storage): update default pruner history limit (#102799) 2025-03-25 17:00:12 +02:00
d3468105d7 Storage: Fix resource history sorting and pagination in Spanner backend (#102777)
* Align spanner ListHistory implementation with sql one

* Move migration logic

* Move migration code

---------

Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
2025-03-25 15:28:24 +01:00
64e939fd5d Unified Storage: Adds span and debug log for paginated List calls (#101912)
Adds span and debug log for paginated List calls
2025-03-21 14:20:27 -06:00
6d570db312 Storage: Add ascending order support for NotOlderThan queries and introduce ResourceVersionMatch_Unset as default (#102505)
* Add support for ASC ordering and introduce ResourceVersionMatch_Unset as default
Add SortAscending to continue token and add integration test for pagination.

* Change protobuf order

* Make backwards compatible

* Update pkg/storage/unified/sql/backend.go

Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>

---------

Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>
2025-03-21 18:35:32 +02:00
c33a53a47a K8s: Remove restore functionality; can be done with list (#102560) 2025-03-20 16:38:32 -05:00
ce350df79b Storage: Add resource version matching in unified storage API (#102417)
Add NotOlderThan support to getHistory

Add support for Exact

Add tests

Refactor tests

Add error test

Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
2025-03-19 16:16:48 +01:00
4927376f32 unistore: use the same connection string as grafana (#102387) 2025-03-19 13:56:54 +01:00
3e3ab8e064 fix(unified-storage): reduce lock contention when pruning (#102425) 2025-03-19 12:15:04 +01:00
0bafd4e99d unistore: update spanner ddl to include unistore tables (#102198)
* provision unistore tables

* update the tests to use generated namespace

* update ddl

* update ddl

* clean streams

* missing space

* use engine.db.query
2025-03-18 14:37:11 +01:00
08994304d8 fix(unified-storage): check resource_api cfg for isHA (#102283) 2025-03-18 12:33:27 +01:00
1700a8aa9f feat(unified-storage): prune history table based on limits (#101970) 2025-03-17 11:36:38 +01:00
6f12712ad6 Add Spanner dialect to pkg/storage/unified/sql/sqltemplate package. (#102121) 2025-03-13 16:38:05 +01:00
31a371e385 fix(unified-storage): remove bleve index metric from global scope (#101825)
* refactor grafana_index_server_index_size to calculate in a goroutine instead of at scrape time and remove grafana_index_server_indexed_docs metric

* use wire to inject bleve index metrics

* remove sprinkles metrics from bleve index metrics

* log error when trying to calculate file index size and bump interval to 1m instead of 5s
2025-03-13 10:09:38 -04:00