138 Commits

Author SHA1 Message Date
e9ed7223a6 Use authlib repo. Use otel (#103178)
* Use authlib repo. Use otel

* Use interceptors on the provider level

* Create a new wire set with otel

* Lint

* Fix test

* make update-workflow

* make update-workspace

* make update-workspace. Try to add authlib as enterprise imports

* make update-workspace
2025-04-07 15:47:40 +02:00
c63cfc11e2 [Chore] Mark CodeQL warning as safe (#101365)
* Mark CodeQL warning as safe

* Temporarily ignore file
2025-04-07 11:52:09 +02:00
9c49c601f1 Unistore: Add generation column (#102222) 2025-04-01 20:38:23 +03:00
f0a6327edc Unified Storage: Don't read before create (#102906)
* Unified Storage: Don't read before create

* test: use the existing test infra

* fix: support pq

We use pgx, but it seems to be wrapped in a pq driver shim, causing the errors to be remapped to pq's type. Weird
situation.

* feat: support CDK backend

* revert: there is a postgres_tests block

* fix(CDK): only check existence on ADDED updates

* fix(CDK): use ReadResource to deal with deleted files
2025-03-31 15:06:31 +02:00
fe1a72e41b fix(unified-storage): unlist the deleted resource's versions (#102588) 2025-03-27 18:34:37 +01:00
a4e8bd16de fix: add index for resource history get with trash on (#102894) 2025-03-26 15:52:13 +01:00
f7b9f1ce69 Unified Storage: Return an already exists error (#102857)
* Unified Storage: Return an already exists error

When inserting a resource that already exists (i.e. race condition), we can safely catch UNIQUE constraint violations
and transform them into a `k8s.io/apimachinery/pkg/api/errors` error that stands the test of `errors.IsAlreadyExists`.

* feat: clarify existing conflict error

* chore: make update-workspace

* feat: make new package for backend error

* fix: assign dependency owner

* feat: use dialect for checking error type

* chore: go generate

* revert: to 5af369166d6
2025-03-26 14:44:44 +01:00
cc8b40585b feat(unified-storage): add traces for all the backend sub-paths (#102876) 2025-03-26 12:50:29 +01:00
65d474bfa4 chore(unified-storage): update default pruner history limit (#102799) 2025-03-25 17:00:12 +02:00
d3468105d7 Storage: Fix resource history sorting and pagination in Spanner backend (#102777)
* Align spanner ListHistory implementation with sql one

* Move migration logic

* Move migration code

---------

Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
2025-03-25 15:28:24 +01:00
64e939fd5d Unified Storage: Adds span and debug log for paginated List calls (#101912)
Adds span and debug log for paginated List calls
2025-03-21 14:20:27 -06:00
6d570db312 Storage: Add ascending order support for NotOlderThan queries and introduce ResourceVersionMatch_Unset as default (#102505)
* Add support for ASC ordering and introduce ResourceVersionMatch_Unset as default
Add SortAscending to continue token and add integration test for pagination.

* Change protobuf order

* Make backwards compatible

* Update pkg/storage/unified/sql/backend.go

Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>

---------

Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>
2025-03-21 18:35:32 +02:00
c33a53a47a K8s: Remove restore functionality; can be done with list (#102560) 2025-03-20 16:38:32 -05:00
ce350df79b Storage: Add resource version matching in unified storage API (#102417)
Add NotOlderThan support to getHistory

Add support for Exact

Add tests

Refactor tests

Add error test

Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
2025-03-19 16:16:48 +01:00
4927376f32 unistore: use the same connection string as grafana (#102387) 2025-03-19 13:56:54 +01:00
3e3ab8e064 fix(unified-storage): reduce lock contention when pruning (#102425) 2025-03-19 12:15:04 +01:00
0bafd4e99d unistore: update spanner ddl to include unistore tables (#102198)
* provision unistore tables

* update the tests to use generated namespace

* update ddl

* update ddl

* clean streams

* missing space

* use engine.db.query
2025-03-18 14:37:11 +01:00
08994304d8 fix(unified-storage): check resource_api cfg for isHA (#102283) 2025-03-18 12:33:27 +01:00
1700a8aa9f feat(unified-storage): prune history table based on limits (#101970) 2025-03-17 11:36:38 +01:00
6f12712ad6 Add Spanner dialect to pkg/storage/unified/sql/sqltemplate package. (#102121) 2025-03-13 16:38:05 +01:00
31a371e385 fix(unified-storage): remove bleve index metric from global scope (#101825)
* refactor grafana_index_server_index_size to calculate in a goroutine instead of at scrape time and remove grafana_index_server_indexed_docs metric

* use wire to inject bleve index metrics

* remove sprinkles metrics from bleve index metrics

* log error when trying to calculate file index size and bump interval to 1m instead of 5s
2025-03-13 10:09:38 -04:00
3589baac68 Unistore: Batch write events (#101381)
* Batch write events

* Improve instrumentation

* Measure batch phases

* Detect lock contention

* remove the execBatch goroutine

* removing tracing prefix

* detect context cancel

* batch channel map
2025-03-13 10:24:12 +02:00
64f1df69d5 Search: Rename managed object index (#101856) 2025-03-10 19:48:53 +03:00
b7a0aeeb0d SCIM: Disable auto assign organization if the user has been provisioned (#101307)
* Add isProvisioned field to model

* Add new isProvisioned column to migration

* Disable auto assignment to organization if the user is provisioned

* add annotation to user model

* add annotation to user models

* Remove IsProvisioned field from Identity

* Move new field assignenment and add default value

* Update annotations for user query results

* Remove isProvisioned from identity

* Add new column to test

* Resolve user from identity at SyncOrgHook
2025-03-03 17:51:23 +01:00
f5e5824bab fix (unified-storage): stop registering unified storage metrics in global state (#101322)
* move prometheus.register for unified storage metrics into metrics.go and do most of the plumbing to get it to work

* convert StorageApiMetrics to pointer and check for nil before using it

* rename type and variables to something more sensible

---------

Co-authored-by: Jean-Philippe Quéméner <jeanphilippe.quemener@grafana.com>
2025-02-28 07:39:39 -05:00
806c043e45 UnifiedStorage: Rename Batch processing to Bulk (#101413) 2025-02-28 08:41:08 +03:00
8c935c8f4a [unistore] Add benchmark for write throughput (#101345)
* Add generic benchmark

* address comments
2025-02-26 17:17:35 +02:00
8f37822050 Unified/Blob: Allow direct access to UIDs (#101339) 2025-02-26 16:18:59 +02:00
fa74d1c36d Authn: Sync authlib and update how we construct authn client interceptor (#101124)
* Sync authlib and update how we construct authn client interceptor

* Remove namespace from checker
2025-02-26 09:22:09 +01:00
53e91fd5e8 unistore: close event stream on context cancelation (#101293)
* add tests for broacaster

* fix sql notifier not closing the stream

* fix sql notifier not closing the stream

* close sub

* fix broadcaster test

* fix broadcaster test

* suggestion
2025-02-25 08:28:31 -08:00
9282e4cad4 fix(unified-storage): don't use in-process if db type is not set (#101254) 2025-02-24 21:44:42 +01:00
e13bd52da6 unistore: move continue to the resource package (#101206)
* minor improvements to tests

* move continue token

* update sql backend
2025-02-24 18:02:30 +02:00
40d1677265 unistore: improve storage backend integration tests (#101134)
* Add Stats tests

* Add integration tests for getstats

* fix signature
2025-02-21 15:06:23 +01:00
7be1fd953a fix(unified-storage): enable in-process events for single-instance (#100807) 2025-02-21 12:25:35 +01:00
b4c4b9abbd Remove embedded WriteEvent from WrittenEvent struct, to avoid having unnecessary fields. (#101081) 2025-02-20 17:34:25 +01:00
6e5de80ef2 unistore: refactor the storage backend test suite (#101061)
* refactor the storage backend test suite

* move testing

* pass a newBackendFunc
2025-02-20 16:43:26 +01:00
4a8b1ba61b tests(unified-storage): use unified storage grpc client (#101063) 2025-02-20 15:03:08 +01:00
6eeb28e312 Authn: use authenticator for grpc (#99573)
* Remove usage of grpc-authenticator

* Cleanup client construction code
2025-02-17 10:58:59 +01:00
d0394bfa7e Extract NewSearchOptions from unified sql client setup. (#100719)
* Extract NewSearchOptions from unified sql client setup.

Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2025-02-14 16:39:48 +01:00
c522a5b13b fix(unified-storage): proper setup TLS in new db_engine for MySQL (#100686) 2025-02-14 16:23:25 +01:00
b1222be02e unistore: add small buffer of watched events (#100431)
* change log level

* Add a small buffer when watching events
2025-02-14 12:18:59 +02:00
a34e7e176d Unified Storage: Sprinkles latency metric (#100542)
* add sprinkles latency metric

* fixes failing tests - forgot to register metric only once
2025-02-12 16:22:18 -06:00
a5355fd66c Storage: Add command line tool to migrate legacy dashboards (and folders) to unified storage (#99199) 2025-02-11 19:57:46 +02:00
3ba0d8d4b5 Unified Storage: Improve observability for indexing latency (#99700)
* adds extra debug logs and a new metric for poller query latency to help us better understand watch and index latency for write events

* adds trace span to the index for handling index write events
2025-01-28 12:30:20 -06:00
0cef2b9ae7 Dashboard Versions: Make compatible with app platform (#99327) 2025-01-28 08:17:52 -06:00
437b7a565d Auth: Add access token to in-proc communication and ServiceIdentity (#98926)
Use fake access token for in-proc grpc and add ServiceIdentity 
---------

Co-authored-by: gamab <gabriel.mabille@grafana.com>
Co-authored-by: Karl Persson <23356117+kalleep@users.noreply.github.com>
2025-01-24 14:03:23 +01:00
c4c934e0bd SQL/Storage: Remove SkipDataMigration flag (#99404) 2025-01-23 12:34:48 +02:00
dd483fc17f Unified Storage: Search permissions put behind feature flag (#99340)
* add feature flag

* puts search permission filtering behind a feature flag

* fixes issue with doc match id. When the match is for an in-memory index, the internal id is a string (this is what we expected). However, when its a file-based index, the internal id is a binary encoded int64 that point to something internally. So to get the id, we need to use ExternalID() instead of relying on the indexInternalID to be the correct format.

* adds debug log

* update comment

* formatting
2025-01-22 05:38:37 -06:00
9d30911107 SQL/Storage: Use xorm code migration for DeletionMarker (#99353) 2025-01-22 14:07:08 +03:00
d91d729749 Storage/SQL: Support more filter by namespace/group/resource (#99249) 2025-01-21 12:15:08 +03:00