589 Commits

Author SHA1 Message Date
b1b9cc43a8 SecretsManager: Adding ability to disable all DEKs (#108444)
* Adding dek deactivation and rename list dek

* disable data keys from manager

* separate interface and don't use in encryption manager
2025-07-25 17:11:17 +01:00
69743ed1a4 SecretsManager: remove unused metric (#108694) 2025-07-25 14:04:43 +00:00
f67713726b add debug logs to search server distributor (#108679) 2025-07-25 12:36:35 +00:00
183cd0843e keep instance in the ring on shutdown (#108609) 2025-07-24 15:39:12 -04:00
16aee8047c Unified Search: Randomise instance selected by distributor (#108619)
* Randomise instance selected by distributor

* add namespace to distributor logging

---------

Co-authored-by: Will Assis <william@williamassis.com>
2025-07-24 19:38:35 +00:00
981fdb29d4 update storage-api to only build index if it owns the namespace (#108418)
* update storage-api to only build index if it owns the namespace

---------

Co-authored-by: Mustafa Sencer Özcan <mustafasencer.ozcan@grafana.com>
2025-07-23 15:59:24 -04:00
0e7b041b27 Secrets: Move decrypt types to contracts and export public at root pkg (#108376)
* Secrets: Move decrypt types to contracts and export public at root pkg

* Provisioning: Replace decrypt pkg imports

* Merge wire changes
2025-07-23 12:14:28 +02:00
34f1be796d unified-storage: expose ring replication factor config (#106345)
* config ring replication factor

* change default

* rename

* fix test

* fix
2025-07-22 09:32:22 +02:00
12a7e03e2f Secrets: encryptionStoreImpl produces metrics (#108390) 2025-07-21 11:00:59 -03:00
6c84461b7a fix: Always return searchWrapper to enable shadow traffic (#108373)
Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>
2025-07-21 13:53:44 +02:00
4b440cf45d Remove timeout info from logs when shadow request to search succeeds (#108324)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-07-18 14:03:42 +00:00
2dba473015 Feature/unified storage search dual reader (#108291)
* Add UnifiedStorageSearchDualReaderEnabled feature flag

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

* Refactor UniSearch Dual Reader

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

* Run make gen-feature-toggles

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

* fix: unit tests search_client

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

* feat: cancels shadow search requests after 500ms

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>
Co-authored-by: Will Assis <william@williamassis.com>
Co-authored-by: Bruno Abrantes <bruno@brunoabrantes.com>
2025-07-18 14:43:56 +02:00
01692bc876 Secrets: remove unused SecureValueMetadataStorage.ReadForDecrypt method (#108181)
* Secrets: remove unused SecureValueMetadataStorage.ReadForDecrypt method

* remove unused struct: secureValueForDecrypt
2025-07-16 11:52:26 -03:00
2c4bbf8b1d Secrets: Reduce amount of feature toggle checks to only app entrypoint (#108110)
* Secrets: Reduce amount of feature toggle checks to only app entrypoint

* Wire: Fix merge conflict

* Wire: Fix merge conflict
2025-07-16 10:56:59 +02:00
3cda233468 Secrets: Remove temporary allowlist (#108118) 2025-07-16 09:39:33 +02:00
4a779c4ccb fix: unified resource server list queries order column (#108094) 2025-07-16 08:09:31 +02:00
6fa3c196d4 Secrets: add example based testing that tries to decrypt deleted secure value (#108140) 2025-07-15 15:40:52 -03:00
b017f42278 Unistore: Fix dualwriter mode 3 (#108092)
* Unistore: Fix dualwriter mode 3

Fallback to legacy if Unified is the main store and we get an error when
reading from it

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>


---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-07-15 11:16:26 -03:00
403d6380fa Secrets: Remove gating with experimental APIs feature toggle (#108058) 2025-07-14 17:46:17 +02:00
baa89f3eac Secrets: encryption encryption storage uses versioning (#108036)
* Secrets: delete unused FakeKeeper

* Secrets: encrypted value storage stores versions

* add version to span

* trigger build

* remove ineffectual assignment

* lint

* drop secret_encrypted_value.uid / add name and version columns
2025-07-14 09:28:07 -03:00
84a4ed612b fix: re-order db initialization logic for resource server (#108059) 2025-07-14 10:25:42 +00:00
2cd0be3cbd Update authlib version (#107939)
* update authlib version

* add latest versions

* make update-workspace

* typo

* Trigger Build

* Trigger Build
2025-07-11 14:55:52 -04:00
9c1b2fb792 Secrets: Bump API version to v1beta1 (#108026) 2025-07-11 19:14:05 +02:00
9786389ae8 SecureValues: Support inline secure values in GrafanaMetaAccessor (#107996) 2025-07-11 16:47:54 +00:00
7002ab90ae unistore: save returns a writecloser (#107955)
* unistore: save returns a writecloser

* go-lint

* address comments
2025-07-11 18:25:48 +02:00
602b7826c4 refactor(unified-storage): remove resource_server config option (#107649) 2025-07-11 15:33:51 +02:00
9d0a23e1f5 Secrets: add crudl+decrypt state machine test (#107971)
* Secrets: add state machine test for CRUDL+decrpt operations

* make update-workspace

* make update-workspace

* make enterprise-dev

* make update-workspace

* fix go.mod

* make update-workspace

* fix gomod

* make update-workspace

---------

Co-authored-by: Matheus Macabu <macabu.matheus@gmail.com>
2025-07-11 09:40:50 -03:00
ea0ddb3fc9 unistore: refactor get to return a reader (#107951) 2025-07-11 11:10:19 +02:00
8fd5739576 [unified-storage/search] Don't expire file-based indexes, check for resource stats when building index on-demand (#107886)
* Get ResourceStats before indexing
* Replaced localcache.CacheService to handle expiration faster (localcache.CacheService / gocache.Cache only expires values at specific interval, but we need to close index faster)
* singleflight getOrBuildIndex for the same key
* expire only in-memory indexes
* file-based indexes have new name on each rebuild
* Sanitize file path segments, verify that generated path is within the root dir.
* Add comment and test for cleanOldIndexes.
2025-07-10 11:54:10 +00:00
b6dd08da2f unistore: fix delete and db closed in kv store (#107918)
* fix delete and db closed

* fix tests
2025-07-10 11:34:36 +02:00
9d2eadcfd2 PoC: replace using testing.Short in storage/unified package with integration test skip method (#107887)
* PoC: check if testing.Short is called from integration tests only.

* Rename helper function.

* Fix logic.

* Remove skipping of integration tests from non-integration tests.

* Remove skipping of integration tests from non-integration tests.

* Fix import.
2025-07-09 15:32:10 +00:00
8283d35e56 Secrets: make operations sync (#107732)
* Secrets: make operations sync

* k8s gen / update query to list secure values to include the version

* always store new version of a secret

* make update-workspace

* go mod tidy

* update queries

* update queries

* improve and use testutils in decrypt_store_test

* fix broken test

* make update-workspace

* ./hack/update-codegen.sh secret

* update Test_SecureValueMetadataStorage_CreateAndRead

* undo dependency changes

* linter: fix remaining errors

---------

Co-authored-by: Matheus Macabu <macabu.matheus@gmail.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-07-09 10:43:34 -03:00
3f502f305d Chore: Update mocks with recent mockery (#107816) 2025-07-09 09:15:34 +02:00
7787ff6b2e unistore: add more kvstore tests (#107692)
add more kvstore tests
2025-07-07 15:00:40 +02:00
650abfd2ff Unified Storage: Remove distributor todo comment (#107623)
Remove todo
2025-07-04 12:59:22 +00:00
6bb74ff56a unistore: add key validation (#107622) 2025-07-04 12:52:24 +00:00
46c38fdbb7 SecretsManager: Introduce worker and secret async service (#107614)
SecretsManager: Introduce worker and secret aysnc service

Co-authored-by: PoorlyDefinedBehaviour <brunotj2015@hotmail.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
2025-07-04 13:13:48 +01:00
dcb853d309 Unified Search: Remove resource-server-specific methods from distributor (#107607)
* Remove resource-server-specific methods from distributor
* Remove BlobStoreServer interface implementation from distributor
2025-07-04 12:48:27 +02:00
15e1aa8855 SecretsManager: Introduce decrypt store (#107586)
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
Co-authored-by: PoorlyDefinedBehaviour <brunotj2015@hotmail.com>
2025-07-04 09:22:10 +01:00
a59ec345c2 SecretsManager: Introduce metrics and logs (#107582)
Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
2025-07-03 17:32:18 +01:00
cfd3b9f582 SecretsManager: outbox use message id (#107472)
* SecretsManager: outbox use message id

Co-authored-by: PoorlyDefinedBehaviour <brunotj2015@hotmail.com>

* Remove query timestamp

* Add missing query

---------

Co-authored-by: PoorlyDefinedBehaviour <brunotj2015@hotmail.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
Co-authored-by: Matheus Macabu <macabu.matheus@gmail.com>
2025-07-03 15:21:47 +01:00
8d8b824f73 unistore: skipping badger test failing atm (#107572)
skipping badger test failing atm
2025-07-03 12:31:07 +00:00
4d8678c7f2 SecretsManager: Add base encryption manager (#107562)
Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-07-03 11:29:14 +01:00
b7153d4d20 fix: grpc resource delete error when qos enabled (#107560) 2025-07-03 12:02:05 +02:00
041c343a86 Unified storage: Respect GF_DATABASE_URL override (#105331)
* Database for unified storage resources now reuses DB code that respects URL override.

Access instrument_queries via section getter.
2025-07-03 09:57:40 +00:00
e076c74869 sqltemplate, dbimpl: Remove single-method function types (#107525)
* Remove dbProviderFunc function.

This removes one extra indirection that made the code bit more difficult to navigate.

* Remove indirection function types implementing single-method interfaces.

This streamlines the code and makes it bit easier to navigate.

* Update pkg/storage/unified/sql/sqltemplate/dialect_mysql.go

Co-authored-by: Mustafa Sencer Özcan <32759850+mustafasencer@users.noreply.github.com>

---------

Co-authored-by: Mustafa Sencer Özcan <32759850+mustafasencer@users.noreply.github.com>
2025-07-03 10:38:12 +02:00
f32d944b23 Secrets: Add initial tracing instrumentation (#107513) 2025-07-02 14:43:36 +02:00
d55541735a unistore: use different ns for each test (#107509)
use different ns for each test
2025-07-02 13:50:30 +02:00
696657bdd1 unistore: add kv based storage backend (#107305)
* Add datastore

* too many slashes

* lint

* add metadata store

* simplify meta

* Add eventstore

* golint

* lint

* Add datastore

* too many slashes

* lint

* pr comments

* extract ParseKey

* readcloser

* remove get prefix

* use dedicated keys

* parsekey

* sameresource

* unrelated

* name

* renmae tests

* add key validation

* fix tests

* refactor a bit

* lint

* allow empty ns

* get keys instead of list

* rename the functions

* refactor yield candidate

* update test

* unistore: add LastResourceVersion to datastore

* lint

* use map string

* missing err check

* fix

* Add storage backend

* remove hasmore

* fix tests

* small refactor

* pre-alloc

* extract the folder

* lint

* refactor

* handle context canceled in ListHistory to pass the tests

* fix the resource test

* unistore: provide generic tests for the kv interface (#107443)

unistore: move the kv tests to the testing package

* Update pkg/storage/unified/resource/storage_backend_test.go

Co-authored-by: Peter Štibraný <pstibrany@gmail.com>

* address comments

* comments

* comments

* comments

* normalise the names and add helper method

* events comments

* rename function

---------

Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
2025-07-02 10:57:37 +00:00
3674d952a6 fix: use database section for resource server db engine configuration (#107495) 2025-07-02 09:00:51 +00:00