351 Commits

Author SHA1 Message Date
f67713726b add debug logs to search server distributor (#108679) 2025-07-25 12:36:35 +00:00
16aee8047c Unified Search: Randomise instance selected by distributor (#108619)
* Randomise instance selected by distributor

* add namespace to distributor logging

---------

Co-authored-by: Will Assis <william@williamassis.com>
2025-07-24 19:38:35 +00:00
981fdb29d4 update storage-api to only build index if it owns the namespace (#108418)
* update storage-api to only build index if it owns the namespace

---------

Co-authored-by: Mustafa Sencer Özcan <mustafasencer.ozcan@grafana.com>
2025-07-23 15:59:24 -04:00
34f1be796d unified-storage: expose ring replication factor config (#106345)
* config ring replication factor

* change default

* rename

* fix test

* fix
2025-07-22 09:32:22 +02:00
6c84461b7a fix: Always return searchWrapper to enable shadow traffic (#108373)
Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>
2025-07-21 13:53:44 +02:00
4b440cf45d Remove timeout info from logs when shadow request to search succeeds (#108324)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-07-18 14:03:42 +00:00
2dba473015 Feature/unified storage search dual reader (#108291)
* Add UnifiedStorageSearchDualReaderEnabled feature flag

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

* Refactor UniSearch Dual Reader

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

* Run make gen-feature-toggles

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

* fix: unit tests search_client

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

* feat: cancels shadow search requests after 500ms

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>
Co-authored-by: Will Assis <william@williamassis.com>
Co-authored-by: Bruno Abrantes <bruno@brunoabrantes.com>
2025-07-18 14:43:56 +02:00
4a779c4ccb fix: unified resource server list queries order column (#108094) 2025-07-16 08:09:31 +02:00
2cd0be3cbd Update authlib version (#107939)
* update authlib version

* add latest versions

* make update-workspace

* typo

* Trigger Build

* Trigger Build
2025-07-11 14:55:52 -04:00
9786389ae8 SecureValues: Support inline secure values in GrafanaMetaAccessor (#107996) 2025-07-11 16:47:54 +00:00
7002ab90ae unistore: save returns a writecloser (#107955)
* unistore: save returns a writecloser

* go-lint

* address comments
2025-07-11 18:25:48 +02:00
ea0ddb3fc9 unistore: refactor get to return a reader (#107951) 2025-07-11 11:10:19 +02:00
8fd5739576 [unified-storage/search] Don't expire file-based indexes, check for resource stats when building index on-demand (#107886)
* Get ResourceStats before indexing
* Replaced localcache.CacheService to handle expiration faster (localcache.CacheService / gocache.Cache only expires values at specific interval, but we need to close index faster)
* singleflight getOrBuildIndex for the same key
* expire only in-memory indexes
* file-based indexes have new name on each rebuild
* Sanitize file path segments, verify that generated path is within the root dir.
* Add comment and test for cleanOldIndexes.
2025-07-10 11:54:10 +00:00
b6dd08da2f unistore: fix delete and db closed in kv store (#107918)
* fix delete and db closed

* fix tests
2025-07-10 11:34:36 +02:00
650abfd2ff Unified Storage: Remove distributor todo comment (#107623)
Remove todo
2025-07-04 12:59:22 +00:00
6bb74ff56a unistore: add key validation (#107622) 2025-07-04 12:52:24 +00:00
dcb853d309 Unified Search: Remove resource-server-specific methods from distributor (#107607)
* Remove resource-server-specific methods from distributor
* Remove BlobStoreServer interface implementation from distributor
2025-07-04 12:48:27 +02:00
b7153d4d20 fix: grpc resource delete error when qos enabled (#107560) 2025-07-03 12:02:05 +02:00
696657bdd1 unistore: add kv based storage backend (#107305)
* Add datastore

* too many slashes

* lint

* add metadata store

* simplify meta

* Add eventstore

* golint

* lint

* Add datastore

* too many slashes

* lint

* pr comments

* extract ParseKey

* readcloser

* remove get prefix

* use dedicated keys

* parsekey

* sameresource

* unrelated

* name

* renmae tests

* add key validation

* fix tests

* refactor a bit

* lint

* allow empty ns

* get keys instead of list

* rename the functions

* refactor yield candidate

* update test

* unistore: add LastResourceVersion to datastore

* lint

* use map string

* missing err check

* fix

* Add storage backend

* remove hasmore

* fix tests

* small refactor

* pre-alloc

* extract the folder

* lint

* refactor

* handle context canceled in ListHistory to pass the tests

* fix the resource test

* unistore: provide generic tests for the kv interface (#107443)

unistore: move the kv tests to the testing package

* Update pkg/storage/unified/resource/storage_backend_test.go

Co-authored-by: Peter Štibraný <pstibrany@gmail.com>

* address comments

* comments

* comments

* comments

* normalise the names and add helper method

* events comments

* rename function

---------

Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
2025-07-02 10:57:37 +00:00
f09e85c048 unified-storage: Distributor rename to better reflect that it'll be used for search (#107409)
* rename distributor/ring references to "storage-api" to "search-server"
2025-07-01 11:15:10 -04:00
974a2c47f9 feat(unified-storage): add qos support for the resource server (#105939) 2025-07-01 09:22:55 +00:00
46094a16c7 unistore: add LastResourceVersion to the datastore (#107303)
* Add datastore

* too many slashes

* lint

* add metadata store

* simplify meta

* Add eventstore

* golint

* lint

* Add datastore

* too many slashes

* lint

* pr comments

* extract ParseKey

* readcloser

* remove get prefix

* use dedicated keys

* parsekey

* sameresource

* unrelated

* name

* renmae tests

* add key validation

* fix tests

* refactor a bit

* lint

* allow empty ns

* get keys instead of list

* rename the functions

* refactor yield candidate

* update test

* unistore: add LastResourceVersion to datastore

* lint

* use map string

* missing err check

* fix
2025-06-30 15:49:26 +02:00
7aad041f70 unified-storage: Add integration tests for distributor (#105771)
* Add integration tests for distributor
2025-06-30 09:31:58 -04:00
Jo
7940da4803 Fix organization deletion error messages (#107380)
* Fix organization deletion error messages

- Improve error message clarity when attempting to delete active organization
- Fix incorrect 'Failed to update organization' message to 'Failed to delete organization'
- Update comment to be more precise about the check being performed

Fixes #92792

* Improve error handling in organization deletion service

- Add contextual error message when dashboard deletion fails during org deletion
- Include organization ID in error message for better debugging
- Import fmt package for error formatting

This helps administrators understand which specific organization failed
during the deletion process when dashboard cleanup encounters issues.

* Fix typo in unified resource access error message

Change 'Namespace missmatch' to 'Namespace mismatch' for correct spelling.
2025-06-30 13:58:37 +02:00
4a272fb61b unistore: add eventstore and notifier (#107182)
* Add datastore

* too many slashes

* lint

* add metadata store

* simplify meta

* Add eventstore

* golint

* lint

* Add datastore

* too many slashes

* lint

* pr comments

* extract ParseKey

* readcloser

* remove get prefix

* use dedicated keys

* parsekey

* sameresource

* unrelated

* name

* renmae tests

* add key validation

* fix tests

* refactor a bit

* lint

* allow empty ns

* get keys instead of list

* rename the functions

* refactor yield candidate

* update test

* lint

* missing err check

* address comments

* increase the timeout
2025-06-30 11:20:57 +02:00
f4ee58db50 unified-storage: split resource index client (#106297)
* split resource server and index server grpc connection if defined in config
2025-06-27 08:15:52 -04:00
0982cfd9a0 Unified Storage/Search: Add max count config for indexing (#107255)
* Add max count config for indexing
* Build empty index when max count is exceeded
* Address linting
* Refactor buildIndexes
* Add test for max count threshold
* Update test doc comments
* Refactor TestBuildIndexes_MaxCountThreshold to not use mock framework
* Rename mocks used in TestBuildIndexes_MaxCountThreshold

* Refactor mockResourceIndex

* Test setting of indexing threshold configs

* Tweak comments, log

* Fix logging in buildEmptyIndex

* Export and reuse TestDocumentBuilderSupplier

* Reuse MockResourceIndex
2025-06-27 14:00:39 +02:00
b2aaa9b3ce kvstore: ensure we return notFound on delete (#107288)
* kvstore: ensure we return notFound on delete

* fix tests
2025-06-27 12:00:03 +02:00
2e9559a376 Include resource name in the event. (#107185) 2025-06-27 09:29:49 +00:00
53ec379af8 unistore: add metadata store (#107151)
* Add datastore

* too many slashes

* lint

* add metadata store

* simplify meta

* Add datastore

* too many slashes

* lint

* pr comments

* extract ParseKey

* readcloser

* remove get prefix

* use dedicated keys

* parsekey

* sameresource

* unrelated

* name

* renmae tests

* fix tests

* lint

* allow empty ns

* get keys instead of list

* rename the functions

* refactor yield candidate
2025-06-27 08:25:27 +00:00
dc79e6f3dc unified-storage: fix auth handling in distributor (#106062)
* disable auth handling in the distributor
2025-06-26 12:32:59 -04:00
2297417374 Unistore: Add datastore (#107137)
* Add datastore

* too many slashes

* lint

* pr comments

* extract ParseKey
2025-06-26 10:36:31 +02:00
eeb01126b3 grpc: wrapContext should not replace existing metadata, but append to it. (#107126)
* grpc: wrapContext should not replace existing metadata, but append to it.

* Add test for wrapContext if there's no previous metadata.

* Rename encodeIdentityInMetadata to encodeIdentityInMetadataPairs
2025-06-25 08:06:45 +00:00
5150ebfba8 unistore: index search references (#106954) 2025-06-24 14:54:07 -05:00
53cb80e58c Unistore: save should use a Reader not ReadCloser (#107128)
* save should use a Reader not ReadCloser

* save should use a Reader not ReadCloser
2025-06-24 15:37:22 +00:00
27a5bab1c1 unistore: Add kvstore interface (#107094)
* Add kvstore interface

* add owner

* go lint

* remove comment

* update comment

* remove GetOptions

* add sortorder unspecified

* nit

* nit

* nit

* move txn

* use io.reader

* use io.reader

* change again the default order + comments

* change again the default order + comments

* use readcloser for Save
2025-06-24 15:50:57 +02:00
6e5056e1be Add tracing to bulk process. (#107047)
Add logging to bulk process.
2025-06-24 09:45:32 +01:00
9062d88ea0 unistore: filter trash requests (#106767)
* deleted by user

* use the correct checker

* add tests

* refactor
2025-06-24 10:17:34 +02:00
b48337a7c8 feat: introduce ttl cache for bleve indices (#106842) 2025-06-17 16:28:51 +02:00
5f21f320f7 fix(unified-storage): use continue token containing both formats for dualwriter (#106525) 2025-06-13 15:59:46 +02:00
5135d5c87d Unified storage: Reconstruct index in the background every 24h (#106422) 2025-06-12 14:34:48 -05:00
ab99211b40 Dashboard Provisioning: Reduce db load (#106114) 2025-05-29 09:39:07 +03:00
9ba031364b fix(unified-storage): use batching for index creation (#106034) 2025-05-27 09:48:39 +02:00
326081cdb7 Chore: Remove unified storage go modules (#106035) 2025-05-27 09:49:21 +03:00
814707152d chore(unified-storage): contextualize the index build logger (#106025) 2025-05-26 21:33:51 +02:00
04d39cbbc6 unistore: split ListIterator and ListHistory in StorageBackend (#105654)
split listIterator from ListHistory
2025-05-23 15:00:18 +02:00
e57be36936 fix(unified-storage): race in resource server watch (#105786) 2025-05-23 09:19:20 +02:00
95eabcf03d refactor(unified-storage): rename cache to avoid import conflicts (#105831) 2025-05-22 17:12:31 +02:00
a13fe75de8 refactor(unified-storage): set the GUID in the resource server (#105683) 2025-05-21 09:49:49 +02:00
bdae4424e8 unified-storage: setup distributor module option A (#104737)
* setup distributor module

* move lifecycler into resource server provider

* remove ring/client pool setup from distributor module and use the same ring/client pool between storage server module and distributor module

* implement resourcestore server methods

* make healthcheck fail if ring is not running
2025-05-19 07:46:03 -04:00