82 Commits

Author SHA1 Message Date
f2bd367f61 fix: keep dual writer mode in sync with kv store (#106692) 2025-06-20 08:37:20 +02:00
1dd59ca599 DualWriter: remove RV+UID for secondary update (#105543) 2025-05-21 18:25:40 +00:00
ebf3a8d205 fix: make data syncer run configurable on mode (2->3) change (#104401) 2025-04-25 13:43:35 +02:00
0764ecb98d K8s/DualWriter: Use dualwriter package for all dual writers (#101393) 2025-03-01 21:26:14 +03:00
58457d41d3 K8s/DualWriter: Remove legacy interface (#101395) 2025-02-27 13:27:28 +03:00
2e82ac0cc1 Unistore: keep current dual writing mode when unable to run data syncer at bootstrap (#100852)
* Unistore: keep current dual writing mode when unable to run data syncer at bootstrap

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-02-20 17:43:36 -03:00
62d502086e K8s: Dualwriter mode3: Return error from unistore (#100992) 2025-02-19 20:40:17 +02:00
77fd572973 dont return not found error in mode 2 (#100758)
* dual writer: dont return not found error in mode 2 when failing to find object in unistore
2025-02-14 20:39:51 +02:00
aca024bcbb fix(unified-storage): use dedicated mocks per storage type (#100389) 2025-02-12 09:59:21 +01:00
d1b4162a33 refactor(unified-storage): measure also failed queries (#100430) 2025-02-11 16:19:43 +01:00
8a9f6416d2 fix(unified-storage): error on failed primary deletes in mode2 (#100427) 2025-02-11 15:39:39 +01:00
6eaf702e96 fix(unified-storage): return legacy data in mode 2 (#100353) 2025-02-11 10:34:56 +01:00
6af67197a7 Dual Writer: Remove list comparisons in mode 2 & 3 (#100215) 2025-02-07 09:17:53 -05:00
364559e856 feat(unified-storage): use continue token when building list (#100143) 2025-02-05 21:02:35 +01:00
39d94eabcd Auth: Fix function name (#100122)
Fix spelling
2025-02-05 15:32:22 +01:00
dde199f684 Unistore: use object from legacy in mode2 (#99533) 2025-01-27 08:11:31 -06:00
437b7a565d Auth: Add access token to in-proc communication and ServiceIdentity (#98926)
Use fake access token for in-proc grpc and add ServiceIdentity 
---------

Co-authored-by: gamab <gabriel.mabille@grafana.com>
Co-authored-by: Karl Persson <23356117+kalleep@users.noreply.github.com>
2025-01-24 14:03:23 +01:00
680e6bc1f8 Authlib: Use types package rather than claims (#99243) 2025-01-21 12:06:55 +03:00
124440ccee Unistore: Block on legacy writes in mode3 (#98147) 2024-12-19 08:47:48 -06:00
3fe2227c82 [auth] make id-token optional (#97831)
make idtoken optional

enure there is always an identity in the context

fix: update token

fix: now it should work

fix: now it should work
2024-12-17 13:28:00 +02:00
1509fadd5b refactor(unified-storage): make datasyncer configurable (#97832) 2024-12-12 21:41:01 +01:00
29cdfdff87 Storage: Revert using real gRPC for integration tests (#96410)
---------

Co-authored-by: Todd Treece <todd.treece@grafana.com>
2024-11-15 16:50:49 -05:00
a1de4cc5fc Allow downgrade from any mode to mode 1 or 2. Allow setting all dual writer combinations (#95968)
* Allow downgrade from any mode to mode 1 or 2

* Allow for all combinations of dual writer setting

* Add all cases

* Include mode3
2024-11-11 10:52:36 +01:00
4e1f0dadbd UnifiedStorage: Default to running unified-grpc in integration tests (#93492) 2024-10-31 16:29:32 +03:00
a03652494c Dual Writer simplification (#93852)
* All objects should have an UID

* Now with a different error message

* Simplify create on DW 2: use the same object to write to both storages

* Run only one test

* Add check for status code

* Add name if it's not present in mode2

* Populate UID in legacy

* Remove logs and commented code

* Change dualwriter1

* Remove commented code

* Fix list test

* remove get on update from dualwriter 2

* Get object before updating. Better var renaming

* Finish rebasing

* Comment test

* Uncomment tests

* Update legacy first. Add preconditions

* Remove preconditions

* Fix update test

* copy RV from unified to legacy objects

* revert changes to playlist xorm store

* Improve logging. Add go routines for mode3

* Add tests for async funcs in mode3

* Lint

* Lint

* Lint. Start to fix tests

* Fix watcher tests

* Fix store tests

* Fiinish fixing watcher tests

* Fix server tests

* add name check

* Update pkg/apiserver/rest/dualwriter_mode1.go

Co-authored-by: Bruno Abrantes <bruno.abrantes@grafana.com>

* All objects should have an UID

* Now with a different error message

* Simplify create on DW 2: use the same object to write to both storages

* Run only one test

* Add check for status code

* Add name if it's not present in mode2

* Populate UID in legacy

* Remove logs and commented code

* Change dualwriter1

* Remove commented code

* Fix list test

* remove get on update from dualwriter 2

* Get object before updating. Better var renaming

* Finish rebasing

* Comment test

* Uncomment tests

* Fix update test

* revert changes to playlist xorm store

* Improve logging. Add go routines for mode3

* Lint

* Fix watcher tests

* Fiinish fixing watcher tests

* Add mode 5 with etcd test case. Add early check to fail on populated RV in payload

* we can't set RV to the found object when updating

* Lint

* Don't fail on update playlists

* Name should not be different when updating and it should be not empty on creating

* Fix tests

* Update pkg/apiserver/rest/dualwriter_mode2.go

Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>

* Lint

* Fix mode 5 tests

* Lint

* Add generateName condition on every mode. Fix tests

* Lint

* Add condition on where name or generate name have to be set

* Fix test

* Lint

* Fix folders test

* We dont need to send name for mode1

* Fail if UID is not present

* Remove change from not running test

* Remove unused line

* Lint

* Update pkg/storage/unified/apistore/store.go

Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>

* Improve error message

* Fix broken watcher test

* Fail on name mismatch on update

* Remove log

* Make sure UIDs match on create in both stores

* Lint

* Write first to unified storage

* Remove uid setting

* Remove RV only in mode2

* Fix test. Remove log line

* test

* No need to asser on RV in mode3

* Remove RV check due to race condition

* Update dualwriter.go

Co-authored-by: Georges Chaudy <chaudyg@gmail.com>

* Update pkg/storage/unified/client.go

* remove unused parameter

* log an error for object is missing UID instead of returning an error

* remove obj.SetResourceVersion("")

* log an error for object is missing UID instead of returning an error

* FInalise merge

* Move RV check to where it was

* Remove name check

* Remove server check for backwards compatibility

* Remove unused fn

* Move test checks for another PR

* Dont commit go work sum changes

* Only log error if RV is present for now.

---------

Co-authored-by: Todd Treece <todd.treece@grafana.com>
Co-authored-by: Bruno Abrantes <bruno.abrantes@grafana.com>
Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>
Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2024-10-23 10:29:41 +02:00
c4f906f7fa UniStore: Fix DualWriter compare method (#94705)
* UniStore: Fix DualWriter compare method

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2024-10-16 16:22:49 +02:00
4a60f29709 UniStore: register dual_writer_data_syncer_outcome metric (#94789)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2024-10-16 16:44:20 +03:00
9dc2ccdbfd UniStore: Test both Mode1 and Mode2 for LegacyToUnifiedStorage Syncer (#94784)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2024-10-16 06:43:01 +02:00
bfd3506549 UniStore: Enable DataSyncer in Mode1 + better logging (#94688)
* UniStore: Enable DataSyncer Mode1 + better logging

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
2024-10-15 06:41:12 +03:00
d60351d8e8 Unistore: fix DualWriter context cancelation on mode 1 (#93377)
* Unistore: fix DualWriter context cancelation on mode 1

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2024-09-23 10:34:36 -03:00
d724d463b1 DualWriter: Add mode 5 that always uses storage (#93169) 2024-09-10 13:07:06 -04:00
22c63ea3c5 DualWriter: remove mode4 wrapper (#93172) 2024-09-10 16:01:14 +03:00
2f792ee4ae DualWriter: Return error for unsupported watch (#93159) 2024-09-10 15:42:17 +03:00
8d2b8378ed Remove UID on update in dual writer mode 2 (#93032)
* Remove UID on update in dual writer mode 2

* WIP: add dashboard tests

* Add more crud tests for dashboards in each mode

* trigger build

* Fix test
2024-09-10 11:57:28 +02:00
2b3d2e5b40 UniStore: Remove leftover references to unifiedStorage feature toggle (#92685)
it got removed on https://github.com/grafana/grafana/pull/92192

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2024-08-29 18:04:36 +03:00
66e0121dd5 Pass resource name into the dual writer initialization (#92654) 2024-08-29 13:30:48 +02:00
09f102b72e Remove condition on where we return from unified storage in mode2 (#92593) 2024-08-28 16:31:53 +02:00
de2c9a06bf Unified Storage: First iteration Dual Write Syncer (#89809)
* Unified Storage: First iteration Dual Write Syncer

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
Co-authored-by: Leonor Oliveira <9090754+leonorfmartins@users.noreply.github.com>
Co-authored-by: Dan Cech <dcech@grafana.com>
2024-08-27 13:31:40 -03:00
81ce3c92d5 Remove optionsStorage and labelSelector (#92196) 2024-08-21 15:44:38 +02:00
f463d622d0 Chore: Remove depguard rule skip in apiserver (#91705) 2024-08-08 21:11:13 +03:00
fb9903ed7c Allow to watch from the dual writer (#91375)
* Allow to watch from the dual writer

* Remove comments

* Lint
2024-08-01 15:09:51 +02:00
8345af928c [Dual writer] Read kind value passed as dualwriter's argument (#91113)
* Include kind on the dualwriter to be used on metrics

* Use kind as argument

* Skip dual writer in mode4
2024-07-29 13:57:44 +02:00
6a1e835f12 Get kind from runtime.Object. Unify metric names (#91041)
* Normalize name labels

* Access kind

* Read kind from the runtime object instead from the options object

* use kind from object on create
2024-07-29 09:58:48 +02:00
ec343f276f Dual writer: mode 4 (#90581)
* Itroduce watcher and mode4

* Logging

* Mode4 should be initialized from the dual writer for observability

* Comment watch while it's not implemented

* Lint

* Use mode log when dual writer is initiated

* Use error from logger
2024-07-24 14:10:10 +02:00
67b74e1e8a Dual writer: mode 3 (#90045)
* Dual writer: mode 3

* Add integration tests for playlits in mode 3

* Remove todo

* Update pkg/apiserver/rest/dualwriter_mode3.go

Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>

* Admin: Fixes an issue where user accounts could not be enabled (#88117)

Fix: unable to enable user

* [REVIEW] FInish mode 3 and add tests

* Improve logging

* Update dependencies

* Update pkg/apiserver/rest/dualwriter_mode3_test.go

Co-authored-by: maicon <maiconscosta@gmail.com>

* remove test assertion

* Use mode log when dual writer is initiated

---------

Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
Co-authored-by: gonvee <gonvee@qq.com>
Co-authored-by: maicon <maiconscosta@gmail.com>
2024-07-24 09:28:14 +02:00
93221f12da Unified Storage: Fix Compare method (#90455)
* Unified Storage: Fix Compare method

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2024-07-16 11:36:21 -03:00
66a384247c Storage: Fix list in mode 4 (#90354) 2024-07-12 08:20:13 -04:00
d2bc4f3255 DualWrite: Cleanup and centralize the dual write creation (#90013) 2024-07-11 23:23:31 +03:00
d0afa60feb Remove unused metric (#90327) 2024-07-11 15:31:37 +02:00
b75276c641 Fix outcome metric (#90322) 2024-07-11 13:50:20 +02:00