10 Commits

Author SHA1 Message Date
cacb9b8eaf Clean up metrics usage in dual writer syncer (#107309)
Create DualWriterMetrics once, and pass them around. Don't use prometheus.DefaultRegisterer.
2025-06-27 15:48:44 +02:00
0764ecb98d K8s/DualWriter: Use dualwriter package for all dual writers (#101393) 2025-03-01 21:26:14 +03:00
58457d41d3 K8s/DualWriter: Remove legacy interface (#101395) 2025-02-27 13:27:28 +03:00
aca024bcbb fix(unified-storage): use dedicated mocks per storage type (#100389) 2025-02-12 09:59:21 +01:00
a03652494c Dual Writer simplification (#93852)
* All objects should have an UID

* Now with a different error message

* Simplify create on DW 2: use the same object to write to both storages

* Run only one test

* Add check for status code

* Add name if it's not present in mode2

* Populate UID in legacy

* Remove logs and commented code

* Change dualwriter1

* Remove commented code

* Fix list test

* remove get on update from dualwriter 2

* Get object before updating. Better var renaming

* Finish rebasing

* Comment test

* Uncomment tests

* Update legacy first. Add preconditions

* Remove preconditions

* Fix update test

* copy RV from unified to legacy objects

* revert changes to playlist xorm store

* Improve logging. Add go routines for mode3

* Add tests for async funcs in mode3

* Lint

* Lint

* Lint. Start to fix tests

* Fix watcher tests

* Fix store tests

* Fiinish fixing watcher tests

* Fix server tests

* add name check

* Update pkg/apiserver/rest/dualwriter_mode1.go

Co-authored-by: Bruno Abrantes <bruno.abrantes@grafana.com>

* All objects should have an UID

* Now with a different error message

* Simplify create on DW 2: use the same object to write to both storages

* Run only one test

* Add check for status code

* Add name if it's not present in mode2

* Populate UID in legacy

* Remove logs and commented code

* Change dualwriter1

* Remove commented code

* Fix list test

* remove get on update from dualwriter 2

* Get object before updating. Better var renaming

* Finish rebasing

* Comment test

* Uncomment tests

* Fix update test

* revert changes to playlist xorm store

* Improve logging. Add go routines for mode3

* Lint

* Fix watcher tests

* Fiinish fixing watcher tests

* Add mode 5 with etcd test case. Add early check to fail on populated RV in payload

* we can't set RV to the found object when updating

* Lint

* Don't fail on update playlists

* Name should not be different when updating and it should be not empty on creating

* Fix tests

* Update pkg/apiserver/rest/dualwriter_mode2.go

Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>

* Lint

* Fix mode 5 tests

* Lint

* Add generateName condition on every mode. Fix tests

* Lint

* Add condition on where name or generate name have to be set

* Fix test

* Lint

* Fix folders test

* We dont need to send name for mode1

* Fail if UID is not present

* Remove change from not running test

* Remove unused line

* Lint

* Update pkg/storage/unified/apistore/store.go

Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>

* Improve error message

* Fix broken watcher test

* Fail on name mismatch on update

* Remove log

* Make sure UIDs match on create in both stores

* Lint

* Write first to unified storage

* Remove uid setting

* Remove RV only in mode2

* Fix test. Remove log line

* test

* No need to asser on RV in mode3

* Remove RV check due to race condition

* Update dualwriter.go

Co-authored-by: Georges Chaudy <chaudyg@gmail.com>

* Update pkg/storage/unified/client.go

* remove unused parameter

* log an error for object is missing UID instead of returning an error

* remove obj.SetResourceVersion("")

* log an error for object is missing UID instead of returning an error

* FInalise merge

* Move RV check to where it was

* Remove name check

* Remove server check for backwards compatibility

* Remove unused fn

* Move test checks for another PR

* Dont commit go work sum changes

* Only log error if RV is present for now.

---------

Co-authored-by: Todd Treece <todd.treece@grafana.com>
Co-authored-by: Bruno Abrantes <bruno.abrantes@grafana.com>
Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>
Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2024-10-23 10:29:41 +02:00
d60351d8e8 Unistore: fix DualWriter context cancelation on mode 1 (#93377)
* Unistore: fix DualWriter context cancelation on mode 1

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2024-09-23 10:34:36 -03:00
502bd5612e Storage: Extract returned object hydration into function (#88012)
* Extract returned object hydration into function

* Finish writing tests for utils func

* Lint

* Update pkg/apiserver/rest/dualwriter_mode2.go

Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>

* Better var naming

* Remove duplicated logic

* Lint

* Fix test

* Lint

* Make type private

* Fix one more test

* Fix test

---------

Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
2024-05-28 15:14:27 +02:00
9e6de035c0 Storage: Add mode 2 dual writing improvements (#87204)
* Fix mode 2 List test
* Set origin timestamp during conversion to k8s resource
* Add instructions for updating a playlist
* Handle partial deletions of a collection in mode 2
2024-05-02 16:06:51 +02:00
9baf96dbd2 Storage: dualwriter update implementation (#85844)
* Add update methods for the dual writer

* improve errors

* [WIP] add tests for the update method

* Move example package to its own package so it can be used by the rest package. Finish tests

* Add codeowners

* Use Pod as a dummy resource

* :int

* Lint

* [REVIEW] rename var

* [REVIEW] don't rely on legacy storage at all in mode4

* Update pkg/apiserver/rest/dualwriter_mode2.go

Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>

* [REVIEW] improve comment

* Update pkg/apiserver/rest/dualwriter_mode1.go

Co-authored-by: Dan Cech <dcech@grafana.com>

* [REVIEW] improve mode3

* Lint

* Move test files

* Update pkg/apiserver/rest/dualwriter_mode2.go

Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>

* Lint

* Update pkg/apiserver/rest/dualwriter_mode4_test.go

Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>

* Fix error

* Lint

* Update pkg/apiserver/rest/dualwriter_mode2.go

Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>

* Don't set the flag to true as updatedObj creates an object in case it's not found

* Lint

* Lint

* Add tests on update

* Lint

---------

Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
Co-authored-by: Dan Cech <dcech@grafana.com>
2024-04-17 08:32:34 -06:00
dcfaf7e434 Storage: Add dualwriter test mocks (#86243)
* WIP: add mocks for dual writer

* Test dualwriter mode1

* Re-add non implementation errors

* Use testify assert

* Write tests for mode2.

* Lint

* Remove comment

* Update pkg/apiserver/rest/dualwriter_mode2_test.go

Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>

* [REVIEW improve readability and call fn legacy fn

* [REVIEW] rename mocks and setupFns

* [REVIEW add missing test case

* [REVIEW] add test case with getting object from storage and not legacy

* Add deleteCollection

* Test deleteCollection

* Rename file

---------

Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
2024-04-17 11:25:19 +02:00