435 Commits

Author SHA1 Message Date
0e7b041b27 Secrets: Move decrypt types to contracts and export public at root pkg (#108376)
* Secrets: Move decrypt types to contracts and export public at root pkg

* Provisioning: Replace decrypt pkg imports

* Merge wire changes
2025-07-23 12:14:28 +02:00
1bd9541507 FrontendService: Add tracing and logging middleware (#107956)
* FrontendService: Add tracing and logging middleware

* tests!

* middleware tests

* context middleware test

* revert http_server back to previous version

* fix lint

* fix test

* use http.NotFound instead of custom http handler

* use existing tracer for package

* use otel/trace.Tracer in request_tracing middleware

* tidy up tracing in contextMiddleware

* fix 404 test

* remove spans from contextMiddleware

* comment
2025-07-22 18:55:44 +01:00
34f1be796d unified-storage: expose ring replication factor config (#106345)
* config ring replication factor

* change default

* rename

* fix test

* fix
2025-07-22 09:32:22 +02:00
e5d2f92384 K8s: Add App SDK installer (#107773) 2025-07-21 13:32:15 -04:00
f7e55f2c5d Secrets: Add authz checks for the single-tenant SecureValue client (#108216) 2025-07-21 11:09:07 +02:00
3fad863fd1 Query Service: Combine SSE handling in single tenant and multi tenant paths (#108041)
* parse via sse

I need to figure out how to handle the pipeline.execute with our own
client. I think this is important for MT reasons, just like using our
own cache (via legacy) is important.

parsing is done though!

* WIP nonsense

* horrible code but i think it works

* Add support for sql expressions config settings

* Cleanup:
- remove spew from nodes.go
- uncomment out plugin context and use in single tenant flow
- make code more readable and add comments

* Cleanup:
- create separate file for mt ds client builder
- ensure error handling is the same for both expressions and regular queries
- other cleanup

* not working but good thoughts

* WIP, vector not working for non sse

* super hacky but i think vectors work now

* delete delete delete

* Comments for future ref

* break out query handling and start test

* add prom debugger

* clean up: remove comments and commented out bits

* fix query_test

* add prom debugger

* create table-driven tests with testsdata files

* Fix test

* Add test

* go mod??

* idk

* Remove comment

* go enterprise issue maybe

* Fix codeowners

* Delete

* Remove test data

* Clean up

* logger

* Remove go changes hopefully

* idk go man

* sad

* idk i ran go mod tidy and this is what it wants

* Fix readme, with much help from adam

* some linting and testing errors

* lint

* fix lint

* fix lint register.go

* another lint

* address lint in test

* fix dead code and linters for query_test

* Go mod?

* Struggling with go mod

* Fix test

* Fix another test

* Revert headers change

* Its difficult to test this in OSS as it depends on functionality defined in enterprise, let's bring these tests back in some form in enterprise

* Fix codeowners

---------

Co-authored-by: Adam Simpson <adam@adamsimpson.net>
2025-07-17 17:22:55 -04:00
f6ed9e6ff0 Plugins: Introduce plugin asset provider (#108063)
* introduce plugin asset provider

* simply with PR feedback

* fix linter
2025-07-17 16:20:35 +01:00
810868c156 Provisioning: use new secure value client (#108227)
* Use provider in Webhooks registration

* First stab at using the new client

* Simplify mock

* Able to generate graph in OSS

* Remove line already ensured by provider

* Handle the K8s not found error as well

* Commit regenerated wire file

* Add the hack also for deletion

* Fix secrets test util

* Format file
2025-07-17 15:17:14 +02:00
b232ba5396 Provisioning: Move provisioning extras to OSS extensions (#108167)
* Move provisioning extensions to OSS

* Fix the wire stuff

* Move it to extras packages

* Add hack to assert test differently for enterprise
2025-07-16 13:24:24 +02:00
2c4bbf8b1d Secrets: Reduce amount of feature toggle checks to only app entrypoint (#108110)
* Secrets: Reduce amount of feature toggle checks to only app entrypoint

* Wire: Fix merge conflict

* Wire: Fix merge conflict
2025-07-16 10:56:59 +02:00
3cda233468 Secrets: Remove temporary allowlist (#108118) 2025-07-16 09:39:33 +02:00
ab51794bdb SecretsManager: Unify KMS across OSS and Enterprise (#108085)
* everything is compiling

* tests passing

* remove used object

* write a test for secret key upgrades

* misc cleanup

* clean up some wording

* lint issues

* fix a typo

* import hashicorp dependency explicitly

* simplify oss kmsprovider package structure

* consolidate current provider and available providers

* add a new manager configuration test

* fix hashivault import

* fix import issue

* fix unit tests

* Update go.mod

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

---------

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-07-15 14:39:20 -04:00
d39a47a89b Provisioning: Begin using secrets store (#108044)
- Provisioning: Begin using secrets store
- Refactor integration with secrets store
- Add back the legacy service
- Separate concerns for encrypt and decrypt
- Handle update within Encrypt function
- Add interface for secure value service
- Add feature flag for using secrets service
- Add the dual service for temporary solution.

* Add first integration tests for encrypted tokens
* Add integration test for app platform secrets
* Validate it has the name or not
* Create wire provider
* Always save to the secret if provided secret

---------

Co-authored-by: Roberto Jimenez Sanchez <roberto.jimenez@grafana.com>
Co-authored-by: Roberto Jiménez Sánchez <jszroberto@gmail.com>
2025-07-15 15:43:17 +02:00
4954822cbb Secrets: Add separate package for resource validation (#108097)
* Secrets: Add SecureValueService interface

* Secrets: Move resource validators to their own package/structs for reusing
2025-07-15 12:33:14 +02:00
e079cb3738 Simplify open feature setup (#107632)
* Simplify open feature setup

* Fix linter issues

* Apply review feedback

* Fix integration tests setup
2025-07-14 16:22:04 +02:00
1912c4ccc3 Regenerate wire 2025-07-11 23:02:19 +02:00
1f3dc0533c Auth: Add tracing to auth clients and AuthToken service (#107878)
* Add tracing to auth clients + authtoken svc

* Fix span names

* Fix ext_jwt.go

* Fix idimpl/service

* Update wire_gen.go

* Add tracing to JWT client

* Lint
2025-07-10 15:41:00 +02:00
37bfea8685 FrontendService: Add tracing to frontend service (#107236)
* FrontendService: Add tracing to frontend service

* wire gen

* fix test
2025-07-09 19:21:31 +01:00
8283d35e56 Secrets: make operations sync (#107732)
* Secrets: make operations sync

* k8s gen / update query to list secure values to include the version

* always store new version of a secret

* make update-workspace

* go mod tidy

* update queries

* update queries

* improve and use testutils in decrypt_store_test

* fix broken test

* make update-workspace

* ./hack/update-codegen.sh secret

* update Test_SecureValueMetadataStorage_CreateAndRead

* undo dependency changes

* linter: fix remaining errors

---------

Co-authored-by: Matheus Macabu <macabu.matheus@gmail.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-07-09 10:43:34 -03:00
1244f5c53d AuthZ: Add CoreRoleStorageBackend to the extension set (#103912)
* Wiring

* Trigger CI

---------

Co-authored-by: mohammad-hamid <mohammad.hamid@grafana.com>
2025-07-07 20:08:44 +02:00
46c38fdbb7 SecretsManager: Introduce worker and secret async service (#107614)
SecretsManager: Introduce worker and secret aysnc service

Co-authored-by: PoorlyDefinedBehaviour <brunotj2015@hotmail.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
2025-07-04 13:13:48 +01:00
15e1aa8855 SecretsManager: Introduce decrypt store (#107586)
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
Co-authored-by: PoorlyDefinedBehaviour <brunotj2015@hotmail.com>
2025-07-04 09:22:10 +01:00
4d8678c7f2 SecretsManager: Add base encryption manager (#107562)
Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-07-03 11:29:14 +01:00
7614089077 Secrets: Only register dependencies to start up (#107504) 2025-07-02 14:02:52 +02:00
f09e85c048 unified-storage: Distributor rename to better reflect that it'll be used for search (#107409)
* rename distributor/ring references to "storage-api" to "search-server"
2025-07-01 11:15:10 -04:00
d76e55371a fix(unified-storage): use the provided connection config parameters (#107455)
* fix(unified-storage): use the provided connection config parameters

* extend tests

* make update-workspace
2025-07-01 11:25:16 +00:00
406923f912 Chore: Commit wire_gen.go (#107410)
* commit wire_gen.go

* do not generate code in ci
2025-07-01 11:56:27 +02:00
974a2c47f9 feat(unified-storage): add qos support for the resource server (#105939) 2025-07-01 09:22:55 +00:00
0fccc01ebe SecretsManager: add data key store (#107396)
* SecretsManager: Add data key store

Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
Co-authored-by: Dana Axinte <53751979+dana-axinte@users.noreply.github.com>

* SecretsManager: Add wiring of data key store

Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
Co-authored-by: Dana Axinte <53751979+dana-axinte@users.noreply.github.com>

---------

Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-06-30 17:17:07 +01:00
7aad041f70 unified-storage: Add integration tests for distributor (#105771)
* Add integration tests for distributor
2025-06-30 09:31:58 -04:00
67a952c34e Implement OFREP compatible feature flag service (#105632)
* Add ofrep pkg

* api server: Use namespace from request in case user is not authenticated

* Add handlers to ofrep api builder

* Add NewOpenFeatureService to initialize mt apiserver

* allow specifying CA and insecure

* Compare namespace with eval ctx stackID

* Organize ofrep package

* Implement AllowedV0Alpha1Resources

* Revert folderimpl changes

* Handle default namespace

* Fix extracting stack id from eval ctx

* Add more logs

* Update pkg/registry/apis/ofrep/register.go

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>

* Update pkg/registry/apis/ofrep/register.go

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>

* Apply review feedback

* Replace contexthandler with types

* Fix identifying authed request

* Refactor checks in the handlers

* Remove anonymous from isAuthenticatedRequest check

---------

Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>
Co-authored-by: Gabriel Mabille <gabriel.mabille@grafana.com>
Co-authored-by: Charandas Batra <charandas.batra@grafana.com>
Co-authored-by: Dave Henderson <dave.henderson@grafana.com>
2025-06-27 18:30:38 +02:00
6f3b619f59 Remove use of prometheus.DefaultRegisterer from ProvideUnifiedStorageGrpcService. (#107315) 2025-06-27 15:58:00 +02:00
dc79e6f3dc unified-storage: fix auth handling in distributor (#106062)
* disable auth handling in the distributor
2025-06-26 12:32:59 -04:00
de28231f2f SecretsManager: Add outbox store (#106613)
SecretsManager: add outbox store

Co-authored-by: PoorlyDefinedBehaviour <brunotj2015@hotmail.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-06-12 13:31:48 +01:00
c22b4845bb SecretsManager: Add encrypted value store (#106607)
* SecretsManager: add encrypted value store

Co-authored-by: Dana Axinte <53751979+dana-axinte@users.noreply.github.com>
Co-authored-by: Leandro Deveikis <leandro.deveikis@gmail.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
Co-authored-by: PoorlyDefinedBehaviour <brunotj2015@hotmail.com>

* SecretsManager: wiring of encrypted value store

---------

Co-authored-by: Leandro Deveikis <leandro.deveikis@gmail.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
Co-authored-by: PoorlyDefinedBehaviour <brunotj2015@hotmail.com>
2025-06-12 11:52:01 +01:00
6fb0ed9886 Bump tempo to 20250529124718-87c2dc380cec (#106220)
* Bump tempo to 20250529124718-87c2dc380cec

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

* Add owners to pkg/build

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

* update workspace

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

* update pkg/build/go.sum

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

* update pkg/build/go.sum

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-05-29 23:43:13 -05:00
6e5e133f7d SecretsManager: Introduce db migrator with keeper table (#105538)
Co-authored-by: PoorlyDefinedBehaviour <brunotj2015@hotmail.com>
Co-authored-by: Leandro Deveikis <leandro.deveikis@gmail.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-05-21 15:02:51 +01:00
a7922912fe SecretsManager: Introduce secrets database wrapper (#105472)
SecretsManager: Introduce secret database wrapper

Co-authored-by: PoorlyDefinedBehaviour <brunotj2015@hotmail.com>
Co-authored-by: Leandro Deveikis <leandro.deveikis@gmail.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-05-20 11:48:47 +01:00
91882cf50e chore(unified-storage): add tracing for unified storage folder implementation (#105601) 2025-05-19 23:25:08 +03:00
bdae4424e8 unified-storage: setup distributor module option A (#104737)
* setup distributor module

* move lifecycler into resource server provider

* remove ring/client pool setup from distributor module and use the same ring/client pool between storage server module and distributor module

* implement resourcestore server methods

* make healthcheck fail if ring is not running
2025-05-19 07:46:03 -04:00
2cddaf0781 RBAC: Remove dashboard and folder guardians (#104646)
* replace usage of folder guardians with access control evaluators

* remove NewByFolderUID guardian

* bring up to date

* fix test

* more test fixes, and don't fetch the folder before evaluating lib element access

* change what error is returned

* fix alerting test

* try to fix linter errors

* replace the use of newByFolder guardian with direct access control evaluator checks

* remove newByFolder guardian

* get rid of dashboard and folder guardians

* undo unwanted change

* undo unwanted change

* undo unwanted change

* update code owners
2025-05-19 09:48:16 +03:00
aa2cf8e398 Remove kube-aggregator from OSS (#103659)
* feat: remove kube-aggregator for OSS and provide injection points with runner iface

* upgrade authlib to support expiresIn

* new FT

* new FT again

* update go.mod

* get rid of the slice implementation

* reconcile conflicts

* gracefully handle enterprise not being linked situation with kubeAggregator FT true

* allow dataplane agg and kube agg to both be added to delegate chain

* make update-workspace

* address feedback

* revert go.mod changes

* go.mod updates

* elaborate on why and how of skipping the Ready channel handling

* after rebase and make run
2025-05-15 11:14:23 -07:00
581d153604 SecretsManager: Introduce DecryptAuthorizer (#105459)
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-05-15 17:52:14 +01:00
43748e43bb Plugins: Add plugins auto update feature (#104112) 2025-05-09 15:58:04 +02:00
82bbbf1a98 Run Init Provisioners at Server Initialization (#105080)
* Run Init Provisioners at Server Initialization

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-05-09 09:21:36 -03:00
ecea6a9366 unified-storage: expose memberlist cluster label config (#104643)
* tweak ringKey/ringName for consistency and allow us to configure memberlist cluster label config

* allow setting memberlist cluster label verification disabled field

---------

Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
2025-04-30 09:43:09 -04:00
4adebd6058 unified-storage: setup ring to shard requests (#103783)
* Updates the instrumentation_server service to use mux instead of the builtin router, and have it store the router in the module server: this is so we can register the /ring endpoint to check the status of the ring
* Create a new Ring service that depends on the instrumentation server and declares it as a dependency for the storage server
* Create standalone MemberlistKV service for Ring service to use
* Update the storage server Search and GetStats handler to distribute requests if applicable
2025-04-25 13:08:44 -04:00
7b492d7e16 FEMT: Add feature toggle and expose the service in regular grafana (#104428) 2025-04-25 12:24:25 +03:00
a8aa6b74a8 FEMT: Basic frontend-service implementation (#104229)
* create the most basic frontend-server module

* expose prom metrics??

* add todo list

* move frontend-service to its own folder in services

* check error from writer.Write

* reword comment, add launch config
2025-04-24 13:14:21 +01:00
56b4e5670d fix(unified-storage): fix graceful termination for grafana target servers (#103520) 2025-04-08 11:45:10 +02:00