20 Commits

Author SHA1 Message Date
4adebd6058 unified-storage: setup ring to shard requests (#103783)
* Updates the instrumentation_server service to use mux instead of the builtin router, and have it store the router in the module server: this is so we can register the /ring endpoint to check the status of the ring
* Create a new Ring service that depends on the instrumentation server and declares it as a dependency for the storage server
* Create standalone MemberlistKV service for Ring service to use
* Update the storage server Search and GetStats handler to distribute requests if applicable
2025-04-25 13:08:44 -04:00
7b492d7e16 FEMT: Add feature toggle and expose the service in regular grafana (#104428) 2025-04-25 12:24:25 +03:00
a8aa6b74a8 FEMT: Basic frontend-service implementation (#104229)
* create the most basic frontend-server module

* expose prom metrics??

* add todo list

* move frontend-service to its own folder in services

* check error from writer.Write

* reword comment, add launch config
2025-04-24 13:14:21 +01:00
216b6e96a9 fix(unified-storage): update instrumentation_server metric gatherer (#102399)
* update instrumentation_server used by the module server to use prometheus gatherer from wire instead of DefaultRegisterer
2025-03-19 08:15:13 -04:00
31a371e385 fix(unified-storage): remove bleve index metric from global scope (#101825)
* refactor grafana_index_server_index_size to calculate in a goroutine instead of at scrape time and remove grafana_index_server_indexed_docs metric

* use wire to inject bleve index metrics

* remove sprinkles metrics from bleve index metrics

* log error when trying to calculate file index size and bump interval to 1m instead of 5s
2025-03-13 10:09:38 -04:00
f5e5824bab fix (unified-storage): stop registering unified storage metrics in global state (#101322)
* move prometheus.register for unified storage metrics into metrics.go and do most of the plumbing to get it to work

* convert StorageApiMetrics to pointer and check for nil before using it

* rename type and variables to something more sensible

---------

Co-authored-by: Jean-Philippe Quéméner <jeanphilippe.quemener@grafana.com>
2025-02-28 07:39:39 -05:00
4837585cab Unified Storage Search: Add sprinkles (#97820)
* Wire up sprinkles to oss and enterprise. Fetching sprinkles not implemented yet.

* Adds wireset for initializing document builders. Had to init it when creating the service to avoid cyclical imports.

* updates to int64 for stats

* adds config for sprinklesApiServer and gets sprinkles from there when its present

* add comment for later

* adds feature toggle for sprinkles. returns empty results when flag not enabled.

* adds unified storage config setting for sprinkles apiserver page limit

* fixes bug where dashboard uid was not getting set

* when creating dashboard summary, use metadata.name as the dashboard uid

* cleans up wire. use existing oss and enterprise sets to generate doc builders

* remove old wireset

* fix linter - adds missing arg for doc builders

* update dashboard stats in tests

* updates test-data dashboards

* log a warning instead of returning an error if we can't get sprinkles for a namespace

* dont read uid from dashboard json
2024-12-18 11:00:54 -06:00
3457f219be Storage: Add blob storage interfaces (#90932)
Co-authored-by: Jean-Philippe Quémémer <jeanphilippe.quemener@grafana.com>
2024-10-17 12:18:29 +02:00
7efadb0a00 Testing: Integration tests for unified-grpc start a local gRPC server (#93201) 2024-09-11 11:50:14 +03:00
4baca6947d ResourceServer: make the resource store the default unified storage backend (#90899)
* make the resource store the default unified storage backend

* add integration tests

* fix test non passing

* Update pkg/storage/unified/sql/test/integration_test.go

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* lint

* fix tests

* fix no rows

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2024-07-25 18:17:39 +02:00
606a74d0af Zanzana: Initial work to run openFGA as embedded or standalone service (#89211)
* Zanana: Initial work to run zanana as ebeddedn or standalone

* Add addr settings for when remote client is used.

* sync dependencies

* Lock mysql driver version
---------

Co-authored-by: Dan Cech <dcech@grafana.com>
2024-06-18 10:04:18 +02:00
5eecc01123 VSCode: Launch Grafana with Storage server (#88351)
* VSCode: Launch Grafana with Storage server

* Fix module_server_test
2024-05-29 10:02:35 +02:00
dd0e646e92 k8s: remove dependency on app_mode development (#85474)
* k8s: remove dependency on app_mode development

* remove dev conditions on storage server

* remove logic to drop entity tables
2024-05-03 11:28:15 +03:00
37d39de36d Storage Api: Add metrics (#85316)
* Storage server runs own instrumentation server if its the sole target. Starts adding some sample metrics for now.

* adds metric for failed optimistic locks

* refactors metrics registration to own method on service for testability. Adds tests.

* Register sql storage server metrics from within the service

* fixes test

* troubleshooting drone test failures. Maybe timing when starting instrumentation server?

* Waits until instrumentation server has started. Updates tests.

* defer wont get called unless theres an error. removing.

* wait for instrumentation server to be running

* linter - close res body

* use port 3000 for metrics and removes test metric inc() call

* fixes test - updates port

* refactors module server to provide an instrumentation server module when there is no ALL or CORE target provided and running as single target

* make instrumentation server a dependency of all modules that do not run their own http server

* adds module server test

* adds tests for instrumentation service and removes old tests that aren't needed

* ignore error in test

* uses helper to start and run service

* when running wait on ctx done or http server err

* wait for http server

* removes println

* updates module server test to be integration test

* require no error in goroutine

* skips integration test when GRAFANA_TEST_DB not defined

* move http server start into start, verify returned content

* make test error when run fails

* try waiting longer and see if drone tests pass

* update integration test mysql creds to match drone

* go back to only waiting half second

* debug log drone mysql connection string

* use same db connection config as drone

* try using same hostname as drone

* cant use localhost as mysql hostname in drone tests. Need to parse it from the cfg db connection string

---------

Co-authored-by: Dan Cech <dcech@grafana.com>
2024-04-08 08:35:01 -06:00
c4c9bfaf2e Storage: Unified Storage based on Entity API (#71977)
* first round of entityapi updates

- quote column names and clean up insert/update queries
- replace grn with guid
- streamline table structure

fixes

streamline entity history

move EntitySummary into proto

remove EntitySummary

add guid to json

fix tests

change DB_Uuid to DB_NVarchar

fix folder test

convert interface to any

more cleanup

start entity store under grafana-apiserver dskit target

CRUD working, kind of

rough cut of wiring entity api to kube-apiserver

fake grafana user in context

add key to entity

list working

revert unnecessary changes

move entity storage files to their own package, clean up

use accessor to read/write grafana annotations

implement separate Create and Update functions

* go mod tidy

* switch from Kind to resource

* basic grpc storage server

* basic support for grpc entity store

* don't connect to database unless it's needed, pass user identity over grpc

* support getting user from k8s context, fix some mysql issues

* assign owner to snowflake dependency

* switch from ulid to uuid for guids

* cleanup, rename Search to List

* remove entityListResult

* EntityAPI: remove extra user abstraction (#79033)

* remove extra user abstraction

* add test stub (but

* move grpc context setup into client wrapper, fix lint issue

* remove unused constants

* remove custom json stuff

* basic list filtering, add todo

* change target to storage-server, allow entityStore flag in prod mode

* fix issue with Update

* EntityAPI: make test work, need to resolve expected differences (#79123)

* make test work, need to resolve expected differences

* remove the fields not supported by legacy

* sanitize out the bits legacy does not support

* sanitize out the bits legacy does not support

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* update feature toggle generated files

* remove unused http headers

* update feature flag strategy

* devmode

* update readme

* spelling

* readme

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2023-12-06 15:21:21 -05:00
440f9a6ffb K8s: Register apiserver as background service, and list real playlists (#75338)
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2023-09-25 15:31:58 -07:00
fa229661e8 chore: use setting.Dev in place of "development" string (#74391)
chore: use development variable instead of string
2023-09-05 13:16:14 -04:00
cc1205d6ba fix: revert module server initialization from CLI (#74386)
not-a-revert reversion of module server initialization
2023-09-05 11:45:56 -04:00
32defdb6bf Fix: make apiserver work behind a feature toggle (#73891)
Co-authored-by: Charandas Batra <charandas.batra@grafana.com>
2023-09-01 22:31:51 +03:00
0de2c9eb96 feat: add ability to launch targeted dskit modules in the grafana server CLI command (#74188)
* feat: add ability to launch targeted dskit modules in the grafana server CLI command

This commit adds a ModuleServer and ModuleRunner suitable for launching dskit services and updates the server cli command to use this instead of the full Server. The default behavior is unchanged and will launch the full Grafana server. Individual services are targeted by setting target=comma,seperated,list in the config file.

* require dev mode to target dskit modules

* remove unused type

* replace setting.CommandLineArgs w/setting.Cfg; the caller can deal with calling setting.NewCfg

* Update pkg/server/module_server.go

Co-authored-by: Serge Zaitsev <serge.zaitsev@grafana.com>

---------

Co-authored-by: Serge Zaitsev <serge.zaitsev@grafana.com>
2023-09-01 08:09:54 -04:00