257 Commits

Author SHA1 Message Date
87c81825b7 K8s: Move standalone apiserver CLI to enterprise (#93799) 2024-09-27 09:04:35 +03:00
a21a232a8e Revert read replica POC (#93551)
* Revert "chore: add replDB to team service (#91799)"

This reverts commit c6ae2d7999aa6fc797db39e9d66c6fea70278f83.

* Revert "experiment: use read replica for Get and Find Dashboards (#91706)"

This reverts commit 54177ca619dbb5ded2dcb158405802d8dbdbc982.

* Revert "QuotaService: refactor to use ReplDB for Get queries (#91333)"

This reverts commit 299c142f6a6e8c5673cfdea9f87b56ac304f9834.

* Revert "refactor replCfg to look more like plugins/plugin config (#91142)"

This reverts commit ac0b4bb34d495914cbe8daad85b7c75c31e8070d.

* Revert "chore (replstore): fix registration with multiple sql drivers, again (#90990)"

This reverts commit daedb358dded00d349d9fac6106aaaa6bf18322e.

* Revert "Chore (sqlstore): add validation and testing for repl config (#90683)"

This reverts commit af19f039b62d9945377292a8e679ee258fd56b3d.

* Revert "ReplStore: Add support for round robin load balancing between multiple read replicas (#90530)"

This reverts commit 27b52b1507f5218a7b38046b4d96bc004d949d46.

* Revert "DashboardStore: Use ReplDB and get dashboard quotas from the ReadReplica (#90235)"

This reverts commit 8a6107cd35f6444c0674ee4230d3d6bcfbbd4a58.

* Revert "accesscontrol service read replica (#89963)"

This reverts commit 77a4869fcadf13827d76d5767d4de74812d6dd6d.

* Revert "Fix: add mapping for the new mysqlRepl driver (#89551)"

This reverts commit ab5a079bcc5b0f0a6929f0a3742eb2859d4a3498.

* Revert "fix: sql instrumentation dual registration error (#89508)"

This reverts commit d988f5c3b064fade6e96511e0024190c22d48e50.

* Revert "Experimental Feature Toggle: databaseReadReplica (#89232)"

This reverts commit 50244ed4a1435cbf3e3c87d4af34fd7937f7c259.
2024-09-25 15:21:39 -08:00
1ede1e32b8 Alerting: Receiver resource permissions service (#93552) 2024-09-20 18:31:42 -04:00
45eb72e95a K8s: Use wire to initalize the resource client (#93221) 2024-09-12 17:22:27 +03:00
8988e04044 RBAC: Add permission registry (#91247)
* RBAC: Permission registry

* Populate permission registry

* Wire

* conflic_user_cmd

* Update pkg/services/accesscontrol/permreg/permreg_test.go

Co-authored-by: Ieva <ieva.vasiljeva@grafana.com>

* PR feedback

Co-authored-by: Ieva <ieva.vasiljeva@grafana.com>

* Remove ToDo, tackle in subsequent PR

---------

Co-authored-by: Ieva <ieva.vasiljeva@grafana.com>
2024-08-02 09:32:06 +02:00
160fe2a3a4 Storage: Remove entity store (#91233) 2024-07-31 09:25:39 +03:00
4c7d8c8cdd RBAC: Refactor RBAC plugin interface code (#90679)
move plugin RBAC registration ifaces to pluginsintegrations
2024-07-22 09:31:36 +01:00
248af65f9c Actionsets: Add ability for plugins to add actions for core actionsets (i.e. folders:edit) (#88776)
* initial commit

* Action sets stored
remove the dependancy for actionsets
got the actionsets registered
storing the permissions

* fix golanglinting

* remove unused struct field

* wip

* actionset registry for a plugin from the actionsetservice

* update to make declareactionset the primary way of plugin registration and modification

* declare actually extends actionsets

* tests fixed

* tests skipped

* skip tests

* skip tests

* skip tests

* skip tests

* change to warning instead

* remove step from pipeline to see if it fails due to plugin not registering

* reintroduce step but remove features dependancy

* add back the tests that were failing

* remove comments and another skip test

* fix a comment and remove unneeded changes

* fix and clean up, put the behaviour behind a feature toggle

* clean up

* fixing tests

* hard-code allowed action sets for plugins

* Apply suggestions from code review

Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>

* small cleanup

---------

Co-authored-by: IevaVasiljeva <ieva.vasiljeva@grafana.com>
Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>
2024-07-19 16:16:23 +01:00
c210617735 K8s: use contexthandler in standalone handler chain (#90102) 2024-07-08 12:22:10 -07:00
77a4869fca accesscontrol service read replica (#89963)
* accesscontrol service read replica
* now using the ReplDB interface
* ReadReplica for GetUser
2024-07-08 10:00:13 -04:00
50244ed4a1 Experimental Feature Toggle: databaseReadReplica (#89232)
This adds a version of the SQLStore that includes a ReadReplica. The primary DB can be accessed directly - from the caller's standpoint, there is no difference between the SQLStore and ReplStore unless they wish to explicitly call the ReadReplica() and use that for the DB sessions.

Currently only the stats service GetSystemStats and GetAdminStats are using the ReadReplica(); if it's misconfigured or if the databaseReadReplica feature flag is not turned on, it will fall back to the usual (SQLStore) behavior.

Testing requires a database and read replica - the replication should already be configured. I have been testing this locally with a docker mysql setup (https://medium.com/@vbabak/docker-mysql-master-slave-replication-setup-2ff553fceef2) and the following config:

[feature_toggles]
databaseReadReplica = true

[database]
type = mysql
name = grafana
user = grafana
password = password
host = 127.0.0.1:3306

[database_replica]
type = mysql
name = grafana
user = grafana
password = password
host = 127.0.0.1:3307
2024-06-18 11:07:15 -04:00
afcb5a855c AuthZ: embed an authorization server (#89018)
* AuthZ: embed an authorization server

* CODEOWNERS

* Remove swagger

* WIP

* Flatten structure and inject wireset

* sync mod files

* Rename authorization package

* Fix swagger gen

* CODEOWNERS

* Use itf instead of impl

---------

Co-authored-by: Karl Persson <kalle.persson@grafana.com>
2024-06-13 11:41:35 +02:00
0fffa31931 Chore: Remove unused code in legacydata package (#88772) 2024-06-05 15:37:32 +02:00
bd2b248f0e RBAC: Clean up action set code (#88147)
* remove unused action set code, refactor the existing code

* fix import ordering

* use a separate interface for permission expansion after all, to avoid circular dependencies

* add comments, fix a test
2024-05-23 12:14:01 +01:00
b8c9ae0eb7 OIDC: Support Generic OAuth org to role mappings (#87394)
* Social: link to OrgRoleMapper

* OIDC: support Generic Oauth org to role mappings

Fixes: #73448
Signed-off-by: Mathieu Parent <math.parent@gmail.com>

* Handle when getAllOrgs fails in the org_role_mapper

* Add more tests

* OIDC: ensure orgs are evaluated from API when not from token

Signed-off-by: Mathieu Parent <math.parent@gmail.com>

* OIDC: ensure AutoAssignOrg is applied with OrgMapping without RoleAttributeStrict

Signed-off-by: Mathieu Parent <math.parent@gmail.com>

* Extend docs

* Fix test, lint

---------

Signed-off-by: Mathieu Parent <math.parent@gmail.com>
Co-authored-by: Mihaly Gyongyosi <mgyongyosi@users.noreply.github.com>
2024-05-23 09:55:45 +02:00
3e77768144 RBAC: Expand action sets when fetching permissions (#87967)
* logic to expand action set to the underlying actions when permissions are fetched from the DB

* updates needed for dependency injection

* clean up some code, also deduplicate scopes when grouping scopes and actions

* expand on a comment

* rename a method
2024-05-21 15:09:26 +01:00
6ddb8aeae3 Chore: Moving to slog (folders) (#87000)
* moving folders to slog

* trying to fix the tests

* fix per-logger filters in slog

* allow using slog.Default()

* bring cfg back to keep the pr small

* fix tests

* back to the roots
2024-05-02 09:14:12 +02:00
ddabef9895 RBAC: Add actionsets struct and write path (#86108)
* Add actionsets struct and failing test

* update from review

* review comments

* review comments update

* refactor: create interface

* actionset service

* fix tests

* move from wireoss to wire

* Apply suggestions from code review

remove unnecessary comments

Co-authored-by: Ieva <ieva.vasiljeva@grafana.com>

* nil for the actionsetservice

* Revert "nil for the actionsetservice"

This reverts commit e3d3cc817180c1927dea430191f1e940f5c01272.

---------

Co-authored-by: Ieva <ieva.vasiljeva@grafana.com>
2024-04-19 15:38:14 +01:00
5dfe4cf407 Storage: Watch tests (#85496)
* basic watch tests working

* refactor to read previous event within poller

* add watch test files

* cleanup

* watch tests passing

* debug cleanup

* special handling for canceled context

* cleanup

* fix wire

* fix sqlite_sequence cleanup issue

* move watch tests to integration, wait for provisioned dashboards

* handle context deadline exceeded and eof errors

* add comment about sleep
2024-04-08 11:42:12 -04:00
faa1244518 Chore: Replace sqlstore with db interface (#85366)
* replace sqlstore with db interface in a few packages

* remove from stats

* remove sqlstore in admin test

* remove sqlstore from api plugin tests

* fix another createUser

* remove sqlstore in publicdashboards

* remove sqlstore from orgs

* clean up orguser test

* more clean up in sso

* clean up service accounts

* further cleanup

* more cleanup in accesscontrol

* last cleanup in accesscontrol

* clean up teams

* more removals

* split cfg from db in testenv

* few remaining fixes

* fix test with bus

* pass cfg for testing inside db as an option

* set query retries when no opts provided

* revert golden test data

* rebase and rollback
2024-04-04 15:04:47 +02:00
504870f10a Auth: Decouple client and hook registration (#85084) 2024-04-04 09:33:00 +02:00
6c1de260a2 API Server: Standalone observability (#84789)
Adds support for logs (specify level), metrics (enable metrics and Prometheus /metrics endpoint 
and traces (jaeger or otlp) for standalone API server. This will allow any grafana core service 
part of standalone apiserver to use logging, metrics and traces as normal.
2024-03-21 17:06:32 +01:00
29d4f6a217 Datasource: Change query filtering (#84656)
* call filterQuery from queryrunner

* test query hide filtering

* fix more broken tests

* lint errrors

* remove redundant filterQuery call

* skip filter in variable queries

* fix broken cypress test

* change tooltip text

* fix translations

* fix comments

* do not execute query is targets are empty

* add more tests

* remove unsued import

* update translations

* revert id change

* change header text

* update comment for hide prop

* rename hide query prop

* change tooltip and introduce different toggle state text

* update tests

* update comment and regenerate types

* run extract again

* fix broken e2e test

* track event

* fix build issues

* revert changes in wire file
2024-03-21 13:39:39 +01:00
473898e47c Core: Remove thema and kindsys dependencies (#84499)
* Move some thema code inside grafana

* Use new codegen instead of thema for core kinds

* Replace TS generator

* Use new generator for go types

* Remove thema from oapi generator

* Remove thema from generators

* Don't use kindsys/thema for core kinds

* Remove kindsys/thema from plugins

* Remove last thema related

* Remove most of cuectx and move utils_ts into codegen. It also deletes wire dependency

* Merge plugins generators

* Delete thema dependency 🎉

* Fix CODEOWNERS

* Fix package name

* Fix TS output names

* More path fixes

* Fix mod codeowners

* Use original plugin's name

* Remove kindsys dependency 🎉

* Modify oapi schema and create an apply function to fix elasticsearch errors

* cue.mod was deleted by mistake

* Fix TS panels

* sort imports

* Fixing elasticsearch output

* Downgrade oapi-codegen library

* Update output ts files

* More fixes

* Restore old elasticsearch generated file and skip its generation. Remove core imports into plugins

* More lint fixes

* Add codeowners

* restore embed.go file

* Fix embed.go
2024-03-21 11:11:29 +01:00
8765c48389 Alerting: Remove legacy alerting (#83671)
Removes legacy alerting, so long and thanks for all the fish! 🐟

---------

Co-authored-by: Matthew Jacobson <matthew.jacobson@grafana.com>
Co-authored-by: Sonia Aguilar <soniaAguilarPeiron@users.noreply.github.com>
Co-authored-by: Armand Grillet <armandgrillet@users.noreply.github.com>
Co-authored-by: William Wernert <rwwiv@users.noreply.github.com>
Co-authored-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
2024-03-14 15:36:35 +01:00
8d9521fb6d Refactor: Email verification (#84393)
* Update template names

* Add verifier that we can use to start verify process

* Use userVerifier when verifying email on update

* Add tests
---------

Co-authored-by: Ieva <ieva.vasiljeva@grafana.com>
2024-03-14 13:25:28 +01:00
d82f3be6f7 QueryService: Use types from sdk (#84029) 2024-03-08 18:12:59 +02:00
80447dd0cc Restructure cloudmigration service (#83211)
* Restructure cloudmigation service

* Adjust codewoners and wire

* Comment out unused metrics
2024-02-26 14:52:16 +01:00
80d6bf6da0 AuthN: Remove embedded oauth server (#83146)
* AuthN: Remove embedded oauth server

* Restore main

* go mod tidy

* Fix problem

* Remove permission intersection

* Fix test and lint

* Fix TestData test

* Revert to origin/main

* Update go.mod

* Update go.mod

* Update go.sum
2024-02-26 11:29:09 +01:00
790e1feb93 Chore: Update test database initialization (#81673)
* streamline initialization of test databases, support on-disk sqlite test db

* clean up test databases

* introduce testsuite helper

* use testsuite everywhere we use a test db

* update documentation

* improve error handling

* disable entity integration test until we can figure out locking error
2024-02-09 09:35:39 -05:00
91754bcda5 K8s: Refactor standalone apiserver initialization (#81932) 2024-02-06 08:40:35 -08:00
67b6be5515 K8s: Refactor config/options for aggregation (#81739) 2024-02-01 17:27:30 -05:00
77775d548a Chore: Add some tests for the Create method (#81364)
* Add test for create method
    Co-authored-by: Tania B <10127682+undef1nd@users.noreply.github.com>
* Change structure of entity package to break import cycle
* Update wire file

---------

Co-authored-by: Tania B <10127682+undef1nd@users.noreply.github.com>
2024-01-30 14:15:12 +01:00
c47b55ae10 Auth: Add SSO settings usage stats (#81143)
* Add usage stats

* UsageStats test + svc rename

* Fix test
2024-01-24 15:39:50 +01:00
cf13cb9f70 Cloud Migrations: Create new service for cloud migrations (#80949)
* introduce feature toggle

* create base service structure

* fix sample metric

* register metrics

* add to codeowners

* separate api dtos from service models

* remove leading newline
2024-01-22 11:09:08 -05:00
2a0f1900d5 Datasources: Add wireset for datasource.DataSourceAPIBuilder (#80914)
tidy up
2024-01-22 15:21:27 +01:00
50f4e78a39 Auth: Use SSO settings service to load social connectors + refactor (#79005)
* Refactor to prevent cyclic dependencies

* Move list authorization to the API layer

* Init connectors using the SSO settings service in case the ssoSettingsApi feature toggle is enabled

* wip, need to handle the cyclic dep

* Remove cyclic dependency

* Align tests + refactor

* Move back OAuthInfo to social

* Delete pkg/login/social/constants

* Move reloadable registration to the social providers

* Rename connectors.Error to connectors.SocialError
2023-12-08 11:20:42 +01:00
9849c954a3 Chore: remove the entity kind registry (#79178) 2023-12-06 14:00:53 -08:00
2a2a132c61 K8s: Refactor metrics to share k8s registry (#79106) 2023-12-06 16:34:23 -05:00
c4c9bfaf2e Storage: Unified Storage based on Entity API (#71977)
* first round of entityapi updates

- quote column names and clean up insert/update queries
- replace grn with guid
- streamline table structure

fixes

streamline entity history

move EntitySummary into proto

remove EntitySummary

add guid to json

fix tests

change DB_Uuid to DB_NVarchar

fix folder test

convert interface to any

more cleanup

start entity store under grafana-apiserver dskit target

CRUD working, kind of

rough cut of wiring entity api to kube-apiserver

fake grafana user in context

add key to entity

list working

revert unnecessary changes

move entity storage files to their own package, clean up

use accessor to read/write grafana annotations

implement separate Create and Update functions

* go mod tidy

* switch from Kind to resource

* basic grpc storage server

* basic support for grpc entity store

* don't connect to database unless it's needed, pass user identity over grpc

* support getting user from k8s context, fix some mysql issues

* assign owner to snowflake dependency

* switch from ulid to uuid for guids

* cleanup, rename Search to List

* remove entityListResult

* EntityAPI: remove extra user abstraction (#79033)

* remove extra user abstraction

* add test stub (but

* move grpc context setup into client wrapper, fix lint issue

* remove unused constants

* remove custom json stuff

* basic list filtering, add todo

* change target to storage-server, allow entityStore flag in prod mode

* fix issue with Update

* EntityAPI: make test work, need to resolve expected differences (#79123)

* make test work, need to resolve expected differences

* remove the fields not supported by legacy

* sanitize out the bits legacy does not support

* sanitize out the bits legacy does not support

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* update feature toggle generated files

* remove unused http headers

* update feature flag strategy

* devmode

* update readme

* spelling

* readme

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2023-12-06 15:21:21 -05:00
7e2aad1d01 Revert "K8s: Refactor metrics to share k8s registry (#77957)" (#79079)
This reverts commit 38bc41651a416bd547adbb284bc1559a5aad1006.
2023-12-05 09:34:07 -05:00
38bc41651a K8s: Refactor metrics to share k8s registry (#77957)
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2023-12-04 10:54:59 -08:00
529271d7a8 Perf: Disable core kind registry (#78568)
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2023-11-28 09:09:54 +01:00
1eb19befaa Login: refactor auth info package (#78459)
* Remove unused stats and metrics

* No longer collect metrics

* Remove unused dependency

* Move database from sub package
2023-11-21 14:47:23 +01:00
2d391c1bf5 postgresql: change plugin id (#77444)
* postgres: change plugin id

* fixed cue file

* codeowners update

* fixed backend test
2023-11-13 08:49:26 +01:00
5285e9503b Auth: SSO settings foundations (#77724)
* inital changes, db migration

* changes

* Implement basic GetAll, Delete

* Add first batch of tests

* Add more tests

* Add service tests for GetForProvider, List

* Update http_server.go + wire.go

* Lint + update fixed role

* Update CODEOWNERS

* Change API init

* Change roles, rename

* Review with @kalleep

* Revert a mistakenly changed part

* Updates based on @dmihai 's feedback

---------

Co-authored-by: Karl Persson <kalle.persson@grafana.com>
2023-11-08 10:50:01 +01:00
35c1ee9686 EntityStore: Remove http access (can use apiserver now) (#77602) 2023-11-03 08:14:51 -07:00
c50ada3a1a auth: wire service account proxy (#77215)
* Add interface verification compliance

* rework service account api to a provider

* wire the service accounts api

* rewire the implementation of sa srv for the proxy

---------

Co-authored-by: Misi <mgyongyosi@users.noreply.github.com>
2023-11-03 10:27:43 +01:00
3015e5921f Chore: Move extsvcaccounts package to serviceaccounts (#76977)
* Chore: Move extsvcaccounts package to serviceaccounts

* Fix proxy

* Fix tests

* Fix linting
2023-10-24 11:01:04 +02:00
82f3127e23 Alerting: Move legacy alert migration from sqlstore migration to service (#72702) 2023-10-12 13:43:10 +01:00