562 Commits

Author SHA1 Message Date
672b1711b0 Instrumentation: measure time until plugin request (#61279)
Signed-off-by: bergquist <carl.bergquist@gmail.com>
2023-01-11 13:19:06 +00:00
4db3b2fd5c Kindsys: Remove defs, Slot->SchemaInterface (#61069)
* kindsys: Remove defs, Slot->SchemaInterface

* Remove excess file

* Fix up tests

* Regenerate kinds report

* Final bits of cleanup

* Stop complaining, linter

* Update pkg/kindsys/kindcat_composable.cue

Co-authored-by: Tania <yalyna.ts@gmail.com>

Co-authored-by: Tania <yalyna.ts@gmail.com>
2023-01-06 12:37:32 -05:00
410b23c3d9 Use preferred package header for generated code (#61059)
* Use preferred package header for generated code

> To convey to humans and machine tools that code is generated, generated source should have a line that matches the following regular expression (in Go syntax):
> `^// Code generated .* DO NOT EDIT\.$`

https://pkg.go.dev/cmd/go#hdr-Generate_Go_files_by_processing_source

Co-authored-by: sam boyer <sdboyer@grafana.com>
Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* Regenerate files with updated header

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>
Co-authored-by: sam boyer <sdboyer@grafana.com>
2023-01-06 04:57:45 -05:00
8f29450594 Kindsys: Remove Raw kind category (#60992)
* Remove Raw references

* Remove more raws

* Re-generate files

* Remove raw folder from veneer

* Fix import

* Fix lint

* Bring back raw folder in grafana-schema

* Another lint

* Remove use of "Structured" word in kinds

* Delete unused function and remove some structured words

* Bunch more removals of structured name

Co-authored-by: sam boyer <sdboyer@grafana.com>
2023-01-05 13:54:42 -05:00
b8253ac3b9 deps: Update thema, use CUE fork (#60496)
* deps: Update thema, use CUE fork

* Remove superfluous openapi-codegen output

* Update thema again to catch cuetsy changes

* Rerun codegen with latest

* Latest thema, removes cuetsy update

* Latest of cuetsy

* Update type usage

* One last cuetsy rollback

* Update playlist summary tests

* More lint fixes

* Remove TS veneer changes for VariableHide

Co-authored-by: Clarity-89 <homes89@ukr.net>
2022-12-22 07:20:02 -05:00
96b7fb15de Plugins: Fix parsing of Azure settings for plugins (#60611)
Fix parsing of Azure settings for plugins
2022-12-21 13:37:33 +01:00
c35c689a96 Plugins: Automatically forward plugin request HTTP headers in outgoing HTTP requests (#60417)
Automatically forward core plugin request HTTP headers in outgoing HTTP requests. 
Core datasource plugin authors don't have to specifically handle forwarding of HTTP 
headers, e.g. do not have to "hardcode" the header-names in the datasource plugin, 
if not having custom needs.

Fixes #57065
2022-12-21 13:25:58 +01:00
48b33ab521 Plugins: Unsigned chromium file should not invalidate signature for Renderer plugin (#59104)
* Plugins: Unsigned chromium file should not invalidate signature for Renderer plugin

* fix test

* re-work solution
2022-12-19 12:46:27 +01:00
6478d0a5ef Plugins: Forward user header (X-Grafana-User) in backend plugin requests (#58646)
Grafana would forward the X-Grafana-User header to backend plugin request when 
dataproxy.send_user_header is enabled. In addition, X-Grafana-User will be automatically
forwarded in outgoing HTTP requests for core/builtin HTTP datasources. 
Use grafana-plugin-sdk-go v0.147.0.

Fixes #47734

Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
2022-12-15 15:28:25 +01:00
7bf7308ea5 Plugins: Remove connection/hop-by-hop request/response headers for call resource (#60077)
Removes request/response connection/hop headers for call resource in similar 
manner as Go's reverse proxy functions. Also removes Prometheus datasource 
custom call resource header manipulation in regards to hop-by-hop headers.

Fixes #60076
Ref #58646

Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
2022-12-12 10:27:53 +01:00
a5ace56be8 Plugins: Add username to datasource plugin logging (#59893)
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
Signed-off-by: bergquist <carl.bergquist@gmail.com>
2022-12-07 13:15:42 +01:00
76233f9997 Plugins: Unexport PluginDir field from PluginDTO (#59190)
* unexport pluginDir from dto

* more err checks

* tidy

* fix tests

* fix dboard file tests

* fix import

* fix tests

* apply PR feedback

* combine interfaces

* fix logs and clean up test

* filepath clean

* use fs.File

* rm explicit type
2022-12-02 13:46:55 +01:00
7f92f1df00 Schema: Refactor plugin code generation (#58901)
* wip

* wip

* almost there..

* wip - change so it can run.

* treelist is working.

* support CODEGEN_VERIFY env variable

* use log.fatal

* comment out old PluginTreeList code generation

* cleanup

* rename corelist package files

* fix makefile

* move pkg/codegen/pluggen.go to pkg/plugins/codegen

* copy and refactor files to pkg/plugins/codegen

* use pkg/plugins/codegen instead of pkg/codegen for core plugins code gen

* remove unneeded files

* remove unused code to resolve linting errors

* adapters first hack

* added flattener

* add back ignore build tags to go generate file

* cleaned up the code a bit.

* seems to work, needs to do some refactoring of the GoTypesJenns and TSTypesJenny.

* one more step, going to get upstream changes in this branch.

* working but need to run import tmpl in jenny_schemapath to have the proper imports.

* added header to generated files.

* added missing jenny.

* preventing plugins with multiple decls/schemas to insert multiple lines in corelist.

* fixed so we use Slot type from kindsys to detect if its group.

* adding a go jenny that only runs if the plugin has a backend.

* added version object to generated ts.

* generating the ts types with the same output as prior to this refactoring.

* removed code that is replaced by the jenny pattern.

* removed the go code that isn't used anymore.

* removed some more unused code and renamed pluggen to util_ts

* fixed linting issue.

* removed unused vars.

* use a jenny list postprocessor for header injection

* moved decl and decl_parser to pfs.

* removed the pre-pended header in the gotypes jenny since it is done in the postprocess.

* moved decl to pfs.

* removed unused template.

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2022-12-02 08:22:28 +01:00
6dbe3b555f Plugins: Refactor forward of cookies, OAuth token and header modifications by introducing client middlewares (#58132)
Adding support for backend plugin client middlewares. This allows headers in outgoing 
backend plugin and HTTP requests to be modified using client middlewares.

The following client middlewares added:
Forward cookies: Will forward incoming HTTP request Cookies to outgoing plugins.Client 
and HTTP requests if the datasource has enabled forwarding of cookies (keepCookies).
Forward OAuth token: Will set OAuth token headers on outgoing plugins.Client and HTTP 
requests if the datasource has enabled Forward OAuth Identity (oauthPassThru).
Clear auth headers: Will clear any outgoing HTTP headers that was part of the incoming 
HTTP request and used when authenticating to Grafana.
The current suggested way to register client middlewares is to have a separate package, 
pluginsintegration, responsible for bootstrap/instantiate the backend plugin client with 
middlewares and/or longer term bootstrap/instantiate plugin management. 

Fixes #54135
Related to #47734
Related to #57870
Related to #41623
Related to #57065
2022-12-01 19:08:36 +01:00
5b71a16acf Slugify: Replace gosimple/slug with a simple function (#59517) 2022-11-30 11:12:56 -05:00
8a09fd3257 Plugins: Fix app child config func typo (#59386)
fix func typo
2022-11-28 13:43:14 +01:00
b814c66c1d Perfomance: add preallocation for some slices (#59263)
add preallocation for some slices
2022-11-28 13:10:24 +01:00
0a7a5b13fe Heatmap: Remove legacy angular based implementation (#59249)
remove angular heatmap
2022-11-23 10:46:21 -08:00
4d8287b319 Performance: add preallocation for some slice/map (#57860)
This change preallocates slices and maps where the size of the data is known before the object is created.

Co-authored-by: Joe Blubaugh <joe.blubaugh@grafana.com>
2022-11-22 20:24:36 +08:00
c4528f9bd8 codejen: Update to latest codejen (#58866) 2022-11-16 15:08:01 -05:00
a8c48b6801 RBAC: Cover plugin includes (#57582)
* RBAC: Add action to plugin includes

* Adding the feature toggle check

* Cue update

* Extract include access control to method

* Suggestion to prevent log when RBAC is disabled

Co-authored-by: ievaVasiljeva <ieva.vasiljeva@grafana.com>

* Rename IsRBACReady to RequireRBACAction

Co-authored-by: ievaVasiljeva <ieva.vasiljeva@grafana.com>
2022-11-16 15:54:04 +01:00
78f0340031 plugindef: Move pluginmeta out of coremodels as standalone thema lineage (#56765)
* Get pluginmeta mostly moved over to pkg/plugins/plugindef

* Remove dead func

* Fix up pfs, use sync.Once in plugindef

* Update to latest thema

* Chase Endec->Codec conversion in Thema

* Comments on slash header gen; use ToSlash

* Also generate JSON schema for plugindef

* Generate JSON Schema as well

* Fix slot loading from kindsys cue decls

* Remove unused vars

* skip generating plugin.schema.json for now

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2022-11-15 14:48:31 +01:00
d999b5bda0 RBAC: remove redundant role name field from plugin role registrations (#58166)
* RBAC: Remove name from role registration

* Inline accesscontrol service

* test fix

* use fmt

Co-Authored-By: marefr <marcus.efraimsson@gmail.com>

Co-authored-by: marefr <marcus.efraimsson@gmail.com>
2022-11-15 09:51:40 +01:00
30fae33f66 RBAC: Allow role registration for plugins (#57387)
* Picking role registration from OnCall POC branch

* Fix test

* Remove include actions from this PR

* Removing unused permission

* Adding test to DeclarePluginRoles

* Add testcase to RegisterFixed role

* Additional test case

* Adding tests to validate plugins roles

* Add test to plugin loader

* Nit.

* Scuemata validation

* Changing the design to decouple accesscontrol from plugin management

Co-authored-by: Kalle Persson <kalle.persson@grafana.com>

* Fixing tests

Co-authored-by: Jguer <joao.guerreiro@grafana.com>

* Add missing files

Co-authored-by: Jguer <joao.guerreiro@grafana.com>

* Remove feature toggle check from loader

* Remove feature toggleimport

* Feedback

Co-Authored-By: marefr <marcus.efraimsson@gmail.com>

* Fix test'

* Make plugins.RoleRegistry interface typed

* Remove comment question

* No need for json tags anymore

* Nit. log

* Adding the schema validation

* Remove group to take plugin Name instead

* Revert sqlstore -> db

* Nit.

* Nit. on tests

Co-authored-by: ievaVasiljeva <ieva.vasiljeva@grafana.com>

* Update pkg/services/accesscontrol/plugins.go

Co-authored-by: Ieva <ieva.vasiljeva@grafana.com>

* Log message

Co-Authored-By: marefr <marcus.efraimsson@gmail.com>

* Log message

Co-Authored-By: marefr <marcus.efraimsson@gmail.com>

* Remove unecessary method. Update test name.

Co-authored-by: ievaVasiljeva <ieva.vasiljeva@grafana.com>

* Fix linting

* Update cue descriptions

* Fix test

Co-authored-by: Kalle Persson <kalle.persson@grafana.com>
Co-authored-by: Jguer <joao.guerreiro@grafana.com>
Co-authored-by: marefr <marcus.efraimsson@gmail.com>
Co-authored-by: ievaVasiljeva <ieva.vasiljeva@grafana.com>
2022-11-07 11:30:45 +01:00
9ea6a43089 Build: clean up and document integration test convention (#58170)
* clean up and document integration test convention

* clarify integration test conventions

* clean up integration tests that don't follow convention

* mark testIntegration* functions as helpers to avoid confusion
2022-11-04 10:14:21 -04:00
7996a1c8ad Chore: Update grafana-plugin-sdk-go to v0.142.0 (#58115)
* Chore: Update grafana-plugin-sdk-go to v0.142.0

* Update tests and golden files for 207 status code

* Chore: Move update flag definition at the top in response_parser_test.go

* retrigger

Co-authored-by: Will Browne <will.browne@grafana.com>
2022-11-03 18:34:27 +01:00
06705a49e2 Instrumentation: Log plugin and datasource info for each plugin request. (#54769)
Signed-off-by: bergquist <carl.bergquist@gmail.com>
2022-11-02 14:51:51 +01:00
0845ac2f53 Profiling: Add Phlare and Parca datasources (#57809)
* Add phlare datasource

* Rename

* Add parca

* Add self field to parca

* Make sure phlare works with add to dashboard flow

* Add profiling category and hide behind feature flag

* Update description and logos

* Update phlare icon

* Cleanup logging

* Clean up logging

* Fix for shift+enter

* onRunQuery to set label

* Update type naming

* Fix lint

* Fix test and quality issues

Co-authored-by: Joey Tawadrous <joey.tawadrous@grafana.com>
2022-10-28 13:33:37 +02:00
9f77bd4728 Plugins: Use public store instead of internal registry (#57631)
* Plugins: Use public store instead of internal registry

* update comments

* fix import

* fix test
2022-10-27 18:44:28 +02:00
6f8fcae01b [main] Plugin fixes (#57399)
* Plugins: Remove support for V1 manifests

* Plugins: Make proxy endpoints not leak sensitive HTTP headers

* Security: Fix do not forward login cookie in outgoing requests

(cherry picked from commit 4539c33fce5ef23badb08ebcbc09cb0cecb1f539)

Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
2022-10-21 13:54:55 +02:00
05709ce411 chore: remove sqlstore & mockstore dependencies from (most) packages (#57087)
* chore: add alias for InitTestDB and Session

Adds an alias for the sqlstore InitTestDB and Session, and updates tests using these to reduce dependencies on the sqlstore.Store.

* next pass of removing sqlstore imports
* last little bit
* remove mockstore where possible
2022-10-19 09:02:15 -04:00
e5a6547a94 coremodels: Update to latest Thema with generics (#56602)
* Update thema to latest

* Deal with s/Library/*Runtime/

* Commit new, working results of codegen
2022-10-11 09:45:07 +01:00
bf264d2f76 Query library: requiresDevMode dummy backend (#56466)
* query library - dummy backend

* fix tests

* dont explicitly marshall backend dataresponse

* skip integration tests

* null check for tests

* added query library to codeowners

* null check for tests

* lint
2022-10-07 11:31:45 -07:00
74c809f544 Plugins: Introduce new Flame graph panel (#56376)
* Flamegraph

* Updated flame graph width/height values

* Fix top table rendering issue

* Add feature toggle for flamegraph in explore

* Update tests

* Hide flamegraph from dash panel viz list if feature toggle not enabled

* Show table if no flameGraphFrames

* Add flame graph to testdata ds

* Minor improvement
2022-10-07 11:39:14 +01:00
1038ac7308 DataSources: Remove unused cfg from core Grafana data source (#56188)
* remove unused cfg from grafana ds

* fix test
2022-10-04 06:32:25 -04:00
4087ad413f re-use fake trace + feature toggles (#56186) 2022-10-04 10:15:10 +02:00
e31cb93ec0 NavTree: Make it possible to configure where in nav tree plugins live (#55484)
* NewIA: Plugin nav config

* progress

* Progress

* Things are working

* Add monitoring node

* Add alerts and incidents

* added experiment with standalone page

* Refactoring by adding a type for navtree root

* First test working

* More tests

* more tests

* Progress on richer config and sorting

* Sort weight working

* Path config

* Improving logic for not including admin or cfg nodes, making it the last step so that enterprise can add admin nodes without having to worry about the section not existing

* fixed index routes

* removed file

* Fixes

* Fixing tests

* Fixing more tests and adding support for weight config

* Updates

* Remove unused fake

* More fixes

* Minor tweak

* Minor fix

* Can now control position using sortweight even when existing items have no sortweight

* Added tests for frontend standalone page logic

* more tests

* Remove unused fake and fixed lint issue

* Moving reading settings to navtree impl package

* remove nav_id setting prefix

* Remove old test file

* Fix trailing newline

* Fixed bug with adding nil node

* fixing lint issue

* remove some code we have to rethink

* move read settings to PrivideService and switch to util.SplitString
2022-09-28 08:29:35 +02:00
701f6d5436 UserService: use the UserService instead of calling sqlstore directly (#55745)
* UserService: update callers to use the UserService instead of calling sqlstore directly

There is one major change hiding in this PR. UserService.Delete originally called a number of services to delete user-related records. I moved everything except the actual call to the user table, and moved those into the API. This was done to avoid dependencies cycles; many of our services depend on the user service, so the user service itself should have as few dependencies as possible.
2022-09-27 07:58:49 -04:00
d0d8544ded Plugins: Create single point of entry for adding / removing plugins (#55463)
* split out plugin manager

* remove whitespace

* fix tests

* split up tests

* updating naming conventions

* simplify manager

* tidy

* explorations

* fix build

* tidy

* fix tests

* add logger helper

* pass the tests

* tidying

* fix tests

* tidy and re-add test

* store depends on loader

* enrich tests

* fix test

* undo gomod changes
2022-09-23 14:27:01 +02:00
03af63d52e SearchV2: add tracing to background jobs (#55250)
* searchv2: add tracing to background jobs

* searchv2: lint

* searchv2: lint

* searchv2: fix context passing

* searchv2: add init org index span

* searchv2: add traceid to logs

* searchv2: add db count to logs
2022-09-20 16:49:44 +02:00
7ce7c9b64c Chore: Move SearchOrgs to org service (#55416)
* Chore: Move SearchOrgs to org service

* Fix lint

* Fix lint 2
2022-09-20 09:55:40 +02:00
cd7a464f7e Plugins: Ensure plugin uninstall success for "pre-installed" community plugins (#55322)
* track pre-installed external plugins in store

* fix tests
2022-09-19 12:14:30 +02:00
29327cbba2 Plugins: Use error plane for api/ds/query (#54750)
* plugin client returns error base

* fix api test

* add plugin client test

* add fallback err

* fix linting

* wip

* replace bad query

* template is an error

* failing test of templated error

* add one test passing

* fix failing test

* move test

* rename ErrBadQuery to ErrQueryValidationFailure

* tidy diff

* Change to one error per specific error kind

* last err + fix test

* fix imports

* more tests

* keep req vars together

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2022-09-14 18:19:57 +02:00
ced53a8dc2 plugins: Introduce generated, static core plugin registry (#54118)
* Refactor towards template/codegen framework

* Add templates for plugin gen

* Add Go codegen for plugins; overhaul framework, too

* Add new codegen output; assorted framework fixes

* Regenerate after merge

* Remove accidental commit file, update templates

* Export the pfs.Tree loader from plugin types

* Print details from cuetsy errors

* Generate loaders for all plugins and list in registry

* Use pfs_gen.go over lineage_gen.go

* Un-un-ignore main file

* Introduce simple List static registry for plugins

* Last tweaks to codegen

* remove unused tvars

* Ensure loop-local instances for both vars

* Generate pfs parsing in-place in registry

* Stop generating pfs_gen.go

* Move Tree into pfs, rename subdir

* Change package name to match dir

* Ignore gocyclo on HTTPServer.getNavTree
2022-09-14 10:15:09 -04:00
0571d98bba [WIP] Plugins: Introduce Plugins specific config (#54854) 2022-09-14 14:35:35 +01:00
b287047052 Chore: Upgrade Go to 1.19.1 (#54902)
* WIP

* Set public_suffix to a pre Ruby 2.6 version

* we don't need to install python

* Stretch->Buster

* Bump versions in lib.star

* Manually update linter

Sort of messy, but the .mod-file need to contain all dependencies that
use 1.16+ features, otherwise they're assumed to be compiled with
-lang=go1.16 and cannot access generics et al.

Bingo doesn't seem to understand that, but it's possible to manually
update things to get Bingo happy.

* undo reformatting

* Various lint improvements

* More from the linter

* goimports -w ./pkg/

* Disable gocritic

* Add/modify linter exceptions

* lint + flatten nested list

Go 1.19 doesn't support nested lists, and there wasn't an obvious workaround.
https://go.dev/doc/comment#lists
2022-09-12 12:03:49 +02:00
101349fe49 RBAC: Add permissions to install and configure plugins (#51829)
* RBAC: Allow app plugins restriction

Co-authored-by: Kalle Persson <kalle.persson@grafana.com>

* Moving declaration to HttpServer

Co-Authored-By: marefr <marcus.efraimsson@gmail.com>

* Picking changes from the other branch

Co-authored-by: Alexander Zobnin <alexanderzobnin@gmail.com>

* Rename plugins.settings to plugins

Co-authored-by: Kalle Persson <kalle.persson@grafana.com>

* Account for PluginAdminExternalManageEnabled

Co-authored-by: Will Browne <will.browne@grafana.com>

* Set metadata on instantiation

Co-authored-by: Jguer <joao.guerreiro@grafana.com>

Co-authored-by: Kalle Persson <kalle.persson@grafana.com>
Co-authored-by: marefr <marcus.efraimsson@gmail.com>
Co-authored-by: Alexander Zobnin <alexanderzobnin@gmail.com>
Co-authored-by: Will Browne <will.browne@grafana.com>
Co-authored-by: Jguer <joao.guerreiro@grafana.com>
2022-09-09 09:44:50 +02:00
15756ecf86 remove unused cfg (#54770) 2022-09-06 08:59:28 -04:00
ecdcafb258 manager exposes renderer + secrets manager (#54629) 2022-09-02 14:20:10 +02:00
43987e7f8c Plugins: Fix plugin registered startup log line (#54634)
* fix plugin registered log

* remove stale test
2022-09-02 14:01:52 +02:00