58 Commits

Author SHA1 Message Date
2d89b16d7e Tracing: Add Insecure to NewOTLPTracingConfig (#103174) 2025-04-01 17:34:42 +02:00
7aa71aa734 Tracing: chore make insecure option configurable (#99236)
* Tracing: chore: make insecure option configurable

In case of insecure set to false, default system cert pool will be used to create otlpgrpc client.

* test: add tracing/insecure config checks

This is to verify whether insecure configuration is properly parsed.

* chore: update configura Grafana docs with new setting

* chore: also update comment in the ini files

---------

Co-authored-by: Matheus Macabu <macabu.matheus@gmail.com>
2025-03-27 12:37:23 -04:00
ced7a84f8b chore(metrics): Add metrics & traces to DB migration (#97181)
Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
2024-12-05 17:22:19 -05:00
89e7e25b60 Backend: Inject server-timing header to match initial loads with client-side telemetry (#94978)
feat: inject server-timing header to match initial loads with client-side telemetry

Faro Web SDK can read the server-timing info and
correlate the initial request with the client side telemetry gather from navigation timings.
2024-11-19 10:01:58 +02:00
85759da360 Tracing: Add Start helper (#93052) 2024-09-06 13:02:17 -04:00
06e67bc573 API Server: Fix stack overflow panic when tracing is disabled (#90075) 2024-07-04 19:11:59 +03:00
99d8025829 Chore: Move identity and errutil to apimachinery module (#89116) 2024-06-13 07:11:35 +03:00
1abaa825c6 Tracing: Only enable traces to profiles for api servers for now (#89126) 2024-06-12 19:36:31 +03:00
40931b6da6 Tracing: Enable traces to profiles (#88896) 2024-06-07 16:58:24 +02:00
0b1aec6767 Tracing: Various improvements (#88308) 2024-05-27 14:21:40 +02:00
78f460c4de Tracing: Allow otel service name and attributes to be overridden from env (#85937)
Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com>
2024-04-11 15:18:46 +02:00
6c1de260a2 API Server: Standalone observability (#84789)
Adds support for logs (specify level), metrics (enable metrics and Prometheus /metrics endpoint 
and traces (jaeger or otlp) for standalone API server. This will allow any grafana core service 
part of standalone apiserver to use logging, metrics and traces as normal.
2024-03-21 17:06:32 +01:00
fcc2f01c7e Chore: Update otel deps (#78069)
Co-authored-by: Giuseppe Guerra <giuseppe@guerra.in>
Co-authored-by: Giuseppe Guerra <giuseppe.guerra@grafana.com>
2023-11-14 14:14:15 +01:00
7b4bcc86f1 Plugins: Pass OTEL sampling config to plugins (#76834)
* Pass OTEL sampling config to plugins

* fix capital letters

* Do not pass sampler env vars if sampling is not configured

* Add tests

* PR review feedback

* Simplify tracing env vars logic

* Update test to reflect pkg/infra/tracing behaviour
2023-11-10 14:56:08 +01:00
bf8af608a7 K8s: Add tracer provider to config (#77003) 2023-10-25 15:19:44 -04:00
e4c1a7a141 Tracing: Standardize on otel tracing (#75528) 2023-10-03 14:54:20 +02:00
0128d0403f Tracing: Use tracing.InitializeTracerForTest (#75479) 2023-09-27 09:51:57 +02:00
534e3ebf0c Plugins: Use a Grafana specific SDK tracer implementation for core plugins (#75388) 2023-09-26 14:46:31 +02:00
ce1169f8b7 Tracing: Support remote, rate-limited, and probabilistic sampling in tracing.opentelemetry config section (#73587)
* tracing: Support remote sampling server

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* Update docs/sources/setup-grafana/configure-grafana/_index.md

Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com>

* Update docs/sources/setup-grafana/configure-grafana/_index.md

Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com>

* Update docs/sources/setup-grafana/configure-grafana/_index.md

Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com>

* Update docs/sources/setup-grafana/configure-grafana/_index.md

Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com>

* Update docs/sources/setup-grafana/configure-grafana/_index.md

* Update docs/sources/setup-grafana/configure-grafana/_index.md

* Update docs/sources/setup-grafana/configure-grafana/_index.md

* Satisfying the doc-validator check

* satisfy prettier

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* back out unnecessary change

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

---------

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com>
2023-09-11 12:13:29 -04:00
025b2f3011 Chore: use any rather than interface{} (#74066) 2023-08-30 18:46:47 +03:00
2245a3d0d1 Tracer: Export ContextWithSpan (#73715)
Export ContextWithSpan
2023-08-25 13:48:58 -05:00
2bfef9e916 Tracing: Differentiate collector and agent better (#71851)
* differentiate collector and agent better

* Add back the max packet size

Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>

---------

Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2023-08-01 14:23:31 +02:00
c4be5cd747 Tracing: Reduce the max packet size for Jaeger exporter (#71659)
Reduce the max packet size for Jaeger exporter

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
2023-07-14 16:31:50 +01:00
c45ff94806 Tracing: supply Grafana build version (#69733)
* tracing: supply Grafana build version

Not etcd client version.

* go mod tidy
2023-06-22 11:55:30 +03:00
95b1f3c875 Fixing typos (#70487) 2023-06-22 09:43:38 +01:00
cf1945d0c3 Tests: use t.Setenv to set env vars (#69516)
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.Setenv

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2023-06-05 11:31:03 +02:00
6d8f9c5bf4 Chore: Remove opentracing and use opentelemetry instead (#67200)
* remove opentracing and use otel instead

* add various samplers for jaeger

* remove useless test that is covered in otel now

* we do not need a struct there

* remove old tests

* restore tests that parse various configurations

* check errors in tests

* Update pkg/infra/tracing/tracing_test.go

fix typo

Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>

* add test for both legacy and new config formats

* use named constants

---------

Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2023-04-27 15:04:43 +02:00
bff9f4c890 Plugins: Move config factory to pluginsintegration package (#65716)
* move config to pluginsintegration package

* change to all plugin toggle

* fixes

* fixes

* fix lerna

* fix test
2023-04-05 14:40:08 +02:00
fb520edd72 Alerting: Use a completely isolated context for state history writes (#64989)
* Add fresh context with timeout and same log properties, re-derive logger

* Unify timeout constants

* Move ctx after shortcut that got added through rebasing

* Unify timeouts

* Port opentracing's SpanFromContext and ContextFromSpan to the grafana tracing package

* Support both opentracing and otel variants

* Better document why we're creating a new ctx

* Add new func to FakeSpan which was added after rebase

* Support grafana-specific traceID key in both tracer implementations
2023-04-04 16:41:46 -05:00
09078b14e1 Plugins: Support for distributed tracing in backend plugins SDK (#63714)
* Tracing: Pass OTLP address and propagation format to plugins

* Fix unit tests

* Fix indentation

* Fix plugin manager integration tests

* Goimports

* Pass plugin version to plugins

* Do not add GF_PLUGIN_VERSION if plugin version is not set, add tests

* Allow disabling plugins distributed tracing on a per-plugin basis

* Moved disabled plugins to tracing.opentelemetry config section

* Pre-allocate DisabledPlugins map to the correct size

* Moved disable tracing setting flags in plugin settings

* Renamed plugin env vars for tracing endpoint and propagation

* Fix plugin initializer tests

* Refactoring: Moved OpentelemetryCfg from pkg/infra to pkg/plugins

* Changed GetSection to Section in parseSettingsOpentelemetry

* Add tests for NewOpentelemetryCfg

* Fix test case names in TestNewOpentelemetryCfg

* OpenTelemetry: Remove redundant error checks
2023-03-30 23:31:14 +02:00
a89202eab2 Plugins: Improve instrumentation by adding metrics and tracing (#61035)
* WIP: Plugins tracing

* Trace ID middleware

* Add prometheus metrics and tracing to plugins updater

* Add TODOs

* Add instrumented http client

* Add tracing to grafana update checker

* Goimports

* Moved plugins tracing to middleware

* goimports, fix tests

* Removed X-Trace-Id header

* Fix comment in NewTracingHeaderMiddleware

* Add metrics to instrumented http client

* Add instrumented http client options

* Removed unused function

* Switch to contextual logger

* Refactoring, fix tests

* Moved InstrumentedHTTPClient and PrometheusMetrics to their own package

* Tracing middleware: handle errors

* Report span status codes when recording errors

* Add tests for tracing middleware

* Moved fakeSpan and fakeTracer to pkg/infra/tracing

* Add TestHTTPClientTracing

* Lint

* Changes after PR review

* Tests: Made "ended" in FakeSpan private, allow calling End only once

* Testing: panic in FakeSpan if span already ended

* Refactoring: Simplify Grafana updater checks

* Refactoring: Simplify plugins updater error checks and logs

* Fix wrong call to checkForUpdates -> instrumentedCheckForUpdates

* Tests: Fix wrong call to checkForUpdates -> instrumentedCheckForUpdates

* Log update checks duration, use Info log level for check succeeded logs

* Add plugin context span attributes in tracing_middleware

* Refactor prometheus metrics as httpclient middleware

* Fix call to ProvidePluginsService in plugins_test.go

* Propagate context to update checker outgoing http requests

* Plugin client tracing middleware: Removed operation name in status

* Fix tests

* Goimports tracing_middleware.go

* Goimports

* Fix imports

* Changed span name to plugins client middleware

* Add span name assertion in TestTracingMiddleware

* Removed Prometheus metrics middleware from grafana and plugins updatechecker

* Add span attributes for ds name, type, uid, panel and dashboard ids

* Fix http header reading in tracing middlewares

* Use contexthandler.FromContext, add X-Query-Group-Id

* Add test for RunStream

* Fix imports

* Changes from PR review

* TestTracingMiddleware: Changed assert to require for didPanic assertion

* Lint

* Fix imports
2023-03-28 11:01:06 +02:00
c0b5d2dfaa Tracing: Support multiple OTel propagators (#61199)
* tracing: Support multiple OTel propagators

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* tracing: add TraceIDString method to fix up tests

This method will be useful elsewhere if we want to log the trace ID.

* improve propagation docs

Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com>

* doc style fix

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* Use tracing.TraceIDFromContext instead of adding TraceIDString method

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

---------

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
Co-authored-by: Bryan Boreham <bjboreham@gmail.com>
Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com>
2023-03-27 14:56:24 -04:00
f531074d89 Chore: Fix goimports grouping in pkg/infra (#62421)
* fix goimports

* fix goimports order
2023-01-30 08:32:25 +00:00
6aef353b59 Tracing: Fix bug where errors are not reported to OTel (#55925) 2022-09-28 15:54:47 +02:00
b0b1e8d568 Tracing: Document Tracer and Span interfaces (#55926) 2022-09-28 15:53:48 +02:00
862a6a2fa6 Logging: Introduce API for contextual logging (#55198)
Introduces a FromContext method on the log.Logger interface that 
allows contextual key/value pairs to be attached, e.g. per request, 
so that any logger using this API will automatically get the per request 
context attached. The proposal makes the traceID available for 
contextual logger , if available, and would allow logs originating from 
a certain HTTP request to be correlated with traceID.
In addition, when tracing not enabled, skip adding
traceID=00000000000000000000000000000000
to logs.
2022-09-20 18:32:06 +02:00
801b61c963 Tracing: Add new [tracing.opentelemetry] custom_attributes config setting (#54110)
* tracing: Add new [tracing.opentelemetry] custom_attributes config setting

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* Fix typos in config

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* Return error when custom_attributes contains malformed entries

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
2022-09-16 09:54:25 -04:00
4d4ecd7fec fix the event attributes (#54117) 2022-08-24 09:59:06 -05:00
806fb8ab7a Tracing: Fix OpenTelemetry Jaeger context propagation (#53269)
* fix otel jaeger context propagation

* add back launch.json

* add back launch.json
2022-08-11 16:11:34 +02:00
56f3f3fe69 Chore: Remove the old log format (#51526)
* remove the old log format

* fix CI

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
2022-07-08 09:56:30 -05:00
ae9491c3a7 Chore: Make test tracer noop and return no errors (#50797) 2022-06-15 12:40:41 +02:00
24c6a73095 Tracing: Deprecate opentracing (#50058)
* Deprecate opentracing

* Fix comment

* Adjust comment

* Fix docs for opentelemetry

* Add deprecated to sample.ini
2022-06-02 14:13:00 +02:00
0d7a3209e7 etcd: Fix vuln CVE-2018-1098 (#49976)
* Update etcd

* Update go.sum

* Replace etcd with etcd/v3
2022-06-01 09:31:34 +02:00
31ff23f542 Fix: add default nop trace exporter to opentelemetry (#48869) 2022-05-10 11:29:36 +02:00
abb1618291 Add OTLP exporter for OpenTelemetry (#47987)
* Add OTLP exporter for OpenTelemtry

* Fix lint

* Refactore parse settings

* Add configuration for propagation + fix tests

* Fix tests and lint

* Fix alerting tests

* Add coments to config

* Add propagation to custom.ini
2022-05-05 10:37:26 +02:00
41012af997 Tracing: Use common traceID context value for opentracing and opentelemetry (#46411)
* use common traceID context value for opentracing and opentelemetry

* support sampled trace IDs as well

* inject traceID into NormalResponse on errors

* Finally the test passed

* fix the test

* fix linter

* change the function parameter

Co-authored-by: Ying WANG <ying.wang@grafana.com>
2022-04-14 17:54:49 +02:00
30aa24a183 Chore: Implement OpenTelemtry in Grafana (#42674)
* Separate Tracer interface to TracerService and Tracer

* Fix lint

* Fix:Make it possible to start spans for both opentracing and opentelemetry in ds proxy

* Add span methods, use span interface for rest of tracing

* Fix logs in tracing

* Fix tests that are related to tracing

* Fix resourcepermissions test

* Fix some tests

* Fix more tests

* Add TracingService to wire cli runner

* Remove GlobalTracer from bus

* Renaming test function

* Remove GlobalTracer from TSDB

* Replace GlobalTracer in api

* Adjust tests to the InitializeForTests func

* Remove GlobalTracer from services

* Remove GlobalTracer

* Remove bus.NewTest

* Remove Tracer interface

* Add InitializeForBus

* Simplify tests

* Clean up tests

* Rename TracerService to Tracer

* Update pkg/middleware/request_tracing.go

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* Initialize tracer before passing it to SQLStore initialization in commands

* Remove tests for opentracing

* Set span attributes correctly, remove unnecessary trace initiliazation form test

* Add tracer instance to newSQLStore

* Fix changes due to rebase

* Add modified tracing middleware test

* Fix opentracing implementation tags

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2022-01-20 11:10:12 +01:00
d993b12415 Add interface Tracer, add Opentelemetry (#41963)
* Add interface Tracer, add Opentelemetry

* Fix lint

* Fix failing tests and return error if config not parsed fo opentelemetry

* Update defaults.ini

Add comment with jaeger url

* go mod tidy

* Remove comments that are not needed

* Move OpentracingSpan to tracing.go

* Add opentelemetry to sample.ini
2021-12-01 17:05:08 +01:00
78596a6756 Migrate to Wire for dependency injection (#32289)
Fixes #30144

Co-authored-by: dsotirakis <sotirakis.dim@gmail.com>
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
Co-authored-by: Ida Furjesova <ida.furjesova@grafana.com>
Co-authored-by: Jack Westbrook <jack.westbrook@gmail.com>
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
Co-authored-by: Leon Sorokin <leeoniya@gmail.com>
Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com>
Co-authored-by: spinillos <selenepinillos@gmail.com>
Co-authored-by: Karl Persson <kalle.persson@grafana.com>
Co-authored-by: Leonard Gram <leo@xlson.com>
2021-08-25 15:11:22 +02:00
e244267b7d Enable tracing when jaeger host and port are set (#33682)
* check for jaeger env variables if tracing address configuration is empty

* add tests to ensure jaeger host/port variables override the settings

* allow default host and custom port, too

* disallow missing tracing.jaeger section in the config

* check for all errors in tests

* make parseSettings() return an error
2021-05-05 11:34:13 +02:00