* parse via sse
I need to figure out how to handle the pipeline.execute with our own
client. I think this is important for MT reasons, just like using our
own cache (via legacy) is important.
parsing is done though!
* WIP nonsense
* horrible code but i think it works
* Add support for sql expressions config settings
* Cleanup:
- remove spew from nodes.go
- uncomment out plugin context and use in single tenant flow
- make code more readable and add comments
* Cleanup:
- create separate file for mt ds client builder
- ensure error handling is the same for both expressions and regular queries
- other cleanup
* not working but good thoughts
* WIP, vector not working for non sse
* super hacky but i think vectors work now
* delete delete delete
* Comments for future ref
* break out query handling and start test
* add prom debugger
* clean up: remove comments and commented out bits
* fix query_test
* add prom debugger
* create table-driven tests with testsdata files
* Fix test
* Add test
* go mod??
* idk
* Remove comment
* go enterprise issue maybe
* Fix codeowners
* Delete
* Remove test data
* Clean up
* logger
* Remove go changes hopefully
* idk go man
* sad
* idk i ran go mod tidy and this is what it wants
* Fix readme, with much help from adam
* some linting and testing errors
* lint
* fix lint
* fix lint register.go
* another lint
* address lint in test
* fix dead code and linters for query_test
* Go mod?
* Struggling with go mod
* Fix test
* Fix another test
* Revert headers change
* Its difficult to test this in OSS as it depends on functionality defined in enterprise, let's bring these tests back in some form in enterprise
* Fix codeowners
---------
Co-authored-by: Adam Simpson <adam@adamsimpson.net>
Adds settings for SQL expressions:
sql_expression_cell_output_limit
Set the maximum number of cells that can be returned from a SQL expression. Default is 100000.
sql_expression_timeout
The duration a SQL expression will run before being cancelled. The default is 10s.
When querying metric data (non-table data) with SQL Expressions, we need to convert the data to table format. This is alternative format which does not have the same issues with sparse data.
There is now a __metric_name__ column and one __value__ column. Also a __display_name__ column if there is DisplayNameFromDS metadata.
---------
Co-authored-by: Adam Simpson <adam@adamsimpson.net>
* expr: Add row limit to SQL expressions
Adds a configurable row limit to SQL expressions to prevent memory issues with large
result sets. The limit is configured via the `sql_expression_row_limit` setting in
the `[expressions]` section of grafana.ini, with a default of 100,000 rows.
The limit is enforced by checking the total number of rows across all input tables
before executing the SQL query. If the total exceeds the limit, the query fails
with an error message indicating the limit was exceeded.
* revert addition of newline
* Switch to table-driven tests
* Remove single-frame test-cases.
We only need to test for the multi frame case. Single frame is a subset
of the multi-frame case
* Add helper function
Simplify the way tests are set up and written
* Support convention, that limit: 0 is no limit
* Set the row-limit in one place only
* Update default limit to 20k rows
As per some discussion here:
https://raintank-corp.slack.com/archives/C071A5XCFST/p1741611647001369?thread_ts=1740047619.804869&cid=C071A5XCFST
* Test row-limit is applied from config
Make sure we protect this from regressions
This is perhaps a brittle test, somewhat coupled to the code here. But
it's good enough to prevent regressions at least.
* Add public documentation for the limit
* Limit total number of cells instead of rows
* Use named-return for totalRows
As @kylebrandt requested during review of #101700
* Leave DF cells as zero values during limits tests
When testing the cell limit we don't interact with the cell values at
all, so we leave them at their zero values both to speed up tests, and
to simplify and clarify that their values aren't used.
* Set SQLCmd limit at object creation - don't mutate
* Test that SQL node receives limit when built
And that it receives it from the Grafana config
* Improve TODO message for new Expression Parser
* Fix failing test by always creating config on the Service
* Under feature flag `sqlExpressions` and is experimental
* Excluded from arm32
* Will not work with the Query Service yet
* Does not have limits in place yet
* Does not working with alerting yet
* Currently requires "prepare time series" Transform for time series viz
---------
Co-authored-by: Sam Jewell <sam.jewell@grafana.com>
Improves log line to help with debugging in Server Side Expressions. In particular, the traceId, datasourceType, and datasourceUid will now be included.
* add GetCommandsFromPipeline
* refactor method GetCommandType to func GetExpressionCommandType
* add function to create fingerprint frames
* add function to determine whether raw query represents a hysteresis command and a function to patch it with loaded metrics
* extend threshold command with second evaluator called `unloadEvaluator`
* Introduce a new expression command Hysteresis and update Threshold unmarshaller to create the HysteresisCommand if the second eval
* add feature flag `recoveryThreshold`
* update unmarshal threshold command to not re-marshall because it breaks frame definition by shuffling the schema and data fields
Changes SSE to not always fail all queries when one fails. Now only the query itself, and nodes that depend on it will error.
---------
Co-authored-by: Gilles De Mey <gilles.de.mey@gmail.com>
Execute all queries to the same datasource in a single request.
Uses the query index and the graph node ID index, and then a stable dependency graph sort based on node input index number in attempt to keep the original query order intact.
* introduce a function checkIfSeriesNeedToBeFixed to scan all value fields in the response and provide a function that updates Series so they can be uniquely identifiable. Only Graphite and TestData are checked.
* update `convertDataFramesToResults` to run this function and provide it to WideToMany
* update WideToMany to run the fix function if it is not nil
* introduce a new node-type ML and implement a command outlier that uses ML plugin as a source of data.
* add feature flag mlExpressions that guards the feature
* add data source uid to QueryError
* add error and datasource uid to tracing
* split queryDataResponseToResults to two functions one to extract frame from the result, and another to convert it
* propagate logger with context to convertDataFramesToResults
Takes a specific code path for data that identifies itself as dataplane instead of "guessing" what the data is.
The data must identify itself by being in the dataplane by having both the following frame metadata properties:
- TypeVersion property that is greater than 0.0
- 'Type' property
The flag is disableSSEDataplane and disables this functionality and uses the old code for all queries regardless.
See https://github.com/grafana/grafana-plugin-sdk-go/blob/main/data/contract_docs/contract.md for dataplane details.
Adding support for backend plugin client middlewares. This allows headers in outgoing
backend plugin and HTTP requests to be modified using client middlewares.
The following client middlewares added:
Forward cookies: Will forward incoming HTTP request Cookies to outgoing plugins.Client
and HTTP requests if the datasource has enabled forwarding of cookies (keepCookies).
Forward OAuth token: Will set OAuth token headers on outgoing plugins.Client and HTTP
requests if the datasource has enabled Forward OAuth Identity (oauthPassThru).
Clear auth headers: Will clear any outgoing HTTP headers that was part of the incoming
HTTP request and used when authenticating to Grafana.
The current suggested way to register client middlewares is to have a separate package,
pluginsintegration, responsible for bootstrap/instantiate the backend plugin client with
middlewares and/or longer term bootstrap/instantiate plugin management.
Fixes#54135
Related to #47734
Related to #57870
Related to #41623
Related to #57065
* create contextual log context provider
* use contextual provider in scheduler
* init logger in the package
* use context for log context
* use context in state manager
* make TimeRange interface and add relative range
* make Execute methods support the current time
* update resample to support relative time range
* update DSNode to support relative time range
* update query service to create queries with absolute time
* make alerting evaluator create relative time ranges
When there is a single frame with no fields (e.g. splunk datasource) SSE errors when trying to figure out the data type. This frame needs to exist since this is where the executedQueryString metadata exists.
This adds a new return type to SSE to represent no data, so the original frame with its metadata can still be maintained.
