* parse via sse
I need to figure out how to handle the pipeline.execute with our own
client. I think this is important for MT reasons, just like using our
own cache (via legacy) is important.
parsing is done though!
* WIP nonsense
* horrible code but i think it works
* Add support for sql expressions config settings
* Cleanup:
- remove spew from nodes.go
- uncomment out plugin context and use in single tenant flow
- make code more readable and add comments
* Cleanup:
- create separate file for mt ds client builder
- ensure error handling is the same for both expressions and regular queries
- other cleanup
* not working but good thoughts
* WIP, vector not working for non sse
* super hacky but i think vectors work now
* delete delete delete
* Comments for future ref
* break out query handling and start test
* add prom debugger
* clean up: remove comments and commented out bits
* fix query_test
* add prom debugger
* create table-driven tests with testsdata files
* Fix test
* Add test
* go mod??
* idk
* Remove comment
* go enterprise issue maybe
* Fix codeowners
* Delete
* Remove test data
* Clean up
* logger
* Remove go changes hopefully
* idk go man
* sad
* idk i ran go mod tidy and this is what it wants
* Fix readme, with much help from adam
* some linting and testing errors
* lint
* fix lint
* fix lint register.go
* another lint
* address lint in test
* fix dead code and linters for query_test
* Go mod?
* Struggling with go mod
* Fix test
* Fix another test
* Revert headers change
* Its difficult to test this in OSS as it depends on functionality defined in enterprise, let's bring these tests back in some form in enterprise
* Fix codeowners
---------
Co-authored-by: Adam Simpson <adam@adamsimpson.net>
* capture errors on metrics
* rename seconds to sql_command_duration_milliseconds to match unit that has been captured, and update buckets
* rename sql_command_errors_total to sql_command_count
Adds settings for SQL expressions:
sql_expression_cell_output_limit
Set the maximum number of cells that can be returned from a SQL expression. Default is 100000.
sql_expression_timeout
The duration a SQL expression will run before being cancelled. The default is 10s.
* SQL Expressions: Bump GMS for STDDEV(), VARIANCE()
Bump Go MySQL Server to the latest version, to get support for STDDEV()
and VARIANCE() functions, and their aliases.
See https://github.com/dolthub/go-mysql-server/pull/2928
* Update app-code to comply with latest GMS version
* Run `go mod tidy`
* Update test which had depended on STDDEV being missing
* Run `make update-workspace`
* wip
* Add prom flavor support for data source variables and export/import dashboards (#103321)
* add dashboard and data source var selection
* use match plugin id instead
* use updated matchpluginid
* formatting
* cleanup
* regex anchor
* update error msg
* Alerting: Clean up prometheus-flavored types and functions (#103703)
* clean up types and utility functions for dealing with
prometheus-flavored data sources
* Refactor alerting datasource types to use constants as source of truth
* Alerting: Clean up prometheus-flavored types and functions on the bac… (#103716)
Alerting: Clean up prometheus-flavored types and functions on the backend
* add matchPluginId tests
* Update matchPluginId func to bidirectional (#103746)
* update matchpluginid func to bidirectional
* lint
* formatting
* use actual isSupportedExternalRulesSourceType in test
* add tests in datasource_srv
* betterer
* remove type assertion
* remove unnecessary case
* use satisifies to not have to convert tuple to an array of string
* add prometheus_flavor test
---------
Co-authored-by: Andrew Hackmann <5140848+bossinc@users.noreply.github.com>
Co-authored-by: Gilles De Mey <gilles.de.mey@gmail.com>
Co-authored-by: Alexander Akhmetov <me@alx.cx>
* SQL Expressions: Allow many more functions and nodes
* Also allow the `REGEXP_SUBSTR` function
* Add window functions
* add more JSON support, remove now and current time (for now)
---------
Co-authored-by: Kyle Brandt <kyle@grafana.com>
Initial support for alerting with SQL expressions
- When `format` is set to `alerting`, SQL expressions output in a format suitable for alerting evaluation.
- Outstanding TODOs:
- Deduplicate output rows
- Add more tests
- Fix broken alerting UI rendering (likely due to shape change to undocumented full-long format)
- Basic usage:
- SQL must return one numeric column and one or more string columns.
- Each row may become an alert.
- The alert fires if the numeric value is non-zero.
- String columns are treated as labels.
---------
Co-authored-by: Konrad Lalik <konradlalik@gmail.com>
Co-authored-by: Tom Ratcliffe <tom.ratcliffe@grafana.com>
Co-authored-by: Sam Jewell <sam.jewell@grafana.com>
* SQL Expressions: Add Null-literal node
* Retain some order in the code - put NullVal with BoolVal
* Add support for `IN` keyword
* Add GROUP_CONCAT keyword
* Add COLLATE keyword
From Claude:
The test case demonstrates a simple use of COLLATE with a string
literal, but in real MySQL queries, COLLATE is often used in more
complex scenarios like:
1. String comparisons:
`SELECT * FROM table WHERE name COLLATE utf8mb4_bin = 'John'`
2. Sorting:
`SELECT * FROM table ORDER BY name COLLATE utf8mb4_unicode_ci`
3. JOINs:
`SELECT * FROM table1 JOIN table2 ON table1.name COLLATE utf8mb4_bin = table2.name`
The COLLATE clause is particularly useful when you need case-sensitive
comparisons (utf8mb4_bin) or specific language-aware sorting rules.
When querying metric data (non-table data) with SQL Expressions, we need to convert the data to table format. This is alternative format which does not have the same issues with sparse data.
There is now a __metric_name__ column and one __value__ column. Also a __display_name__ column if there is DisplayNameFromDS metadata.
---------
Co-authored-by: Adam Simpson <adam@adamsimpson.net>
* SQL Expressions: Add CASE/WHEN SQL nodes to allowlist
* Fixed and test for functions added in #102011
* Add remaining functions to the test-case
These are mostly aliases, so the LLM chose to omit them originally.
But adding now for completeness
* Fix ordering of allowed nodes
* expr: Add row limit to SQL expressions
Adds a configurable row limit to SQL expressions to prevent memory issues with large
result sets. The limit is configured via the `sql_expression_row_limit` setting in
the `[expressions]` section of grafana.ini, with a default of 100,000 rows.
The limit is enforced by checking the total number of rows across all input tables
before executing the SQL query. If the total exceeds the limit, the query fails
with an error message indicating the limit was exceeded.
* revert addition of newline
* Switch to table-driven tests
* Remove single-frame test-cases.
We only need to test for the multi frame case. Single frame is a subset
of the multi-frame case
* Add helper function
Simplify the way tests are set up and written
* Support convention, that limit: 0 is no limit
* Set the row-limit in one place only
* Update default limit to 20k rows
As per some discussion here:
https://raintank-corp.slack.com/archives/C071A5XCFST/p1741611647001369?thread_ts=1740047619.804869&cid=C071A5XCFST
* Test row-limit is applied from config
Make sure we protect this from regressions
This is perhaps a brittle test, somewhat coupled to the code here. But
it's good enough to prevent regressions at least.
* Add public documentation for the limit
* Limit total number of cells instead of rows
* Use named-return for totalRows
As @kylebrandt requested during review of #101700
* Leave DF cells as zero values during limits tests
When testing the cell limit we don't interact with the cell values at
all, so we leave them at their zero values both to speed up tests, and
to simplify and clarify that their values aren't used.
* Set SQLCmd limit at object creation - don't mutate
* Test that SQL node receives limit when built
And that it receives it from the Grafana config
* Improve TODO message for new Expression Parser
* Fix failing test by always creating config on the Service
Simplify, by retaining quotes when escaping query
We also don't need to convert to string when passing to fmt.Sprintf
a slice of Bytes is accepted by Sprintf instead.
* serviceaccounts/secretscan: test Service more thoroughly
* middleware/cookies: add tests for CookieOptions
* anonymous/anonimpl: cover a couple more methods
* components/imguploader: Implement WebDAV integration tests
* components/apikeygen: also check IsValid method
* bus: cover invalid callback signature cases
* cloudmigration/objectstorage: add basic unit tests
* login/social/connectors: add test case for GitHub OAuth fetch emails+orgs
* expr/classic: cover more evaluator types in tests
- use types.Convert to get a more normalized type returned from GMS. After syncing with GMS maintainers, the underlying type of the row could be different things (e.g. int when unit32, so this calls the Convert method from the GMS types library to normalize the interface.
- Clean up fieldValFromRow more
- Use IsText to capture different string types
- Add more types to test, also update test to use same cmp.Diff method as others
---------
Co-authored-by: Sam Jewell <2903904+samjewell@users.noreply.github.com>
* Under feature flag `sqlExpressions` and is experimental
* Excluded from arm32
* Will not work with the Query Service yet
* Does not have limits in place yet
* Does not working with alerting yet
* Currently requires "prepare time series" Transform for time series viz
---------
Co-authored-by: Sam Jewell <sam.jewell@grafana.com>
