47 Commits

Author SHA1 Message Date
694b9dfe50 Chore: Replace xorm.io/xorm imports (#104458)
* replace xorm.io/xorm imports

* replace xorm from other go.mod files

* clean up workspace

* nolint does not make sense anymore as it is not a module

* try if nolint directive helps

* use nolint:all for xorm

* add more nolints

* try to skip xorm in linter config

* exclude xorm differently

* retrigger ci
2025-05-02 17:13:01 +02:00
e85acf047a Remove use of dialect.LikeStr (#104460)
* Fix TestIntegrationUpdatingProvisionionedDashboards* tests when running on Spanner by fixing case-insensitive search for dashboard title.

* Fix use of case-insensitive LIKE when running on Spanner.

* Fix use of LikeStr in anonstore.

* Fix use of LikeStr in ngalert/store and org/orgimpl.

* Fix use of LikeStr in queryhistory search.

* Fix use of LikeStr in serviceaccounts.

* Fix use of LikeStr in serviceaccounts.

* Fix use of LikeStr in services/team.

* Remove LikeStr method.
2025-05-02 11:23:57 +03:00
54b8dde1e1 services/user: Fix userimpl tests when running on Spanner. (#103715)
Fix userimpl tests when running on Spanner.
2025-04-11 10:28:21 +02:00
f7f28757f6 Skip noop migrations, instead of executing SELECT 0. (#103086)
* Skip noop migrations, instead of executing SELECT 0.
2025-03-31 15:19:32 +02:00
f3df64b7f4 Pass BOOL values as real types (int/bool) instead of strings to SQL parameters. (#101961)
* Pass BOOL values as real types (int/bool) instead of strings to SQL parameters.

Fixes following integration tests when running with Spanner:
* TestIntegrationDataAccess
    * GetDataSourcesByType/Get_prunable_data_sources
* TestIntegrationUserAuthToken:
    * expires_correctly
    * can_properly_rotate_tokens
    *  keeps_prev_token_valid_for_1_minute_after_it_is_confirmed

* Fix more places where "true" or "false" string was passed as query parameter instead of bool value.

* Removed unit test because it brought unwanted dependencies on xorm into multiple modules.
2025-03-12 15:40:11 +01:00
fd6a4908f1 Support Spanner's UNION syntax, which needs to be UNION DISTINCT or UNION ALL. (#101768)
* Support Spanner's UNION syntax, which needs to be UNION DISTINCT or UNION ALL.
2025-03-10 12:33:52 +01:00
9e690bd620 Spanner support enhancements (#101634)
* Adds ability to run integration tests against spanner (by using GRAFANA_TEST_DB=spanner env variable. SPANNER_DB variable then specifies database to use: spannertest, emulator or string like /projects/<project>/instances/<instance>/databases/<db>)
* Adds feature to migration dialects to create database from a snapshot, instead of running individual migrations.
* Adds first version of Spanner snapshot, prepared from "OSS" migrations.
* Uses generated bit-reversed-positive values instead of auto_increment. (As an experiment)
2025-03-06 16:11:20 +01:00
a21a232a8e Revert read replica POC (#93551)
* Revert "chore: add replDB to team service (#91799)"

This reverts commit c6ae2d7999aa6fc797db39e9d66c6fea70278f83.

* Revert "experiment: use read replica for Get and Find Dashboards (#91706)"

This reverts commit 54177ca619dbb5ded2dcb158405802d8dbdbc982.

* Revert "QuotaService: refactor to use ReplDB for Get queries (#91333)"

This reverts commit 299c142f6a6e8c5673cfdea9f87b56ac304f9834.

* Revert "refactor replCfg to look more like plugins/plugin config (#91142)"

This reverts commit ac0b4bb34d495914cbe8daad85b7c75c31e8070d.

* Revert "chore (replstore): fix registration with multiple sql drivers, again (#90990)"

This reverts commit daedb358dded00d349d9fac6106aaaa6bf18322e.

* Revert "Chore (sqlstore): add validation and testing for repl config (#90683)"

This reverts commit af19f039b62d9945377292a8e679ee258fd56b3d.

* Revert "ReplStore: Add support for round robin load balancing between multiple read replicas (#90530)"

This reverts commit 27b52b1507f5218a7b38046b4d96bc004d949d46.

* Revert "DashboardStore: Use ReplDB and get dashboard quotas from the ReadReplica (#90235)"

This reverts commit 8a6107cd35f6444c0674ee4230d3d6bcfbbd4a58.

* Revert "accesscontrol service read replica (#89963)"

This reverts commit 77a4869fcadf13827d76d5767d4de74812d6dd6d.

* Revert "Fix: add mapping for the new mysqlRepl driver (#89551)"

This reverts commit ab5a079bcc5b0f0a6929f0a3742eb2859d4a3498.

* Revert "fix: sql instrumentation dual registration error (#89508)"

This reverts commit d988f5c3b064fade6e96511e0024190c22d48e50.

* Revert "Experimental Feature Toggle: databaseReadReplica (#89232)"

This reverts commit 50244ed4a1435cbf3e3c87d4af34fd7937f7c259.
2024-09-25 15:21:39 -08:00
daedb358dd chore (replstore): fix registration with multiple sql drivers, again (#90990)
* replstore: fix registration with multiple sql drivers, again
* only compile regex once
2024-07-25 15:13:36 -04:00
ab5a079bcc Fix: add mapping for the new mysqlRepl driver (#89551) 2024-06-21 08:35:46 -04:00
5e88d29814 Folders: Introduce folder service function for fetching folders by org and UIDs that contain optionally the folder full path (#80716)
* Folders: Expose function for getting all org folders with specific UIDs

* Return all org folders if UIDs is empty

* Filter out not accessible folders by the user

* Modify query to optionally returning a string that contains the UIDs of all parent folders separated by slash.
2024-01-25 09:27:13 +02:00
ca9d147a44 Give dialects control over how insert and update queries are performed (#79946)
* Refactor insert, update
* Add separate insert, update methods
* Refactor Insert, Update signatures
2024-01-11 19:55:45 +01:00
48612063dd Grafana app platform: an aggregator cmd and package (#79948) 2024-01-08 22:33:42 +02:00
67b2972052 Chore: add/update sqlstore-related helper functions (#77408)
* add/update sqlstore-related helper functions

* add documentation & tests for InsertQuery and UpdateQuery, make generated SQL deterministic by sorting columns

* remove old log line
2023-11-03 10:30:52 -04:00
025b2f3011 Chore: use any rather than interface{} (#74066) 2023-08-30 18:46:47 +03:00
44972d0cd5 SQLStore: Fix Postgres dialect treating "false" migrator default as true (#69353)
* SQLStore: Fix Postgres dialect treating "false" migrator default as true

Previously, when creating a migration you could choose a default value for a new 
boolean column that looked correct but would be interpreted incorrectly by the 
Postgres dialect. For example, values such as "false" or "FALSE" would be treated 
as true by the Postgres dialect.

This refactors how migration dialects determine the Default column value for boolean
type columns. Each dialect now uses the same base code to parse the Default literal 
and panics if an unknown value is encountered.

So, now AddColumnMigration and AddTableMigration will ensure that across dialects:

- The exact same Default literals will be allowed.
- The literals are converted to equivalent defaults in their DDL.
- An error will be thrown if an invalid literal is provided.
2023-06-22 15:09:36 -04:00
a6279b2d62 Database: Make dialects independent of xorm Engine (#69955)
* make dialects independent of xorm Engine

* goimports
2023-06-14 16:13:36 -04:00
ea8c5ce929 SQLStore: Create utility for bulk inserts (#57632)
* Add batchsize to dialects

* Some intermediary progress

* Reflection-based implementation

* Validate batch size

* Implement a few simple tests

* Fix linter error

* Add debug message when falling back to default batch size
2022-11-01 13:24:32 -05:00
b287047052 Chore: Upgrade Go to 1.19.1 (#54902)
* WIP

* Set public_suffix to a pre Ruby 2.6 version

* we don't need to install python

* Stretch->Buster

* Bump versions in lib.star

* Manually update linter

Sort of messy, but the .mod-file need to contain all dependencies that
use 1.16+ features, otherwise they're assumed to be compiled with
-lang=go1.16 and cannot access generics et al.

Bingo doesn't seem to understand that, but it's possible to manually
update things to get Bingo happy.

* undo reformatting

* Various lint improvements

* More from the linter

* goimports -w ./pkg/

* Disable gocritic

* Add/modify linter exceptions

* lint + flatten nested list

Go 1.19 doesn't support nested lists, and there wasn't an obvious workaround.
https://go.dev/doc/comment#lists
2022-09-12 12:03:49 +02:00
a0f96ed4e1 SQLStore: Support Upserting multiple rows. (#52228)
This will be used to reduce write load when the alerting system writes a
large number of events.
2022-07-19 16:42:48 +08:00
5f1305d280 SQLStore: Add deprecation comments for breaking migrations (#49740)
* Migrator: Extend support to rename columns

* SQLStore: Add deprecation comments for breaking migrations
2022-06-03 17:42:08 +02:00
e43879e55d Encryption: Add support for multiple data keys per day (#47765)
* Add database migrations

* Use short uids as data key ids

* Add support for manual data key rotation

* Fix duplicated mutex unlocks

* Fix migration

* Manage current data keys per name

* Adjust key re-encryption and test

* Modify rename column migration for MySQL compatibility

* Refactor secrets manager and data keys cache

* Multiple o11y adjustments

* Fix stats query

* Apply suggestions from code review

Co-authored-by: Tania <yalyna.ts@gmail.com>

* Fix linter

* Docs: Rotate data encryption keys API endpoint

Co-authored-by: Tania <yalyna.ts@gmail.com>
2022-05-23 13:13:55 +02:00
f46038ed3a Search: sort results correctly when using postgres (#46466)
* Search: sort results correctly when using postgres

postgresql puts nulls first while both mysql and sqlite puts them last

* linting
2022-03-15 15:08:40 +01:00
d718ee1918 SQLStore: Prevent concurrent migrations (#44101)
* SQLStore: Prevent concurrent migrations

* Hide behind a feature toggle

* Configurable locking attempt timeout

* Update docs/sources/administration/configuration.md

Co-authored-by: Igor Suleymanov <radiohead@users.noreply.github.com>
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
2022-02-15 18:54:27 +02:00
8c31e25926 AlertingNG: Save alert instances (#30223)
* AlertingNG: Save alert instances

Co-authored-by: Kyle Brandt <kyle@grafana.com>

* Rename alert instance fields/columns

* Include definition title in listing alert instances

* Delete instances when deleting defintion

Co-authored-by: Kyle Brandt <kyle@grafana.com>
2021-01-18 20:57:17 +02:00
b5379c5335 Chore: Fix SQL related Go variable naming (#28887)
* Chore: Fix variable naming

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-11-11 06:21:08 +01:00
676d393ec9 Chore: Fix issues reported by staticcheck; enable stylecheck linter (#28866)
* Chore: Fix issues reported by staticcheck

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Apply suggestions from code review

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
2020-11-05 15:37:11 +01:00
74d1d3c6a8 Instrumentation: Add counters and histograms for database queries (#28236)
Signed-off-by: bergquist <carl.bergquist@gmail.com>

Co-authored-by: Sofia Papagiannaki <papagian@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-10-19 20:06:12 +02:00
4937f0daab SQLStore: Run tests as integration tests (#28265)
* sqlstore: Run tests as integration tests

* Truncate database instead of re-creating it on each test

* Fix test description

See https://github.com/grafana/grafana/pull/12129

* Fix lint issues

* Fix postgres dialect after review suggestion

* Rename and document functions after review suggestion

* Add periods

* Fix auto-increment value for mysql dialect

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
2020-10-16 10:46:14 +03:00
e5dd7efdee Provisioning: Allows specifying uid for datasource and use that in derived fields (#23585)
* Add uid to datasource

* Fix uid passing when provisioning

* Better error handling and Uid column type change

* Fix test and strict null error counts

* Add backend tests

* Add tests

* Fix strict null checks

* Update test

* Improve tests

* Update pkg/services/sqlstore/datasource.go

Co-Authored-By: Arve Knudsen <arve.knudsen@gmail.com>

* Variable rename

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-04-20 15:48:38 +02:00
ea35ae4d1f Chore: Update the xorm dependency (#22376)
* Upgrade xorm

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-04-01 15:57:21 +02:00
7b70e7db2d AuthProxy: Optimistic lock pattern for remote cache Set (#17485)
* Implementation of optimistic lock pattern

Try to insert the remote cache key and handle integrity error

* Remove transaction

Integrity error inside a transaction results in deadlock

* Remove check for existing remote cache key

Is no longer needed since integrity constrain violations are handled

* Add check for integrity constrain violation

Do not update the row if the insert statement fails
for other than an integrity constrain violation

* Handle failing inserts because of deadlocks

If the insert statement fails because of a deadlock
try to update the row

* Add utility function for returning SQL error code

Useful for debugging

* Add logging for failing expired cache key deletion

Do not shallow it completely

* Revert "Add utility function for returning SQL error code"

This reverts commit 8e0b82c79633e7d8bc350823cbbab2ac7a58c0a5.

* Better log for failing deletion of expired cache key

* Add some comments

* Remove check for existing cache key

Attempt to insert the key without checking if it's already there
and handle the error situations

* Do not propagate deadlocks created during update

Most probably somebody else is trying to insert/update
the key at the same time so it is safe enough to ignore it
2019-06-13 15:36:09 +02:00
12f3c8f9d6 fix only add column if not exists for mysql 2018-12-18 23:02:08 +01:00
09b3014683 fix only create/drop database indices if not exists/exists 2018-12-18 21:47:45 +01:00
d093244282 sqlstore: add support for checking if error is constraint validation error 2018-09-27 13:38:22 +02:00
1dfff74da9 move database-specific code into dialects (#11884) 2018-05-10 16:54:21 +02:00
5d95601720 Merge variable declaration with assignment (gosimple)
This fixes:
pkg/cmd/grafana-cli/commands/upgrade_all_command.go:56:3: should merge variable declaration with assignment on next line (S1021)
pkg/login/ldap.go:406:4: should merge variable declaration with assignment on next line (S1021)
pkg/services/sqlstore/migrator/dialect.go:87:2: should merge variable declaration with assignment on next line (S1021)
pkg/services/sqlstore/migrator/dialect.go:165:2: should merge variable declaration with assignment on next line (S1021)
pkg/tsdb/cloudwatch/metric_find_query_test.go:185:2: should merge variable declaration with assignment on next line (S1021)
2018-04-16 21:04:57 +02:00
3d1c624c12 WIP: Protect against brute force (frequent) login attempts (#10031)
* db: add login attempt migrations

* db: add possibility to create login attempts

* db: add possibility to retrieve login attempt count per username

* auth: validation and update of login attempts for invalid credentials

If login attempt count for user authenticating is 5 or more the last 5 minutes
we temporarily block the user access to login

* db: add possibility to delete expired login attempts

* cleanup: Delete login attempts older than 10 minutes

The cleanup job are running continuously and triggering each 10 minute

* fix typo: rename consequent to consequent

* auth: enable login attempt validation for ldap logins

* auth: disable login attempts validation by configuration

Setting is named DisableLoginAttemptsValidation and is false by default
Config disable_login_attempts_validation is placed under security section
#7616

* auth: don't run cleanup of login attempts if feature is disabled

#7616

* auth: rename settings.go to ldap_settings.go

* auth: refactor AuthenticateUser

Extract grafana login, ldap login and login attemp validation together
with their tests to separate files.
Enables testing of many more aspects when authenticating a user.
#7616

* auth: rename login attempt validation to brute force login protection

Setting DisableLoginAttemptsValidation => DisableBruteForceLoginProtection
Configuration disable_login_attempts_validation => disable_brute_force_login_protection
#7616
2018-01-26 10:41:41 +01:00
24d4e50343 utf8mb4 encoding (#7959)
* use utf8mb4 character set for connections to mysql

* use utf8mb4 character set for tables, shorten varchar fields used in unique indexes

* migration type to update table character set

* update table character sets

* set charset for temp_user.status

* gofmt
2017-03-28 14:34:53 +02:00
e5c64732f1 fix(sql): Add boolstr to all dialects
closes #6116
2016-09-23 09:36:15 +02:00
c7e3ed096f fix(postgres): fixes db migration issue with_credentials column for postgres, fixes #3505 2015-12-14 10:19:53 +01:00
835fd383ad fix(postgres): Dashboard search is now case insensitive when using Postgres, fixes #1896 2015-08-10 20:25:01 +02:00
ed68a4bb9a More work on SQL migrations 2015-02-24 17:59:21 +01:00
02a89c752b Progress on database schema migration for account -> org refactor 2015-02-24 11:46:34 +01:00
60541a455f Worked on search filter flag IsStarred, and updated frontend with new dashboard list panel 2015-02-04 11:35:59 +01:00
0a695ba17a Final work on migration, now there is no usage of xorm table sync 2015-01-20 14:44:37 +01:00
afb847acc8 a lot of work on database schema and migration setup, postgres now works, every integration test passes for all database types, only token table left to do 2015-01-20 14:15:48 +01:00