What is this feature?
A follow-up for #101184, adds AlertRule.MissingSeriesEvalsToResolve to the APIs.
missing_series_evals_to_resolve must be specified too and it must be > 0.
POST /api/ruler/grafana/api/v1/rules/{folderUID} works in the following way:
If missing_series_evals_to_resolve is not sent or null, the rule keeps its existing value
If missing_series_evals_to_resolve > 0: updates to that value
If missing_series_evals_to_resolve = 0: resets to default (nil).
AlertRule.MissingSeriesEvalsToResolve can't be 0, so I used it to reset
In the Provisioning API, the value is just set if present and > 0. Otherwise it's reset:
PUT to /api/v1/provisioning/alert-rules/{UID}:
If missing_series_evals_to_resolve is nil, it's reset to the default value
If missing_series_evals_to_resolve > 0, it's updated
What is this feature?
This PR introduces a new alert rule configuration option, keep_firing_for (Prometheus documentation).
keep_firing_for prevents alerts from resolving immediately after the alert condition returns to normal. Instead, they transition into a "Recovering" state and are not considered resolved by the Alertmanager. Once the recovery period ends (or after the next evaluation if it is bigger than keep_firing_for), the alert transitions to "Normal" if it doesn't start alerting again:
Before
+----------+ +----------+
| Alerting |---->| Normal |
+----------+ +----------+
-----
After
+----------+ +------------+ +----------+
| Alerting |----->| Recovering |---->| Normal |
+----------+ +------------+ +----------+
Why do we need this feature?
This feature prevents flapping alerts by adding a recovery period. This helps avoid false resolutions caused by brief alert
* add query parameter to existing APIs to control the permanent deletion of rules
* add GUID to gettable rule
* add new endpoint /ruler/grafana/api/v1/trash/rule/guid/{RuleGUID} to delete rules from trash permanently
---------
Signed-off-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
When creating Grafana-managed alerts from Prometheus rule definitions with mimirtool or cortextool, the rules are marked as "provisioned" and are not editable in the Grafana UI. This PR allows changing this by providing an extra header: --extra-header="X-Disable-Provenance=true".
When provenance is disabled, we do not keep the original rule definition in YAML, so it is impossible to read it back using the Prometheus conversion API (mimirtool/cortextool). This is intentional because if we did keep it and the rule was later changed in the UI, its Prometheus YAML definition would no longer reflect the latest version of the alert rule, as it would be unchanged.
What is this feature?
Adds target datasource UID to the recording rules so that they write to the same datasource used for alerting rule queries after the import.
Why do we need this feature?
Target datasourse support was added in #101678, and under a feature flag grafanaManagedRecordingRulesDatasources (#101778).
This PR makes the importing process:
Check if the import contains recording rules
Verify both recording rules and the grafanaManagedRecordingRulesDatasources feature flag are enabled
If either check fails, return an error
If both checks pass, create recording rules with the provided datasource UID set as both the query and target datasource
What is this feature?
Allows the creation of alert rules with mimirtool in a specified folder.
Why do we need this feature?
Currently, the APIs for mimirtool create namespaces and rule groups in the root folder without the ability to set a custom folder. For example, it could be a special "Imported" folder, etc.
This PR makes it possible with a special header: mimirtool ... --extra-headers="X-Grafana-Alerting-Folder-UID=123". If it's not present, the root folder is used, otherwise, the specified one is used.
mimirtool does not support nested folder structures, while Grafana allows folder nesting. To keep compatibility, we return only direct child folders of the working folder (as namespaces) with rule groups and rules that are directly in these child folders as if there are no nested folders.
For example, given this folder structure in Grafana:
```
grafana/
├── production/
│ ├── service1/
│ │ └── alerts/
│ └── service2/
└── testing/
└── service3/
```
If the working folder is "grafana":
Only namespaces "production" and "testing" are returned
Only rule groups directly within these folders are included
If the working folder is "production":
- Only namespaces "service1" and "service2" are returned
Only rule groups directly within these folders are included
Initially, Metadata had only the EditorSettings, and HasMetadata was used to understand if the incoming update request had metadata in the body because it could be omitted if it was empty. For example, when the rule is updated via the provisioning API or has only false values. If it was in the request, we used that; if not, we used the metadata from the existing rule from the database. If the rule was updated via the AlertRuleService, we didn't change Metadata at all if the rule already existed.
But now, Metadata also has the Prometheus rule definition, and we always need to update it with the new version of the AlertRuleService when the rule exists in the DB and has the same UID. HasMetadata is renamed to HasEditorSettings to keep the old behaviour only for EditorSettings.
Now, the provisioning API and the conversion API will overwrite everything except EditorSettings with the new data.
What is this feature?
Adds an API endpoint to create alert rules with mimirtool:
- POST /convert/prometheus/config/v1/rules/{NamespaceTitle} - Accepts a single rule group in a Prometheus YAML format and creates or updates a Grafana rule group from it.
The endpoint uses the conversion package from #100224.
Key parts
The API works similarly to the provisioning API. If the rule does not exist, it will be created, otherwise updated. Any rules not present in the new group will be deleted, ensuring the group is fully synchronized with the provided configuration.
Since the API works with namespace titles (folders), the handler automatically creates a folder in the root based on the provided title if it does not exist. It also requires a special header, X-Grafana-Alerting-Datasource-UID. This header specifies which datasource to use for the new rules.
If the rule group's evaluation interval is not specified, it uses the DefaultRuleEvaluationInterval from settings.
* Alerting: Fix Alertmanager configuration updates
Alertmanager configuration updates would behave inconsistently when performing no-op updates with `mysql` as the store.
In particular this bug manifested as a failure to reload the provisioned alertmanager configuration components with no changes to the configuration itself. This would result in a 500 error with mysql store only.
The core issue is that we were relying on the number of rows affected by the update query to determine if the configuration was found in the db or not.
While this behavior works for certain sql dialects, mysql does not return the number of rows matched by the update query but rather the number of rows actually updated.
Also discovered and fixed the mismatched `xorm` tag for the `CreatedAt` field to match the actual column name in the db.
References: https://dev.mysql.com/doc/refman/8.4/en/update.html
When exporting contact-points, mute-timings, and notification policies in the provisioning API, we need to escape the `$` character which is used in interpolation by file provisioning.
Follow up to #97985
* Support delete endpoint for folders
* Include authorizer
* Add test for delete verb
* Add delete command to delete options
* Pass query string to context to admission
* Dont support nested folder deletion for now
* Skip test if feature flag is present
* Add test case
* Remove comment
* Only rely on the storage type config to run alerting tests
* Dont change legacy subpath
* Remove unised function
* Add test case when an editor can delete alert rules
* Lint
* Revert "chore: add replDB to team service (#91799)"
This reverts commit c6ae2d7999aa6fc797db39e9d66c6fea70278f83.
* Revert "experiment: use read replica for Get and Find Dashboards (#91706)"
This reverts commit 54177ca619dbb5ded2dcb158405802d8dbdbc982.
* Revert "QuotaService: refactor to use ReplDB for Get queries (#91333)"
This reverts commit 299c142f6a6e8c5673cfdea9f87b56ac304f9834.
* Revert "refactor replCfg to look more like plugins/plugin config (#91142)"
This reverts commit ac0b4bb34d495914cbe8daad85b7c75c31e8070d.
* Revert "chore (replstore): fix registration with multiple sql drivers, again (#90990)"
This reverts commit daedb358dded00d349d9fac6106aaaa6bf18322e.
* Revert "Chore (sqlstore): add validation and testing for repl config (#90683)"
This reverts commit af19f039b62d9945377292a8e679ee258fd56b3d.
* Revert "ReplStore: Add support for round robin load balancing between multiple read replicas (#90530)"
This reverts commit 27b52b1507f5218a7b38046b4d96bc004d949d46.
* Revert "DashboardStore: Use ReplDB and get dashboard quotas from the ReadReplica (#90235)"
This reverts commit 8a6107cd35f6444c0674ee4230d3d6bcfbbd4a58.
* Revert "accesscontrol service read replica (#89963)"
This reverts commit 77a4869fcadf13827d76d5767d4de74812d6dd6d.
* Revert "Fix: add mapping for the new mysqlRepl driver (#89551)"
This reverts commit ab5a079bcc5b0f0a6929f0a3742eb2859d4a3498.
* Revert "fix: sql instrumentation dual registration error (#89508)"
This reverts commit d988f5c3b064fade6e96511e0024190c22d48e50.
* Revert "Experimental Feature Toggle: databaseReadReplica (#89232)"
This reverts commit 50244ed4a1435cbf3e3c87d4af34fd7937f7c259.
* add RenameTimeIntervalInNotificationSettings to storage
* update dependencies when the time interval is renamed
---------
Co-authored-by: William Wernert <william.wernert@grafana.com>
* Feature (quota service): Use ReplDB for quota service Gets
This adds the replDB to the quota service, as well as some more test helper functions to simplify updating tests. My intent is that the helper functions can be removed when this is fully rolled out (or not) and we're consistently using the ReplDB interface (or not!)
* test updates
* Check if a time interval is used in alert rules before deleting it
* Add time interval to parameters of ListAlertRulesQuery and ListNotificationSettings of DbStore
== Refacorings ==
* refactor isMuteTimeInUse to accept a single route
* update getMuteTiming to not return err
* update delete to get the mute timing from config first
The contact point deletion API was returning 500 when it should have been
returning a 4xx error, when the contact point is in use:
- When in use by a notificiation policy, we were missing
the `.Errorf("")` to convert `errutil.Base` into `errutil.Error`.
- When in use by an alert rule, an regular error was returned.
* add test for the bug
* remove unused struct
* update db store to post process filters by group using go-lang's case-sensitive string comparison
--------
Co-authored-by: Alexander Weaver <weaver.alex.d@gmail.com>
* add version to time-interval models
* set time interval fingerprint as version
* update to check provided version
* delete to check if version is provided in query parameter 'version'
* update integration tests
* update specs
