**What this PR does / why we need it**:
This updates the PrepareShutdown method so it supports GET and DELETE
methods as well. This makes it similar to Mimir:
https://github.com/grafana/mimir/pull/4718.
The status is now stored in a local file. A new config setting had to be
added for this file as there is no obvious place to store it.
**Checklist**
- [X] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [X] Documentation added
- [X] Tests updated
- [x] `CHANGELOG.md` updated
- [x] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/upgrading/_index.md`
---------
Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>
**What this PR does / why we need it**:
Describes how exactly the `start` and `end` timestamps operate on the
time range boundaries.
**Checklist**
- [x] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [x] Documentation added
---------
Co-authored-by: J Stickler <julie.stickler@grafana.com>
**What this PR does / why we need it**:
Add a new `/ingester/prepare_shutdown` endpoint. This new endpoint
configures the ingester to release resources on the next SIGTERM.
---------
Co-authored-by: J Stickler <julie.stickler@grafana.com>
Source code:
https://github.com/grafana/technical-documentation/tree/main/tools/doc-validator
Signed-off-by: Jack Baldry <jack.baldry@grafana.com>
**What this PR does / why we need it**:
Adds a CI step for pull requests that validates links in technical
documentation and performs linting of technical documentation source
files.
**Special notes for your reviewer**:
- Lots of issues will need to be fixed before we can merge this.
- The tool is still in early development so each error should be
scrutinized for validity.
**Checklist**
- [ ] Documentation added
- [na] Tests updated
- [na] Is this an important fix or new feature? Add an entry in the
`CHANGELOG.md`.
- [na] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/upgrading/_index.md`
Signed-off-by: Jack Baldry <jack.baldry@grafana.com>
Signed-off-by: Edward Welch <edward.welch@grafana.com>
**What this PR does / why we need it**:
logcli has a command line parameter called `since` which makes it easy
to query logs relative to the current time, however this is some
syntactic sugar used in logcli to calculate `start` and `end` query
parameters for the query sent to Loki.
This PR adds native support for a `since` query parameter to make it a
little easier to consume the Loki query API without having to calculate
precise start/end times with every request.
This is useful if you are polling the Loki API from a third party app
which doesn't let you easily build dynamic URL's and all you want to do
is poll for say the last 3 hours of logs.
**Which issue(s) this PR fixes**:
Fixes #<issue number>
**Special notes for your reviewer**:
**Checklist**
- [x] Reviewed the `CONTRIBUTING.md` guide
- [x] Documentation added
- [x] Tests updated
- [x] `CHANGELOG.md` updated
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/upgrading/_index.md`
Signed-off-by: Edward Welch <edward.welch@grafana.com>
Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
**What this PR does / why we need it**:
PR documents the default value for `delete_ring_tokens` params on
`/ingester/shutdown` endpoint.
**Which issue(s) this PR fixes**:
Fixes #NA
**Special notes for your reviewer**:
**Checklist**
- [x] Documentation added
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
When sharded deletes that are partially completed are force-removed,
everything is removed except the shards that are currently being
deleted. The shards being deleted aren't necessarily in numerical order
so there's no way to know assume which shards have and haven't been
deleted.
The Delete store assumed it could look up requests in sequential order
and returned empty requests where there were gaps in the stored data.
This change takes the `sequence number` from the stored key of the
delete requests it finds and uses that to look up details.
**What this PR does / why we need it**:
- Fix a typo
- Fix the links in for the deletion API
**Checklist**
- [X] Documentation added
- [ ] Tests updated
- [ ] Is this an important fix or new feature? Add an entry in the
`CHANGELOG.md`.
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/upgrading/_index.md`
Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
* Refactor the store API to be in terms of DeleteRequest
* backfill remove tests
* backfill GetAll tests
* Make the delete request store deal in groups of requests instead of single ones
* Make cancelation aware of request groups
* remove cancelation deadline
* shard deletes based on a query parameter
* merge request group on gets
* global configuration for max query range
* sort delete requests by start time so batches are grouped to similar chunks
* Fix status when some deletes haven't happened yet
* fix tests
* Review feedback
* review feedback
* make max_interval > deletion interval a validation error
* list partially completed deletes
* list partially completed deletes
* fix flaky test
* clarify naming
* disallow cancelation of partial deletes unless the 'force' flag is passed
* more comment specificity for the delete request hash
* break out the middleware
* Add validation to the API
- Time must be in RFC3339 or a 10-digit-unix-seconds timestamp
- Start time must always exist
* lint
* docs
* improve tests
* cleanup
* cleanup
* access runtime config via function on validation
* Add API delete curl examples
* Update docs/sources/api/_index.md
Co-authored-by: Jennifer Villa <jvilla2013@gmail.com>
* Update docs/sources/api/_index.md
Co-authored-by: Jennifer Villa <jvilla2013@gmail.com>
* Update docs/sources/api/_index.md
Co-authored-by: Jennifer Villa <jvilla2013@gmail.com>
* Update docs/sources/api/_index.md
Co-authored-by: Jennifer Villa <jvilla2013@gmail.com>
Co-authored-by: Jennifer Villa <jvilla2013@gmail.com>
* Update grafana/dskit to 07166f9
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
* Add /ingester/shutdown handler
This handler replaces the deprecated /ingester/flush_shutdown handler
and can be used to gracefully shut down a Loki instance and delete the
file that persists the tokens of the ingester ring.
In production environments you usually want to persist ring tokens so
that during a restart of an ingester instance, or during rollout, the
tokens from that instance are not re-distributed to other instances, but
instead kept so that the same streams end up on the same instance once
it is up and running again. For that, the tokens are written to a file
that can be specified via the `-ingester.tokens-file-path` argument.
In certain cases, however, you want to forget the tokens and
re-distribute them when shutting down an ingester instance. This was
already possible by calling `/ingester/flush_shutdown`, deleting the
tokens file and terminating the process. The new handler
`/ingester/shutdown` combines these manual steps into a
single handler.
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
* Add changelog entry
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
* docs: describe ordering in http query results
* improved text
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
* better text
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
* better text
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
* Add docs for updated deletion api
Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
* Fix parameter name
Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
* Clarify that only the delete requests for the current tenant are shown
Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
* Update docs/sources/api/_index.md
Co-authored-by: Christian Simon <simon@swine.de>
* Update docs/sources/operations/storage/logs-deletion.md
Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>
* Update docs/sources/operations/storage/logs-deletion.md
Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>
* Update docs/sources/operations/storage/logs-deletion.md
Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>
* Specify all delete endpoints use authentication
Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
* Add description of delete modes
Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
* Apply suggestions from code review
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
Co-authored-by: Christian Simon <simon@swine.de>
Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
* Documented /services API endpoint
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
* Grammar
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
* docs: update the labels timespan info
* updated docs
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
* updated docs
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
* Log time in queue per request
* Add querierID to log message
* Add unescape request url and tenant_ids to log message
* Add query enqueue time to metrics.go
* Remove "querier request dequeued" log from scheduler
* Address comments
* Rename enqueue_time to queue_time
* Store queue time in nanoseconds (int64)
* Use CanonicalMIMEHeaderKey for setting the httpgrpc header
* Implement new IngestionRateStrat Enum.
- This new IngestionRateStrat is used internally to represent the chosen
ingestion rate strategy, to avoid using plain strings.
* Add a ring to the distributor.
As of now, the distributor only has a pointer to its lifecycler. Since
the ring HTTP handler isn't implemented for the lifecycler but it is
for the ruler, this extends the distributor by also holding a ring
instance.
* Implement /distributor/ring page.
* Reuse existing global strat const.
* Add distributor ring HTTP page tests.
* Add missing CHANGELOG entry.
* Document missing ring status pages.
* Fork Cortex util/http.go into source code.
* update changelog, release notes, and upgrade guide for 2.4
* tweak release notes a little
* make path relative for kafka config
* update upgrade guide
* more docs
fix the cache size not applied to results cache.
* fix prefix name in ring_config
The product name is Grafana Loki. The first use of the product name on
each web page should be the full product name. After that, within each
page, it is fine to use just Loki.
This PR also set up some frontmatter with weights where it was missing
on a few files, and correct where "Loki" was not capitalized.
I merged their change before the CLA was signed. We reverted that in #4299 and now I'm reopening
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
As far as I can tell from reading the "next" documentation, the
/loki/api/v1/status/buildinfo endpoint should return the build
information. The /version endpoint is also listed further down in the
documentation but from the code prior to this change, it looks like it
should perform the same function as buildinfo endpoint so I have
removed it.
I've updated the documentation to reflect my changes.
Signed-off-by: Jack Baldry <jack.baldry@grafana.com>
* Return build info under `/version`.
Closes#3221
* Update documentation.
* Correct docs.
* Format version handler code.
* Ignore write error.
* Sort imports.
* Use `PrometheusVersion` struct.
* Set Go version.
- Revise section titles
- Add weight metadata to specify ordering of subsections
- Remove TOC info from content prose, as it is already in the right side navigation panel
- Remove a redundant section about upgrading.
