The intent here is to remove an extension of a file, namely .gz. This is currently
implemented with a strings.Trim, which will remove ALL occurrences of ., g, and z
from the start and end of the string. For example, that'd break egg.gz as it'd return just e.
This commit restores the intent of removing only the .gz suffix.
I haven't been able to find an exploitable path for this, luckily, so I'm only aiming
to restore the intent that seems to be originally held here.
(Detected with help of CodeQL)
The region (and other settings) for the S3 configuration can be specified in various ways.
When the region was already set by LoadDefaultConfig it should not be set to InvalidAWSRegion.
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
### Summary
This PR fixes a regression in how Loki's S3 configuration options are handled to create the S3 client which was introduced with the upgrade of the AWS SDK from v1 to v2 (#19205).
With the v1 SDK the `s3` field of the `aws` configuration block was parsed to extract credentials, region, host, and schema.
```
storage_config:
aws:
s3: s3://<accesskey>:<secret>@<endpoint_or_region>/<bucketnames>
```
After the upgrade, only credentials were extraced, leaving all other parts of the URL as default, which would require explicit overrides.
This was a breaking change and not documented in the upgrade notes. We want to keep backwards compatibility, even though the Thanos object client is going to supersed the custom S3/GCS/... clients.
ref: #19908
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
### Summary
The v1 engine has a mechanism to rename labels in case they have the same name but different origin, such as labels, structured metadata, or parsed fields.
1. In case a log line has a structured metadata key with the same name as the label name of the stream, than the metadata key is suffixed with `_extracted`, such as `service_extracted`, if `service` exists in both `labels` and `metadata`.
2. In case a parser creates a parsed field with the same as the label name of the stream, then the parsed key is suffixed with `_extracted` in the same way as case 1. However, if the field name also collides with a structured metadata key, then the extracted structured metadata is replaced with the extracted parsed field.
This PR only implements the first case. As a follow up PR, the second case needs to be implemented as well. Additionally, the newly introduced "compatibility node" should also be made optional with a feature flag and/or per-request.
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
When downloading a potentially compressed file, the file is first downloaded into a temp file and this temp file is then re-opened and processed. The re-opened *File is Close()d as expected but the initial temp *File is not.
This PR adds the missing Close().
The download queue has a fixed size of 100k elements and is implemented using a channel. Since the channel has a fixed size, this will allocate `capacity * object size`, where the object is `downloadRequest[BlockRef, BlockDirectory]`.
This PR changes the channel from a channel of objects (`chan downloadRequest[T, R]`) to a channel of pointers (`chan *downloadRequest[T, R]`) which should significantly reduce the permanently allocated memory for the queue.
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
This is another step towards supporting Prometheus' stringlabels implementation in Loki. It adds support in the `logql/log` package.
Part of https://github.com/grafana/loki/issues/17122
The tests should now compile and pass with `-tags stringlabels`.
---
Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>
