**What this PR does / why we need it**:
Add support for natively supporting logs ingestion in OTLP format.
`/otlp/v1/logs` is the new endpoint where users can push logs in OTLP
format. It accepts logs serialized in JSON or proto format. Since OTEL
format is very different than what Loki storage model, here is how data
in OTEL format will be mapped to Loki data model:
* Index labels: The Resource Attributes map quite well to Index labels
in Loki since both usually identify the source of the logs. The problem
however is that Resource attributes in OTLP can have an unbounded number
of values while Loki has a default limit of having up to 30 labels.
Since Index labels in Loki can largely drive the kind of querying
experience the users are going to have, we have chosen select attributes
which would be picked as Index Labels. The ones that are not picked up
as Index labels would be stored as Structured Metadata with each log
entry.
* Timestamp: LogRecord.TimeUnixNano
* LogLine: LogRecord.Body holds the body of the log. However, since Loki
only supports Log body in string format, we will stringify non-string
values using [AsString method from OTEL collector
lib](ab3d6c5b64/pdata/pcommon/value.go (L353)).
* Structured Metadata: Anything which can’t be stored in Index labels
and LogLine. Here is a non-exhaustive list of what will be stored in
Structured Metadata to give a sense of what it will hold:
* Resource Attributes not stored as Index labels is replicated and
stored with each log entry.
* Everything under InstrumentationScope is replicated and stored with
each log entry.
* Everything under LogRecord except LogRecord.Body,
LogRecord.TimeUnixNano and sometimes LogRecord.ObservedTimestamp.
*NOTES*:
* Since Loki does not support `.` or any other special characters other
than `_` in label names, we replace all non-supported characters with
`_`.
* Since Loki only supports string in values of Index Labels and
Structured Metadata, all the complex types are converted as follows:
* Map would be flattened into label keys using `_` as separator, same as
how we do it in [json parser in
LogQL](https://grafana.com/docs/loki/latest/query/log_queries/#json).
* Everything else is stringified using [AsString method from OTEL
collector
lib](ab3d6c5b64/pdata/pcommon/value.go (L353))
**Special notes for your reviewer**:
I will open follow-up PRs for:
* Documentation
* Make blessed attributes list configurable per tenant.
**Checklist**
- [x] Tests updated
- [x] `CHANGELOG.md` updated
- [ ] If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
**What this PR does / why we need it**:
Debugging integration tests can be difficult without Loki's logs being
output.
Changing the call from `cluster.New()` to `cluster.New(true,
level.WarnValue())` now allows the author the option to temporarily or
permanently enable logging for their test, as well as configure a level
filter to control the noise.
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
**What this PR does / why we need it**:
This adds a 3rd target to SSD/Scalable mode, as well as a config flag to
run the legacy `read` mode from the original 2 target configuration in
order to give people time to migrate before we remove this option in
Loki 3.0 (hopefully).
The 3rd target has the two major advantages:
1. Allows the read path to be run as a deployment and thus auto-scaled
using our existing auto-scaling logic for queriers
2. Creates consistency with Mimir since they went with a 3 target model
for their SSD deployment.
**What this PR does / why we need it**:
- Reduce flakiness of GCPLog tests by reusing context and stopping GCP
topic
- Reduce flakiness of integration tests by:
- using port as they're assigned
- reusing context
- adding timeout to calls (hence, if something get stuck, tests fail
earlier)
