When running a single podman logs this is not really important since we
will exit when we finish reading the logs. However for the system
service this is very important. Leaking goroutines will cause an
increased memory and CPU ussage over time.
Both the the event and log backend have goroutine leaks with both the
file and journald drivers.
The journald backend has the problem that journal.Wait(IndefiniteWait)
will block until we get a new journald event. So when a client closes
the connection the goroutine would still wait until there is a new
journal entry. To fix this we just wait for a maximum of 5 seconds,
after that we can check if the client connection was closed and exit
correctly in this case.
For the file backend we can fix this by waiting for either the log line
or context cancel at the same time. Currently it would block waiting for
new log lines and only check afterwards if the client closed the
connection and thus hang forever if there are no new log lines.
[NO NEW TESTS NEEDED] I am open to ideas how we can test memory leaks in
CI.
To test manually run a container like this:
`podman run --log-driver $driver --name test -d alpine sh -c 'i=1; while [ "$i" -ne 1000 ]; do echo "line $i"; i=$((i + 1)); done; sleep inf'`
where `$driver` can be either `journald` or `k8s-file`.
Then start the podman system service and use:
`curl -m 1 --output - --unix-socket $XDG_RUNTIME_DIR/podman/podman.sock -v 'http://d/containers/test/logs?follow=1&since=0&stderr=1&stdout=1' &>/dev/null`
to get the logs from the API and then it closes the connection after 1 second.
Now run the curl command several times and check the memory usage of the service.
Fixes#14879
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
We now use the golang error wrapping format specifier `%w` instead of
the deprecated github.com/pkg/errors package.
[NO NEW TESTS NEEDED]
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
Lint the systemd code and fix the reported problems.
The remoteclient tag is no longer used so I just removed it.
[NO NEW TESTS NEEDED]
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
--tail=1 is not working f you restart a container with journald logging.
We see the exit status and then call into the logging a second time
causing all of the logs to print.
Removing the tail log on exited seems to fix the problem.
Fixes: https://github.com/containers/podman/issues/13098
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
When reading logs from the journal, keep going after the container
exits, in case it gets restarted.
Events logged to the journal via the normal paths don't include
CONTAINER_ID_FULL, so don't bother adding it to the "history" event we
use to force at least one entry for the container to show up in the log.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
The following problems regarding `logs --tail` with the journald log
driver are fixed:
- One more line than a specified value is displayed.
- '--tail 0' displays all lines while the other log drivers displays
nothing.
- Partial lines are not considered.
- If the journald events backend is used and a container has exited,
nothing is displayed.
Integration tests that should have detected the bugs are also fixed. The
tests are executed with json-file log driver three times without this
fix.
Signed-off-by: Hironori Shiina <shiina.hironori@jp.fujitsu.com>
When looking for a cursor that matches the first journal entry for a
given container, wait and try to find it using exponential backoff.
[NO NEW TESTS NEEDED]
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When reading log entries from the journal, don't skip past the first
matching entry after we've positioned the cursor at it.
Make the first blank-line entry that we logged so that the container
would always have at least one log entry for us to find (until it gets
vacuumed out, at least) a fake history entry, so that `logs` doesn't
pass it on for display.
CI already has tests that exercise journal-based logging, so
[NO TESTS NEEDED]
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
For docker compat include information about available volume, log and
network drivers which should be listed under the plugins key.
Fixes#11265
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
compat containers/logs was missing actual usage of until query param.
This led me to implement the until param for libpod's container logs as well. Added e2e tests.
Signed-off-by: cdoern <cdoern@redhat.com>
Fix a race in journald driver. Following the logs implies streaming
until the container is dead. Streaming happened in one goroutine,
waiting for the container to exit/die and signaling that event happened
in another goroutine.
The nature of having two goroutines running simultaneously is pretty
much the core of the race condition. When the streaming goroutines
received the signal that the container has exitted, the routine may not
have read and written all of the container's logs.
Fix this race by reading both, the logs and the events, of the container
and stop streaming when the died/exited event has been read. The died
event is guaranteed to be after all logs in the journal which guarantees
not only consistencty but also a deterministic behavior.
Note that the journald log driver now requires the journald event
backend to be set.
Fixes: #10323
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
The initial version of libimage changed the order of layers which has
now been restored to remain backwards compatible.
Further changes:
* Fix a bug in the journald logging which requires to strip trailing
new lines from the message. The system tests did not pass due to
empty new lines. Triggered by changing the default logger to
journald in containers/common.
* Fix another bug in the journald logging which embedded the container
ID inside the message rather than the specifid field. That surfaced
in a preceeding whitespace of each log line which broke the system
tests.
* Alter the system tests to make sure that the k8s-file and the
journald logging drivers are executed.
* A number of e2e tests have been changed to force the k8s-file driver
to make them pass when running inside a root container.
* Increase the timeout in a kill test which seems to take longer now.
Reasons are unknown. Tests passed earlier and no signal-related
changes happend. It may be CI VM flake since some system tests but
other flaked.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Some log tests were duplicated, and some didn't need to be repeated for
every driver. Also, added some comments
Signed-off-by: Ashley Cui <acui@redhat.com>
We missed bumping the go module, so let's do it now :)
* Automated go code with github.com/sirkon/go-imports-rename
* Manually via `vgrep podman/v2` the rest
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
A podman could not read logs written to journald properly, due to a tail config bug.
Added a system test to check this - since e2e tests don't like journald
Signed-off-by: Ashley Cui <acui@redhat.com>
Add a `context.Context` to the log APIs to allow for cancelling
streaming (e.g., via `podman logs -f`). This fixes issues for
the remote API where some go routines of the server will continue
writing and produce nothing but heat and waste CPU cycles.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
With the advent of Podman 2.0.0 we crossed the magical barrier of go
modules. While we were able to continue importing all packages inside
of the project, the project could not be vendored anymore from the
outside.
Move the go module to new major version and change all imports to
`github.com/containers/libpod/v2`. The renaming of the imports
was done via `gomove` [1].
[1] https://github.com/KSubedi/gomove
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Update the outdated systemd and dbus dependencies which are now provided
as go modules. This will further tighten our dependencies and releases
and pave the way for the upcoming auto-update feature.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
change the default to -1, so that we can change the semantic of
"--tail 0" to not print any existing log line.
Closes: https://github.com/containers/libpod/issues/4396
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Add a journald reader that translates the journald entry to a k8s-file formatted line, to be added as a log line
Note: --follow with journald hasn't been implemented. It's going to be a larger undertaking that can wait.
Signed-off-by: Peter Hunt <pehunt@redhat.com>
