261 Commits

Author SHA1 Message Date
3a0a727110 userns: support --userns=auto
automatically pick an empty range and create an user namespace for the
container.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2020-04-06 16:32:36 +02:00
4352d58549 Add support for containers.conf
vendor in c/common config pkg for containers.conf

Signed-off-by: Qi Wang qiwan@redhat.com
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
2020-03-27 14:36:03 -04:00
b6954758bb Attempt manual removal of CNI IP allocations on refresh
We previously attempted to work within CNI to do this, without
success. So let's do it manually, instead. We know where the
files should live, so we can remove them ourselves instead. This
solves issues around sudden reboots where containers do not have
time to fully tear themselves down, and leave IP address
allocations which, for various reasons, are not stored in tmpfs
and persist through reboot.

Fixes #5433

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2020-03-19 17:20:31 -04:00
118e78c5d6 Add structure for new exec session tracking to DB
As part of the rework of exec sessions, we need to address them
independently of containers. In the new API, we need to be able
to fetch them by their ID, regardless of what container they are
associated with. Unfortunately, our existing exec sessions are
tied to individual containers; there's no way to tell what
container a session belongs to and retrieve it without getting
every exec session for every container.

This adds a pointer to the container an exec session is
associated with to the database. The sessions themselves are
still stored in the container.

Exec-related APIs have been restructured to work with the new
database representation. The originally monolithic API has been
split into a number of smaller calls to allow more fine-grained
control of lifecycle. Support for legacy exec sessions has been
retained, but in a deprecated fashion; we should remove this in
a few releases.

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2020-03-18 11:02:14 -04:00
521ff14d83 Revert "exec: get the exit code from sync pipe instead of file"
This reverts commit 4b72f9e4013411208751df2a92ab9f322d4da5b2.

Continues what began with revert of
d3d97a25e8c87cf741b2e24ac01ef84962137106 in previous commit.

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2020-03-09 09:50:55 -04:00
ac354ac94a Fix spelling mistakes in code found by codespell
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
2020-03-07 10:30:44 -05:00
4b72f9e401 exec: get the exit code from sync pipe instead of file
Before, we were getting the exit code from the file, in which we waited an arbitrary amount of time (5 seconds) for the file, and segfaulted if we didn't find it. instead, we should be a bit more certain conmon has sent the exit code. Luckily, it sends the exit code along the sync pipe fd, so we can read it from there

Adapt the ExecContainer interface to pass along a channel to get the pid and exit code from conmon, to be able to read both from the pipe

Signed-off-by: Peter Hunt <pehunt@redhat.com>
2020-03-03 15:35:35 -05:00
4004f646cd Add basic deadlock detection for container start/remove
We can easily tell if we're going to deadlock by comparing lock
IDs before actually taking the lock. Add a few checks for this in
common places where deadlocks might occur.

This does not yet cover pod operations, where detection is more
difficult (and costly) due to the number of locks being involved
being higher than 2.

Also, add some error wrapping on the Podman side, so we can tell
people to use `system renumber` when it occurs.

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2020-02-24 09:29:34 -05:00
921f29c902 populate resolv.conf with dnsname responses when in usernamespace
when using usernamespace, dnsname respondes from cni were not making it into the containers /etc/resolv.conf because of a timing issue.  this corrects that behavior.

Fixes: #5256

Signed-off-by: Brent Baude <bbaude@redhat.com>
2020-02-20 08:00:40 -06:00
c140ecdc9b Do not copy up when volume is not empty
When Docker performs a copy up, it first verifies that the volume
being copied into is empty; thus, for volumes that have been
modified elsewhere (e.g. manually copying into then), the copy up
will not be performed at all. Duplicate this behavior in Podman
by checking if the volume is empty before copying.

Furthermore, move setting copyup to false further up. This will
prevent a potential race where copy up could happen more than
once if Podman was killed after some files had been copied but
before the DB was updated.

This resolves CVE-2020-1726.

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2020-02-12 10:58:42 -05:00
6d487c781a Tear down network when restarting containers
This makes restart a bit slower for root containers, but it does
make it more consistent with `podman stop` and `podman start` on
a container. Importantly, `podman restart` will now recreate
firewall rules if they were somehow purged.

Fixes #5051

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2020-02-02 12:51:03 -05:00
67165b7675 make lint: enable gocritic
`gocritic` is a powerful linter that helps in preventing certain kinds
of errors as well as enforcing a coding style.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2020-01-13 14:27:02 +01:00
c3956b1974 address review comments before merge
Signed-off-by: baude <bbaude@redhat.com>
2020-01-10 09:47:19 -06:00
d924494f56 Initial commit on compatible API
Signed-off-by: Jhon Honce <jhonce@redhat.com>

Create service command

Use cd cmd/service && go build .

$ systemd-socket-activate -l 8081 cmd/service/service &
$ curl http://localhost:8081/v1.24/images/json

Signed-off-by: Jhon Honce <jhonce@redhat.com>

Correct Makefile

Signed-off-by: Jhon Honce <jhonce@redhat.com>

Two more stragglers

Signed-off-by: Jhon Honce <jhonce@redhat.com>

Report errors back as http headers

Signed-off-by: Jhon Honce <jhonce@redhat.com>

Split out handlers, updated output

Output aligned to docker structures

Signed-off-by: Jhon Honce <jhonce@redhat.com>

Refactored routing, added more endpoints and types

* Encapsulated all the routing information in the handler_* files.
* Added more serviceapi/types, including podman additions. See Info

Signed-off-by: Jhon Honce <jhonce@redhat.com>

Cleaned up code, implemented info content

* Move Content-Type check into serviceHandler
* Custom 404 handler showing the url, mostly for debugging
* Refactored images: better method names and explicit http codes
* Added content to /info
* Added podman fields to Info struct
* Added Container struct

Signed-off-by: Jhon Honce <jhonce@redhat.com>

Add a bunch of endpoints

containers: stop, pause, unpause, wait, rm
images: tag, rmi, create (pull only)

Signed-off-by: baude <bbaude@redhat.com>

Add even more handlers

* Add serviceapi/Error() to improve error handling
* Better support for API return payloads
* Renamed unimplemented to unsupported these are generic endpoints
  we don't intend to ever support.  Swarm broken out since it uses
  different HTTP codes to signal that the node is not in a swarm.
* Added more types
* API Version broken out so it can be validated in the future

Signed-off-by: Jhon Honce <jhonce@redhat.com>

Refactor to introduce ServiceWriter

Signed-off-by: Jhon Honce <jhonce@redhat.com>

populate pods endpoints

/libpod/pods/..

exists, kill, pause, prune, restart, remove, start, stop, unpause

Signed-off-by: baude <bbaude@redhat.com>

Add components to Version, fix Error body

Signed-off-by: Jhon Honce <jhonce@redhat.com>

Add images pull output, fix swarm routes

* docker-py tests/integration/api_client_test.py pass 100%
* docker-py tests/integration/api_image_test.py pass 4/16
+ Test failures include services podman does not support

Signed-off-by: Jhon Honce <jhonce@redhat.com>

pods endpoint submission 2

add create and others; only top and stats is left.

Signed-off-by: baude <bbaude@redhat.com>

Update pull image to work from empty registry

Signed-off-by: Jhon Honce <jhonce@redhat.com>

pod create and container create

first pass at pod and container create.  the container create does not
quite work yet but it is very close.  pod create needs a partial
rewrite.  also broken off the DELETE (rm/rmi) to specific handler funcs.

Signed-off-by: baude <bbaude@redhat.com>

Add docker-py demos, GET .../containers/json

* Update serviceapi/types to reflect libpod not podman
* Refactored removeImage() to provide non-streaming return

Signed-off-by: Jhon Honce <jhonce@redhat.com>

create container part2

finished minimal config needed for create container.  started demo.py
for upcoming talk

Signed-off-by: baude <bbaude@redhat.com>

Stop server after honoring request

* Remove casting for method calls
* Improve WriteResponse()
* Update Container API type to match docker API

Signed-off-by: Jhon Honce <jhonce@redhat.com>

fix namespace assumptions

cleaned up namespace issues with libpod.

Signed-off-by: baude <bbaude@redhat.com>

wip

Signed-off-by: baude <bbaude@redhat.com>

Add sliding window when shutting down server

* Added a Timeout rather than closing down service on each call
* Added gorilla/schema dependency for Decode'ing query parameters
* Improved error handling
* Container logs returned and multiplexed for stdout and stderr
  * .../containers/{name}/logs?stdout=True&stderr=True
* Container stats
  * .../containers/{name}/stats

Signed-off-by: Jhon Honce <jhonce@redhat.com>

Improve error handling

* Add check for at least one std stream required for /containers/{id}/logs
* Add check for state in /containers/{id}/top
* Fill in more fields for /info
* Fixed error checking in service start code

Signed-off-by: Jhon Honce <jhonce@redhat.com>

get rest  of image tests for pass

Signed-off-by: baude <bbaude@redhat.com>

linting our content

Signed-off-by: baude <bbaude@redhat.com>

more linting

Signed-off-by: baude <bbaude@redhat.com>

more linting

Signed-off-by: baude <bbaude@redhat.com>

pruning

Signed-off-by: baude <bbaude@redhat.com>

[CI:DOCS]apiv2 pods

migrate from using args in the url to using a json struct in body for
pod create.

Signed-off-by: baude <bbaude@redhat.com>

fix handler_images prune

prune's api changed slightly to deal with filters.

Signed-off-by: baude <bbaude@redhat.com>

[CI:DOCS]enabled base container create tests

enabling the base container create tests which allow us to get more into
the stop, kill, etc tests. many new tests now pass.

Signed-off-by: baude <bbaude@redhat.com>

serviceapi errors: append error message to API message

I dearly hope this is not breaking any other tests but debugging
"Internal Server Error" is not helpful to any user.  In case, it
breaks tests, we can rever the commit - that's why it's a small one.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>

serviceAPI: add containers/prune endpoint

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>

add `service` make target

Also remove the non-functional sub-Makefile.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>

add make targets for testing the service

 * `sudo make run-service` for running the service.

 * `DOCKERPY_TEST="tests/integration/api_container_test.py::ListContainersTest" \
 	make run-docker-py-tests`
   for running a specific tests.  Run all tests by leaving the env
   variable empty.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>

Split handlers and server packages

The files were split to help contain bloat. The api/server package will
contain all code related to the functioning of the server while
api/handlers will have all the code related to implementing the end
points.

api/server/register_* will contain the methods for registering
endpoints.  Additionally, they will have the comments for generating the
swagger spec file.

See api/handlers/version.go for a small example handler,
api/handlers/containers.go contains much more complex handlers.

Signed-off-by: Jhon Honce <jhonce@redhat.com>

[CI:DOCS]enabled more tests

Signed-off-by: baude <bbaude@redhat.com>

[CI:DOCS]libpod endpoints

small refactor for libpod inclusion and began adding endpoints.

Signed-off-by: baude <bbaude@redhat.com>

Implement /build and /events

* Include crypto libraries for future ssh work

Signed-off-by: Jhon Honce <jhonce@redhat.com>

[CI:DOCS]more image implementations

convert from using for to query structs among other changes including
new endpoints.

Signed-off-by: baude <bbaude@redhat.com>

[CI:DOCS]add bindings for golang

Signed-off-by: baude <bbaude@redhat.com>

[CI:DOCS]add volume endpoints for libpod

create, inspect, ls, prune, and rm

Signed-off-by: baude <bbaude@redhat.com>

[CI:DOCS]apiv2 healthcheck enablement

wire up container healthchecks for the api.

Signed-off-by: baude <bbaude@redhat.com>

[CI:DOCS]Add mount endpoints

via the api, allow ability to mount a container and list container
mounts.

Signed-off-by: baude <bbaude@redhat.com>

[CI:DOCS]Add search endpoint

add search endpoint with golang bindings

Signed-off-by: baude <bbaude@redhat.com>

[CI:DOCS]more apiv2 development

misc population of methods, etc

Signed-off-by: baude <bbaude@redhat.com>

rebase cleanup and epoch reset

Signed-off-by: baude <bbaude@redhat.com>

[CI:DOCS]add more network endpoints

also, add some initial error handling and convenience functions for
standard endpoints.

Signed-off-by: baude <bbaude@redhat.com>

[CI:DOCS]use helper funcs for bindings

use the methods developed to make writing bindings less duplicative and
easier to use.

Signed-off-by: baude <bbaude@redhat.com>

[CI:DOCS]add return info for prereview

begin to add return info and status codes for errors so that we can
review the apiv2

Signed-off-by: baude <bbaude@redhat.com>

[CI:DOCS]first pass at adding swagger docs for api

Signed-off-by: baude <bbaude@redhat.com>
2020-01-10 09:41:39 -06:00
2a5c235f78 Ensure SizeRw is shown when a user does 'inspect --size -t container'.
Currently, if a user requests the size on a container (inspect --size -t container),
the SizeRw does not show up if the value is 0. It's because InspectContainerData is
defined as int64 and there is an omit when empty.

We do want to display it even if the value is empty. I have changed the type of SizeRw to be a pointer to an int64 instead of an int64. It will allow us todistinguish the empty value to the missing value.

I updated the test "podman inspect container with size" to ensure we check thatSizeRw is displayed correctly.

Closes #4744

Signed-off-by: NevilleC <neville.cain@qonto.eu>
2019-12-27 23:49:34 +01:00
123b8c627d if container is not in a pid namespace, stop all processes
When a container is in a PID namespace, it is enought to send
the stop signal to the PID 1 of the namespace, only send signals
to all processes in the container when the container is not in
a pid namespace.

Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
2019-12-19 13:33:17 -05:00
bd44fd5c81 Reap exec sessions on cleanup and removal
We currently rely on exec sessions being removed from the state
by the Exec() API itself, on detecting the session stopping. This
is not a reliable method, though. The Podman frontend for exec
could be killed before the session ended, or another Podman
process could be holding the lock and prevent update (most
notable in `run --rm`, when a container with an active exec
session is stopped).

To resolve this, add a function to reap active exec sessions from
the state, and use it on cleanup (to clear sessions after the
container stops) and remove (to do the same when --rm is passed).
This is a bit more complicated than it ought to be because Kata
and company exist, and we can't guarantee the exec session has a
PID on the host, so we have to plumb this through to the OCI
runtime.

Fixes #4666

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2019-12-12 16:35:37 -05:00
25cc43c376 Add ContainerStateRemoving
When Libpod removes a container, there is the possibility that
removal will not fully succeed. The most notable problems are
storage issues, where the container cannot be removed from
c/storage.

When this occurs, we were faced with a choice. We can keep the
container in the state, appearing in `podman ps` and available for
other API operations, but likely unable to do any of them as it's
been partially removed. Or we can remove it very early and clean
up after it's already gone. We have, until now, used the second
approach.

The problem that arises is intermittent problems removing
storage. We end up removing a container, failing to remove its
storage, and ending up with a container permanently stuck in
c/storage that we can't remove with the normal Podman CLI, can't
use the name of, and generally can't interact with. A notable
cause is when Podman is hit by a SIGKILL midway through removal,
which can consistently cause `podman rm` to fail to remove
storage.

We now add a new state for containers that are in the process of
being removed, ContainerStateRemoving. We set this at the
beginning of the removal process. It notifies Podman that the
container cannot be used anymore, but preserves it in the DB
until it is fully removed. This will allow Remove to be run on
these containers again, which should successfully remove storage
if it fails.

Fixes #3906

Signed-off-by: Matthew Heon <mheon@redhat.com>
2019-11-19 15:38:03 -05:00
fa415f07a1 Also delete winsz fifo
In conmon 2.0.3, we add another fifo to handle window resizing. This needs to be cleaned up for commands like restore, where the same path is used.

Signed-off-by: Peter Hunt <pehunt@redhat.com>
2019-11-15 12:44:15 -05:00
5f8bf3d07d Add ensureState helper for checking container state
We have a lot of checks for container state scattered throughout
libpod. Many of these need to ensure the container is in one of a
given set of states so an operation may safely proceed.
Previously there was no set way of doing this, so we'd use unique
boolean logic for each one. Introduce a helper to standardize
state checks.

Note that this is only intended to replace checks for multiple
states. A simple check for one state (ContainerStateRunning, for
example) should remain a straight equality, and not use this new
helper.

Signed-off-by: Matthew Heon <mheon@redhat.com>
2019-10-28 13:09:01 -04:00
cab7bfbb21 Add a MissingRuntime implementation
When a container is created with a given OCI runtime, but then it
is uninstalled or removed from the configuration file, Libpod
presently reacts very poorly. The EvictContainer code can
potentially remove these containers, but we still can't see them
in `podman ps` (aside from the massive logrus.Errorf messages
they create).

Providing a minimal OCI runtime implementation for missing
runtimes allows us to behave better. We'll be able to retrieve
containers from the database, though we still pop up an error for
each missing runtime. For containers which are stopped, we can
remove them as normal.

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2019-10-15 15:59:20 -04:00
17a7596af4 Unwrap errors before comparing them
Unwrap errors before directly comparing them with errors defined by the
storage and image libraries.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2019-10-14 13:49:06 -04:00
6f630bc09b Move OCI runtime implementation behind an interface
For future work, we need multiple implementations of the OCI
runtime, not just a Conmon-wrapped runtime matching the runc CLI.

As part of this, do some refactoring on the interface for exec
(move to a struct, not a massive list of arguments). Also, add
'all' support to Kill and Stop (supported by runc and used a bit
internally for removing containers).

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2019-10-10 10:19:32 -04:00
c35d71e3da catch runc v2 error
when runc returns an error about not being v2 complient, catch the error
and logrus an actionable message for users.

Signed-off-by: baude <bbaude@redhat.com>
2019-10-09 09:15:18 -05:00
ec940b08c6 rootless: do not attempt a CNI refresh
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2019-10-01 14:10:04 +02:00
e87012de12 Merge pull request #4065 from mheon/unconditional_conmon_rm
Unconditionally remove conmon files before starting
2019-09-27 15:08:14 -07:00
b57d2f4cc7 Force a CNI Delete on refreshing containers
CNI expects that a DELETE be run before re-creating container
networks. If a reboot occurs quickly enough that containers can't
stop and clean up, that DELETE never happens, and Podman
currently wipes the old network info and thinks the state has
been entirely cleared. Unfortunately, that may not be the case on
the CNI side. Some things - like IP address reservations - may
not have been cleared.

To solve this, manually re-run CNI Delete on refresh. If the
container has already been deleted this seems harmless. If not,
it should clear lingering state.

Fixes: #3759

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2019-09-24 09:52:11 -04:00
407fba4942 Unconditionally remove conmon files before starting
We've been seeing a lot of issues (ref: #4061, but there are
others) where Podman hiccups on trying to start a container,
because some temporary files have been retained and Conmon will
not overwrite them.

If we're calling start() we can safely assume that we really want
those files gone so the container starts without error, so invoke
the cleanup routine. It's relatively cheap (four file removes) so
it shouldn't hurt us that much.

Also contains a small simplification to the removeConmonFiles
logic - we don't need to stat-then-remove when ignoring ENOENT is
fine.

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2019-09-20 09:30:15 -04:00
cabe1345f8 Unmounting a container that is already unmounted is OK
We should not be throwing errors because the operation we wanted
to perform is already done. Now, it is definitely strange that a
container is actually unmounted, but shows as mounted in the DB -
if this reoccurs in a way where we can investigate, it's worth
tearing into.

Fixes #4033

Signed-off-by: Matthew Heon <mheon@redhat.com>
2019-09-16 09:22:26 -04:00
88ebc33840 Report errors when trying to pause rootless containers
If you are running a rootless container on cgroupV1
you can not pause the container.  We need to report the proper error
if this happens.

Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
2019-09-13 08:29:46 -04:00
7b68cd0b3d clean up after healthcheck execs
when executing a healthcheck, we were not cleaning up after exec's use
of a socket.  we now remove the socket file and ignore if for reason it
does not exist.

Fixes: #3962

Signed-off-by: baude <bbaude@redhat.com>
2019-09-12 14:30:46 -05:00
7ac6ed3b4b Merge pull request #3581 from mheon/no_cgroups
Support running containers without CGroups
2019-09-11 00:58:46 +02:00
c2284962c7 Add support for launching containers without CGroups
This is mostly used with Systemd, which really wants to manage
CGroups itself when managing containers via unit file.

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2019-09-10 10:52:37 -04:00
b6106341fb When first mounting any named volume, copy up
Previously, we only did this for volumes created at the same time
as the container. However, this is not correct behavior - Docker
does so for all named volumes, even those made with
'podman volume create' and mounted into a container later.

Fixes #3945

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2019-09-09 17:17:39 -04:00
de9a394fcf Correctly report errors on unmounting SHM
When we fail to remove a container's SHM, that's an error, and we
need to report it as such. This may be part of our lingering
storage woes.

Also, remove MNT_DETACH. It may be another cause of the storage
removal failures.

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2019-09-05 17:12:27 -04:00
a760e325f3 Add ability for volumes with options to mount/umount
When volume options and the local volume driver are specified,
the volume is intended to be mounted using the 'mount' command.
Supported options will be used to volume the volume before the
first container using it starts, and unmount the volume after the
last container using it dies.

This should work for any local filesystem, though at present I've
only tested with tmpfs and btrfs.

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2019-09-05 17:12:27 -04:00
56a65cffac generate systemd: support pods and geneartig files
Support generating systemd unit files for a pod.  Podman generates one
unit file for the pod including the PID file for the infra container's
conmon process and one unit file for each container (excluding the infra
container).

Note that this change implies refactorings in the `pkg/systemdgen` API.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2019-08-21 17:28:30 +02:00
909ab59419 container stop: kill conmon
Old versions of conmon have a bug where they create the exit file before
closing open file descriptors causing a race condition when restarting
containers with open ports since we cannot bind the ports as they're not
yet closed by conmon.

Killing the old conmon PID is ~okay since it forces the FDs of old
conmons to be closed, while it's a NOP for newer versions which should
have exited already.

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
2019-08-05 09:16:18 +02:00
9dcd76e369 Ensure we generate a 'stopped' event on force-remove
When forcibly removing a container, we are initiating an explicit
stop of the container, which is not reflected in 'podman events'.
Swap to using our standard 'stop()' function instead of a custom
one for force-remove, and move the event into the internal stop
function (so internal calls also register it).

This does add one more database save() to `podman remove`. This
should not be a terribly serious performance hit, and does have
the desirable side effect of making things generally safer.

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2019-07-31 17:29:14 -04:00
ebacfbd091 podman: fix memleak caused by renaming and not deleting
the exit file

If the container exit code needs to be retained, it cannot be retained
in tmpfs, because libpod runs in a memcg itself so it can't leave
traces with a daemon-less design.

This wasn't a memleak detectable by kmemleak for example. The kernel
never lost track of the memory and there was no erroneous refcounting
either. The reference count dependencies however are not easy to track
because when a refcount is increased, there's no way to tell who's
still holding the reference. In this case it was a single page of
tmpfs pagecache holding a refcount that kept pinned a whole hierarchy
of dying memcg, slab kmem, cgropups, unrechable kernfs nodes and the
respective dentries and inodes. Such a problem wouldn't happen if the
exit file was stored in a regular filesystem because the pagecache
could be reclaimed in such case under memory pressure. The tmpfs page
can be swapped out, but that's not enough to release the memcg with
CONFIG_MEMCG_SWAP_ENABLED=y.

No amount of more aggressive kernel slab shrinking could have solved
this. Not even assigning slab kmem of dying cgroups to alive cgroup
would fully solve this. The only way to free the memory of a dying
cgroup when a struct page still references it, would be to loop over
all "struct page" in the kernel to find which one is associated with
the dying cgroup which is a O(N) operation (where N is the number of
pages and can reach billions). Linking all the tmpfs pages to the
memcg would cost less during memcg offlining, but it would waste lots
of memory and CPU globally. So this can't be optimized in the kernel.

A cronjob running this command can act as workaround and will allow
all slab cache to be released, not just the single tmpfs pages.

    rm -f /run/libpod/exits/*

This patch solved the memleak with a reproducer, booting with
cgroup.memory=nokmem and with selinux disabled. The reason memcg kmem
and selinux were disabled for testing of this fix, is because kmem
greatly decreases the kernel effectiveness in reusing partial slab
objects. cgroup.memory=nokmem is strongly recommended at least for
workstation usage. selinux needs to be further analyzed because it
causes further slab allocations.

The upstream podman commit used for testing is
1fe2965e4f672674f7b66648e9973a0ed5434bb4 (v1.4.4).

The upstream kernel commit used for testing is
f16fea666898dbdd7812ce94068c76da3e3fcf1e (v5.2-rc6).

Reported-by: Michele Baldessari <michele@redhat.com>
Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>

<Applied with small tweaks to comments>
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2019-07-31 17:28:42 -04:00
6665269ab8 Merge pull request #3233 from wking/fatal-requested-hook-directory-does-not-exist
libpod/container_internal: Make all errors loading explicitly configured hook dirs fatal
2019-07-29 16:39:08 +02:00
a1a79c08b7 Implement conmon exec
This includes:
	Implement exec -i and fix some typos in description of -i docs
	pass failed runtime status to caller
	Add resize handling for a terminal connection
	Customize exec systemd-cgroup slice
	fix healthcheck
	fix top
	add --detach-keys
	Implement podman-remote exec (jhonce)
	* Cleanup some orphaned code (jhonce)
	adapt remote exec for conmon exec (pehunt)
	Fix healthcheck and exec to match docs
		Introduce two new OCIRuntime errors to more comprehensively describe situations in which the runtime can error
		Use these different errors in branching for exit code in healthcheck and exec
	Set conmon to use new api version

Signed-off-by: Jhon Honce <jhonce@redhat.com>

Signed-off-by: Peter Hunt <pehunt@redhat.com>
2019-07-22 15:57:23 -04:00
db826d5d75 golangci-lint round #3
this is the third round of preparing to use the golangci-lint on our
code base.

Signed-off-by: baude <bbaude@redhat.com>
2019-07-21 14:22:39 -05:00
a78c885397 golangci-lint pass number 2
clean up and prepare to migrate to the golangci-linter

Signed-off-by: baude <bbaude@redhat.com>
2019-07-11 09:13:06 -05:00
edc7f52c95 Merge pull request #3425 from adrianreber/restore-mount-label
Set correct SELinux label on restored containers
2019-07-08 20:31:59 +02:00
1d36501f96 code cleanup
clean up code identified as problematic by golands inspection

Signed-off-by: baude <bbaude@redhat.com>
2019-07-08 09:18:11 -05:00
f7407f2eb5 Merge pull request #3472 from haircommander/generate-volumes
generate kube with volumes
2019-07-04 22:22:07 +02:00
fec1de6ef4 trivial cleanups from golang
the results of a code cleanup performed by the goland IDE.

Signed-off-by: baude <bbaude@redhat.com>
2019-07-03 15:41:33 -05:00
38c6199b80 Wipe PID and ConmonPID in state after container stops
Matches the behavior of Docker.

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2019-07-02 19:10:51 -04:00
a1bb1987cc Store Conmon's PID in our state and display in inspect
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
2019-07-02 18:52:55 -04:00