Originally, during pod removal, we locked every container in the
pod at once, did a number of validity checks to ensure everything
was safe, and then removed all the containers in the pod.
A deadlock was recently discovered with this approach. In brief,
we cannot lock the entire pod (or much more than a single
container at a time) without causing a deadlock. As such, we
converted to an approach where we just looped over each container
in the pod, removing them individually. Unfortunately, this
removed a lot of the validity checking of the earlier approach,
allowing for a lot of unintended bad things. Infra containers
could be removed while containers in the pod still depended on
them, for example.
There's no easy way to do validity checks while in a simple loop,
so I implemented a version of our graph-traversal logic that
currently handles pod start. This version acts in the reverse
order of startup: startup starts from containers which depend on
nothing and moves outwards, while removal acts on containers which
have nothing depend on them and moves inwards. By doing graph
traversal, we can guarantee that nothing is removed while
something that depends on it still exists - so the infra
container should be the last thing in a pod that is removed, for
example.
In the (unlikely) case that a graph of the pod's containers
cannot be built (most likely impossible without database editing)
the old method of pod removal has been retained to ensure that
even misbehaving pods can be forcibly evicted from the state.
I'm fairly confident that this resolves the problem, but there
are a lot of assumptions around dependency structure built into
the original pod removal code and I am not 100% sure I have
captured all of them.
Fixes#15526
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
In view of https://github.com/containers/storage/pull/1337, do this:
for f in $(git grep -l stringid.GenerateNonCryptoID | grep -v '^vendor/'); do
sed -i 's/stringid.GenerateNonCryptoID/stringid.GenerateRandomID/g' $f;
done
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Also, do a general cleanup of all the timeout code. Changes
include:
- Convert from int to *uint where possible. Timeouts cannot be
negative, hence the uint change; and a timeout of 0 is valid,
so we need a new way to detect that the user set a timeout
(hence, pointer).
- Change name in the database to avoid conflicts between new data
type and old one. This will cause timeouts set with 4.2.0 to be
lost, but considering nobody is using the feature at present
(and the lack of validation means we could have invalid,
negative timeouts in the DB) this feels safe.
- Ensure volume plugin timeouts can only be used with volumes
created using a plugin. Timeouts on the local driver are
nonsensical.
- Remove the existing test, as it did not use a volume plugin.
Write a new test that does.
The actual plumbing of the containers.conf timeout in is one line
in volume_api.go; the remainder are the above-described cleanups.
Signed-off-by: Matthew Heon <mheon@redhat.com>
NFS Servers will thrown ENOTSUPP error if you attempt to
chown a directory to the same UID and GID as the directory
already has. If volumes are stored on NFS directories this
throws an ugly error and then works on the next try.
Bottom line don't chown directories that already have the correct
UID and GID.
Fixes: https://github.com/containers/podman/issues/14766
[NO NEW TESTS NEEDED] Difficult to setup an NFS Server in testing.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
* Correct spelling and typos.
* Improve language.
Co-authored-by: Ed Santiago <santiago@redhat.com>
Signed-off-by: Erik Sjölund <erik.sjolund@gmail.com>
We now use the golang error wrapping format specifier `%w` instead of
the deprecated github.com/pkg/errors package.
[NO NEW TESTS NEEDED]
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
PR containers/podman/pull/14449 had an outdated base. Merging it broke
builds.
[NO NEW TESTS NEEDED]
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
add two new options to the volume create command: copy and nocopy.
When nocopy is specified, the files from the container image are not
copied up to the volume.
Closes: https://github.com/containers/podman/issues/14722
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Libpod requires that all volumes are stored in the libpod db. Because
volume plugins can be created outside of podman, it will not show all
available plugins. This podman volume reload command allows users to
sync the libpod db with their external volume plugins. All new volumes
from the plugin are also created in the libpod db and when a volume from
the db no longer exists it will be removed if possible.
There are some problems:
- naming conflicts, in this case we only use the first volume we found.
This is not deterministic.
- race conditions, we have no control over the volume plugins. It is
possible that the volumes changed while we run this command.
Fixes#14207
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
add an option to configure the driver timeout when creating a volume.
The default is 5 seconds but this value is too small for some custom drivers.
Signed-off-by: cdoern <cdoern@redhat.com>
add a new option to completely disable xfs quota usage for a volume.
xfs quota set on a volume, even just for tracking disk usage, can
cause weird errors if the volume is later re-used by a container with
a different quota projid. More specifically, link(2) and rename(2)
might fail with EXDEV if the source file has a projid that is
different from the parent directory.
To prevent such kind of issues, the volume should be created
beforehand with `podman volume create -o o=noquota $ID`
Closes: https://github.com/containers/podman/issues/14049
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
This primarily served to protect us against shutting down the
Libpod runtime while operations (like creating a container) were
happening. However, it was very inconsistently implemented (a lot
of our longer-lived functions, like pulling images, just didn't
implement it at all...) and I'm not sure how much we really care
about this very-specific error case?
Removing it also removes a lot of potential deadlocks, which is
nice.
[NO NEW TESTS NEEDED]
Signed-off-by: Matthew Heon <mheon@redhat.com>
Fix handling of "bind" and "tmpfs" olumes to actually work.
Allow bind, tmpfs local volumes to work in rootless mode.
Also removed the string "error" from all error messages that begine with it.
All Podman commands are printed with Error:, so this causes an ugly
stutter.
Fixes: https://github.com/containers/podman/issues/12013
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Currently we are not passing the force flag down to the removal of
the running container. If the container is running, and we set
--force when removing the volume, the container should be stopped.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Add --time flag to podman container rm
Add --time flag to podman pod rm
Add --time flag to podman volume rm
Add --time flag to podman network rm
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Originally, Podman would unconditionally remove volumes from the
DB, even if they failed to be removed from the volume plugin;
this was a safety measure to ensure that `volume rm` can always
remove a volume from the database, even if the plugin is
misbehaving.
However, this is a significant deivation from Docker, which
refuses to remove if the plugin errors. These errors can be
legitimate configuration issues which the user should address
before the volume is removed, so Podman should also use this
behaviour.
Fixes#11214
Signed-off-by: Matthew Heon <mheon@redhat.com>
We missed bumping the go module, so let's do it now :)
* Automated go code with github.com/sirkon/go-imports-rename
* Manually via `vgrep podman/v2` the rest
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
This implements support for mounting and unmounting volumes
backed by volume plugins. Support for actually retrieving
plugins requires a pull request to land in containers.conf and
then that to be vendored, and as such is not yet ready. Given
this, this code is only compile tested. However, the code for
everything past retrieving the plugin has been written - there is
support for creating, removing, mounting, and unmounting volumes,
which should allow full functionality once the c/common PR is
merged.
A major change is the signature of the MountPoint function for
volumes, which now, by necessity, returns an error. Named volumes
managed by a plugin do not have a mountpoint we control; instead,
it is managed entirely by the plugin. As such, we need to cache
the path in the DB, and calls to retrieve it now need to access
the DB (and may fail as such).
Notably absent is support for SELinux relabelling and chowning
these volumes. Given that we don't manage the mountpoint for
these volumes, I am extremely reluctant to try and modify it - we
could easily break the plugin trying to chown or relabel it.
Also, we had no less than *5* separate implementations of
inspecting a volume floating around in pkg/infra/abi and
pkg/api/handlers/libpod. And none of them used volume.Inspect(),
the only correct way of inspecting volumes. Remove them all and
consolidate to using the correct way. Compat API is likely still
doing things the wrong way, but that is an issue for another day.
Fixes#4304
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
With the advent of Podman 2.0.0 we crossed the magical barrier of go
modules. While we were able to continue importing all packages inside
of the project, the project could not be vendored anymore from the
outside.
Move the go module to new major version and change all imports to
`github.com/containers/libpod/v2`. The renaming of the imports
was done via `gomove` [1].
[1] https://github.com/KSubedi/gomove
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
We can easily tell if we're going to deadlock by comparing lock
IDs before actually taking the lock. Add a few checks for this in
common places where deadlocks might occur.
This does not yet cover pod operations, where detection is more
difficult (and costly) due to the number of locks being involved
being higher than 2.
Also, add some error wrapping on the Podman side, so we can tell
people to use `system renumber` when it occurs.
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
If we attempt to label a volume and the file system
does not support labeling, then just warn. SELinux
may or may not work, on the volume.
There is no way to setup a private label on a newly
created volume without using the container mountlabel.
If we don't have a mount label at the time of creation of
the volume, the only option we have is to create a shared
label.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
`gocritic` is a powerful linter that helps in preventing certain kinds
of errors as well as enforcing a coding style.
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
When you try and create a new volume with the name of a volume
that already exists, you presently get a thoroughly unhelpful
error from `mkdir` as the volume attempts to create the
directory it will be mounted at. An EEXIST out of mkdir is not
particularly helpful to Podman users - it doesn't explain that
the name is already taken by another volume.
The solution here is potentially racy as the runtime is not
locked, so someone else could take the name while we're still
getting things set up, but that's a narrow timing window, and we
will still return an error - just an error that's not as good as
this one.
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
Also, ensure that we don't try to mount them without root - it
appears that it can somehow not error and report that mount was
successful when it clearly did not succeed, which can induce this
case.
We reuse the `--force` flag to indicate that a volume should be
removed even after unmount errors. It seems fairly natural to
expect that --force will remove a volume that is otherwise
presenting problems.
Finally, ignore EINVAL on unmount - if the mount point no longer
exists our job is done.
Fixes: #4247Fixes: #4248
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
When we fail to remove a container's SHM, that's an error, and we
need to report it as such. This may be part of our lingering
storage woes.
Also, remove MNT_DETACH. It may be another cause of the storage
removal failures.
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
When volume options and the local volume driver are specified,
the volume is intended to be mounted using the 'mount' command.
Supported options will be used to volume the volume before the
first container using it starts, and unmount the volume after the
last container using it dies.
This should work for any local filesystem, though at present I've
only tested with tmpfs and btrfs.
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
This will require a 'podman system renumber' after being applied
to get lock numbers for existing volumes.
Add the DB backend code for rewriting volume configs and use it
for updating lock numbers as part of 'system renumber'.
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
Begin to separate the internal structures and frontend for
inspect on volumes. We can't rely on keeping internal data
structures for external presentation - separating presentation
and internal data format is good practice.
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
the compilation demands of having libpod in main is a burden for the
remote client compilations. to combat this, we should move the use of
libpod structs, vars, constants, and functions into the adapter code
where it will only be compiled by the local client.
this should result in cleaner code organization and smaller binaries. it
should also help if we ever need to compile the remote client on
non-Linux operating systems natively (not cross-compiled).
Signed-off-by: baude <bbaude@redhat.com>
Instead of rewriting the logic, reuse the standard logic we use
for removing containers, which is much better tested.
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
The Commit test is blatantly wrong and testing buggy behavior. We
should be commiting the destination, if anything - and more
likely nothing at all.
When force-removing volumes, don't remove the volumes of
containers we need to remove. This can lead to a chicken and the
egg problem where the container removes the volume before we can.
When we re-add volume locks this could lead to deadlocks. I don't
really want to deal with this, and this doesn't seem a
particularly harmful quirk, so we'll let this slide until we get
a bug report.
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
This swaps the previous handling (parse all volume mounts on the
container and look for ones that might refer to named volumes)
for the new, explicit named volume lists stored per-container.
It also deprecates force-removing volumes that are in use. I
don't know how we want to handle this yet, but leaving containers
that depend on a volume that no longer exists is definitely not
correct.
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
Before, there were SELinux denials when a volume was bind-mounted by podman play kube.
Partially fix this by setting the default private label for mounts created by play kube (with DirectoryOrCreate)
For volumes mounted as Directory, the user will have to set their own SELinux permissions on the mount point
also remove left over debugging print statement
Signed-off-by: Peter Hunt <pehunt@redhat.com>
In lipod, we now log major events that occurr. These events
can be displayed using the `podman events` command. Each
event contains:
* Type (container, image, volume, pod...)
* Status (create, rm, stop, kill, ....)
* Timestamp in RFC3339Nano format
* Name (if applicable)
* Image (if applicable)
The format of the event and the varlink endpoint are to not
be considered stable until cockpit has done its enablement.
Signed-off-by: baude <bbaude@redhat.com>
I was looking into why we have locks in volumes, and I'm fairly
convinced they're unnecessary.
We don't have a state whose accesses we need to guard with locks
and syncs. The only real purpose for the lock was to prevent
concurrent removal of the same volume.
Looking at the code, concurrent removal ought to be fine with a
bit of reordering - one or the other might fail, but we will
successfully evict the volume from the state.
Also, remove the 'prune' bool from RemoveVolume. None of our
other API functions accept it, and it only served to toggle off
more verbose error messages.
Signed-off-by: Matthew Heon <matthew.heon@pm.me>
