For the past two months we've been splitting system tests
into two categories: those that CAN be run in parallel,
and those that CANNOT. Much work has been done to replace
hardcoded names (mycontainer, mypod) with safename().
Hundreds of test runs, in CI and on Ed's laptop, have
proven this approach viable.
make {local,remote}system now runs in two steps: first
the serial ones, then the parallel ones. hack/bats will
now recognize the 'ci:parallel' tag and add --jobs (nprocs).
This requires some tweaking of leak_check, because there
can be umpteen tests running (affecting image/container/pod/etc
state) when any given test completes.
Rules for enabling parallelization in tests:
* use unique container/pod/volume/network names (safename)
* do not run 'podman rm -a' or 'rmi -a'
* never use the -l (--latest) option
* do not run 'podman ps/images' and expect precise output
Signed-off-by: Ed Santiago <santiago@redhat.com>
While these are not really slow they still take about 100-250ms if I
time this locally. Given they are run for every test this adds up
quickly. Looking at CI logs I can see the timings for skipped
tests are all in 600ms range. So I think it is safe to assume that these
functions need to get faster.
We have over 670 test cases currently so we talk about over 400s spend
in these functions in CI. This allows for big gains.
Now overall this is a tricky trade of, while all tests should cleanup
after themselves there is no guarantee for that as such errors can be
leaked into other tests making debugging much harder. To work at least a
bit against this teardown checks if the test was successful and only
skips the podman commands bases on that. Without it a single flake could
cause all following tets to fail.
As such this commit does the proper setup once one suite start then only
after a test failed.
In order for this to work at all we have to fix all leaks first, see
previous commits. And then for the future keep a very strong eye on
this during reviews.
Also add a PODMAN_BATS_LEAK_CHECK option
By default test must cleanup themselves and to speed up CI we no longer
do any cleanup in teardown by default. However there is still many cases
where we might have to debug a leak so add a new PODMAN_BATS_LEAK_CHECK
env option that can be set and should cause teardown to fail if the test
did not cleanup properly.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
pasta only works when we run as container_runtime_exec_t, now that pasta
is the default this means that the current binary will not work when
doing local dev without manually fixing the label.
There are also other parts where the correct label is important. So as a
simple fix always set the proper label in the bin/podman target.
This also means we can drop this line from the hack/bats script.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
BATS 1.8.0 introduces tags: metadata that can be applied to
a single test or one entire file, then used for filtering
in a test run.
Issue #19299 introduces the possibility of using OpenQA
for podman reverse dependency testing: continuous CI on
all packages that can affect podman, so we don't go two
months with no bodhi builds then get caught by surprise
when systemd or kernel or crun change in ways that break us.
This PR introduces one bats tag, "distro-integration".
The intention is for OpenQA (or other) tests to install
the podman-tests package and run:
bats --filter-tags distro-integration /usr/share/podman/test/system
Goal is to keep the test list short and sweet: we do not
need to test command-line option parsing. We *DO* need to
test interactions with systemd, kernel, nethack, and other
critical components.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Even though we still rely on the user to start the podman
system service, enable the option anyway. It's much, much
friendlier than requiring 'env PODMAN=etc-etc'.
Signed-off-by: Ed Santiago <santiago@redhat.com>
The ROOT{,LESS}_ONLY logic is confusing and unmaintainable.
Change it to an easier-to-read positive check.
Signed-off-by: Ed Santiago <santiago@redhat.com>
This adds basic container and volume system tests for quadlet. These
install and run actual systemd units and ensure they work.
Signed-off-by: Alexander Larsson <alexl@redhat.com>
Followup to #15616, which is not usable as it is (way, way, way
too much noise) but actually found a few real nits that should
be fixed.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Start inching our way back to having tests for the sudo form
of podman image scp. Basically, copy an image to another user
and then back, using a pseudorandom name. Confirm that the
image makes it to the remote end, and that when we copy it
back, the original image digest is preserved.
When scp'ing as root, we identify the destination rootless
user account via the $PODMAN_ROOTLESS_USER envariable. Setting
this and creating the account is left as an exercise for the
CI framework (be it github, or Fedora/CentOS/RHEL gating, or
other).
Also: amend hack/bats to set and relay $PODMAN_ROOTLESS_USER,
so developers can test locally.
Also: remove what I'm 99% sure is a debugging printf.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Some time in the last month, podman started to depend on a bunch
of external helper binaries: rootlessport, pause, catatonit.
System tests fail without these.
Update the hack/bats script to pass $CONTAINERS_HELPER_BINARIES_DIR
(set to ./bin); podman will then use locally-built helpers. (This
requires https://github.com/containers/common/pull/823 , which as
of this PR is not yet vendored into podman. There is no harm in
merging this while we wait.)
Also: if bats helper is invoked as root, run only once; i.e.,
skip the "rootless" step.
Also (piggybacked): the name of the podman pause image has
changed, from pause to podman-pause. Adjust that in our
teardown so we don't leave droppings.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Fix day-one sloppiness: when I first wrote this framework
it compared strings using 'expr', not '=', to be more
forgiving of extra cruft in output. This was a bad decision.
It means that warnings or additional text are ignored:
is "all is ok, NOT!" "all is ok" <-- this would pass
Solution: tighten up the 'is' check. Use '=' (direct
compare) first. If it fails, look for wild cards ('*')
or character classes ('[') in the expect string. If
so, and only then, use 'expr'. And, thanks to a clever
suggestion from Luap99, include '(using expr)' in the
error message when we do so; this could make it easier
for a developer to understand a string mismatch.
This change exposes a lot of instances in which we weren't
doing proper comparisons. Fix those. Thankfully, there
weren't as many as I'd feared.
Also, and completely unrelated, add '-T' flag to bats
helper, for showing timing results. (I will open this
as a separate PR if requested. I too find it offensive
to jumble together unrelated commits.)
Signed-off-by: Ed Santiago <santiago@redhat.com>
Well, new to you. It's been something I've used for years.
Simple, but it takes care of a lot of housekeeping, and
makes it ever-so-much-more pleasant to invoke bats tests.
And when it's easier to run tests, tests get run.
Signed-off-by: Ed Santiago <santiago@redhat.com>
