Files
podman/hack/bats
Paul Holzinger 81c90f51c2 test/system: speed up basic_{setup,teardown}()
While these are not really slow they still take about 100-250ms if I
time this locally. Given they are run for every test this adds up
quickly. Looking at CI logs I can see the timings for skipped
tests are all in 600ms range. So I think it is safe to assume that these
functions need to get faster.

We have over 670 test cases currently so we talk about over 400s spend
in these functions in CI. This allows for big gains.

Now overall this is a tricky trade of, while all tests should cleanup
after themselves there is no guarantee for that as such errors can be
leaked into other tests making debugging much harder. To work at least a
bit against this teardown checks if the test was successful and only
skips the podman commands bases on that. Without it a single flake could
cause all following tets to fail.

As such this commit does the proper setup once one suite start then only
after a test failed.

In order for this to work at all we have to fix all leaks first, see
previous commits. And then for the future keep a very strong eye on
this during reviews.

Also add a PODMAN_BATS_LEAK_CHECK option

By default test must cleanup themselves and to speed up CI we no longer
do any cleanup in teardown by default. However there is still many cases
where we might have to debug a leak so add a new PODMAN_BATS_LEAK_CHECK
env option that can be set and should cause teardown to fail if the test
did not cleanup properly.

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
2024-06-18 11:06:50 +02:00

153 lines
4.7 KiB
Bash
Executable File

#!/bin/bash
#
# bats wrapper - invokes bats, root & rootless, on podman system tests
#
###############################################################################
# BEGIN usage message
usage="Usage: $0 [--root] [--rootless] [FILENAME-PATTERN[:TEST-PATTERN]]
$0 is a wrapper for invoking podman system tests.
--root Run only as root
--rootless Run only as user (i.e. you)
--remote Run with podman-remote (see below)
FILENAME-PATTERN Run only test files that match 'test/system/*name*',
e.g. '500' or 'net' will match 500-networking.bats.
TEST-PATTERN When appended to a filename-pattern, and you have a
modern-enough version of bats installed (i.e. Fedora
but not RHEL), runs with '--filter TEST-PATTERN' which
runs only subtests within FILENAME-PATTERH whose names
match that string.
--tag=TAG Passed on to bats as '--filter-tags TAG'
As of 2023-07-26 the only tag used is 'distro-integration'
-T Passed on to bats, which will then show timing data
--help display usage message
By default, tests ./bin/podman. To test a different podman, do:
\$ env PODMAN=/abs/path/to/podman $0 ....
To test podman-remote, start your own servers (root and rootless) via:
\$ bin/podman system service --timeout=0 &
\$ sudo !!
...then invoke this script with --remote. (This script can't start the
servers, because we can sudo *starting* the service but can't sudo
stopping it: by the time the bats tests finish, the sudo timeout will
have expired. We apologize for the inconvenience.)
$0 also passes through \$OCI_RUNTIME, should you need to test that.
Examples:
\$ $0 220:\"restart cleans up\"
... only the \"restart cleans up\" test in 220-healthcheck.bats
\$ $0 --root 160:\"ps -f\"
... runs all tests in 160-volumes.bats that match \"ps -f\" (root only)
"
# END usage message
###############################################################################
# BEGIN initialization and command-line arg checking
# By default, test the podman in our working directory.
# Some tests cd out of our workdir, so abs path is important
export PODMAN=${PODMAN:-$(pwd)/bin/podman}
export QUADLET=${QUADLET:-$(pwd)/bin/quadlet}
# Directory in which
TESTS=test/system
REMOTE=
TEST_ROOT=1
TEST_ROOTLESS=1
declare -a bats_opts=()
declare -a bats_filter=()
for i;do
value=`expr "$i" : '[^=]*=\(.*\)'`
case "$i" in
-h|--help) echo "$usage"; exit 0;;
--root) TEST_ROOTLESS= ;;
--rootless) TEST_ROOT= ;;
--remote) REMOTE=remote ;;
--ts|-T) bats_opts+=("-T") ;;
--tag=*) bats_filter=("--filter-tags" "$value") ;;
*/*.bats) TESTS=$i ;;
*)
if [[ $i =~ : ]]; then
tname=${i%:*} # network:localhost -> network
filt=${i#*:} # network:localhost -> localhost
TESTS=$(echo $TESTS/*$tname*.bats)
bats_filter=("--filter" "$filt")
else
TESTS=$(echo $TESTS/*$i*.bats)
fi
;;
esac
done
# With --remote, use correct binary and make sure daem--I mean server--is live
if [[ "$REMOTE" ]]; then
if ! [[ $PODMAN =~ -remote ]]; then
PODMAN=${PODMAN}-remote
fi
if [[ -n "$TEST_ROOT" ]]; then
sudo $PODMAN info >/dev/null || exit 1
fi
if [[ -n "$TEST_ROOTLESS" ]]; then
$PODMAN info >/dev/null || exit 1
fi
fi
# END initialization and command-line arg checking
###############################################################################
rc=0
# As of 2021-11 podman has a bunch of external helper binaries
if [[ -z "$CONTAINERS_HELPER_BINARY_DIR" ]]; then
export CONTAINERS_HELPER_BINARY_DIR=$(pwd)/bin
fi
# Used in 120-load test to identify rootless destination for podman image scp
export PODMAN_ROOTLESS_USER=$(id -un)
# Make sure to always check for leaks when running locally
export PODMAN_BATS_LEAK_CHECK=1
# Root
if [[ "$TEST_ROOT" ]]; then
echo "# bats ${bats_filter[*]} $TESTS"
sudo --preserve-env=PODMAN \
--preserve-env=QUADLET \
--preserve-env=PODMAN_TEST_DEBUG \
--preserve-env=OCI_RUNTIME \
--preserve-env=CONTAINERS_HELPER_BINARY_DIR \
--preserve-env=PODMAN_ROOTLESS_USER \
bats "${bats_opts[@]}" "${bats_filter[@]}" $TESTS
rc=$?
fi
# Rootless. (Only if we're not already root)
if [[ "$TEST_ROOTLESS" && "$(id -u)" != 0 ]]; then
echo "--------------------------------------------------"
echo "\$ bats ${bats_filter[*]} $TESTS"
bats "${bats_opts[@]}" "${bats_filter[@]}" $TESTS
rc=$((rc | $?))
fi
exit $rc