System tests: tighten 'is' operator

Fix day-one sloppiness: when I first wrote this framework
it compared strings using 'expr', not '=', to be more
forgiving of extra cruft in output. This was a bad decision.
It means that warnings or additional text are ignored:

    is "all is ok, NOT!"  "all is ok"  <-- this would pass

Solution: tighten up the 'is' check. Use '=' (direct
compare) first. If it fails, look for wild cards ('*')
or character classes ('[') in the expect string. If
so, and only then, use 'expr'. And, thanks to a clever
suggestion from Luap99, include '(using expr)' in the
error message when we do so; this could make it easier
for a developer to understand a string mismatch.

This change exposes a lot of instances in which we weren't
doing proper comparisons. Fix those. Thankfully, there
weren't as many as I'd feared.

Also, and completely unrelated, add '-T' flag to bats
helper, for showing timing results. (I will open this
as a separate PR if requested. I too find it offensive
to jumble together unrelated commits.)

Signed-off-by: Ed Santiago <santiago@redhat.com>
This commit is contained in:
Ed Santiago
2021-09-28 07:13:51 -06:00
parent ba72b17d28
commit bf94ebf423
16 changed files with 75 additions and 40 deletions

View File

@ -19,6 +19,8 @@ $0 is a wrapper for invoking podman system tests.
version of bats installed, runs with '--filter pattern'
which runs only subtests that match 'pattern'
-T Passed on to bats, which will then show timing data
--help display usage message
By default, tests ./bin/podman. To test a different podman, do:
@ -60,6 +62,8 @@ REMOTE=
ROOT_ONLY=
ROOTLESS_ONLY=
declare -a bats_opts=()
declare -a bats_filter=()
for i;do
@ -69,6 +73,7 @@ for i;do
--root) ROOT_ONLY=1 ;;
--rootless) ROOTLESS_ONLY=1 ;;
--remote) REMOTE=remote; echo "--remote is TBI"; exit 1;;
--ts|-T) bats_opts+=("-T") ;;
*/*.bats) TESTS=$i ;;
*)
if [[ $i =~ : ]]; then
@ -94,7 +99,7 @@ if [ -z "$ROOTLESS_ONLY" ]; then
sudo --preserve-env=PODMAN \
--preserve-env=PODMAN_TEST_DEBUG \
--preserve-env=OCI_RUNTIME \
bats "${bats_filter[@]}" $TESTS
bats "${bats_opts[@]}" "${bats_filter[@]}" $TESTS
rc=$?
fi
@ -102,7 +107,7 @@ fi
echo "--------------------------------------------------"
if [ -z "$ROOT_ONLY" ]; then
echo "\$ bats ${bats_filter[@]} $TESTS"
bats "${bats_filter[@]}" $TESTS
bats "${bats_opts[@]}" "${bats_filter[@]}" $TESTS
rc=$((rc | $?))
fi