bd0582dd34
Various small fixes to get BATS tests working again. Split from #2947 because that one keeps getting stalled, and I'm hoping these separate changes get approved. I consider these changes urgent because RHEL8 gating tests are failing, and will fail even more if/when #2272 gets picked up and packaged for RHEL8, and I consider it important to have clean passing tests for RHEL8. * info test: 'insecure registries' is gone. A recent commit (d1a7378aa) changed the format of 'podman info', removing the 'insecure registries' key. Deal with it. * info test: remove check for .host.{Conmon,OCIRuntime}.package; the value on f28 and f29 is 'Unknown' (instead of an NVR). We can live without this check. * 'load' test: skip when running in CI, because stdin is not a tty. * container restore: fix arg processing. #2272 broke argument processing: 'podman container restore', with no args, should exit with 'argument required' error. Root cause is that the new --import option takes the place of an argument, so the checkAllAndLatest() call had to be changed to not exit on error. Workaround is (sigh) to copy/paste the skipped checkAllAndLatest() code, with minor tweaks to accommodate --import. Signed-off-by: Ed Santiago <santiago@redhat.com>
Quick overview of podman system tests. The idea is to use BATS, but with a framework for making it easy to add new tests and to debug failures.
Quick Start
Look at 030-run.bats for a simple but packed example. This introduces the basic set of helper functions:
-
setup(implicit) - resets container storage so there's one and only one (standard) image, and no running containers. -
parse_table- you can define tables of inputs and expected results, then read those in awhileloop. This makes it easy to add new tests. Because bash is not a programming language, the caller ofparse_tablesometimes needs to massage the returned values;015-run.batsoffers examples of how to deal with the more typical such issues. -
run_podman- runs command defined in$PODMAN(default: 'podman' but could also be './bin/podman' or 'podman-remote'), with a timeout. Checks its exit status. -
is- compare actual vs expected output. Emits a useful diagnostic on failure. -
die- output a properly-formatted message to stderr, and fail test -
skip_if_rootless- if rootless, skip this test with a helpful message. -
random_string- returns a pseudorandom alphanumeric string
Test files are of the form NNN-name.bats where NNN is a three-digit
number. Please preserve this convention, it simplifies viewing the
directory and understanding test order. In particular, 00x tests
should be reserved for a first-pass fail-fast subset of tests:
bats test/system/00*.bats || exit 1
bats test/system
...the goal being to provide quick feedback on catastrophic failures without having to wait for the entire test suite.
Analyzing test failures
The top priority for this scheme is to make it easy to diagnose
what went wrong. To that end, podman_run always logs all invoked
commands, their output and exit codes. In a normal run you will never
see this, but BATS will display it on failure. The goal here is to
give you everything you need to diagnose without having to rerun tests.
The is comparison function is designed to emit useful diagnostics,
in particular, the actual and expected strings. Please do not use
the horrible BATS standard of [ x = y ]; that's nearly useless
for tracking down failures.
If the above are not enough to help you track down a failure:
Debugging tests
Some functions have dprint statements. To see the output of these,
set PODMAN_TEST_DEBUG="funcname" where funcname is the name of
the function or perhaps just a substring.
Requirements
The jq tool is needed for parsing JSON output.
Further Details
TBD. For now, look in helpers.bash; each helper function
has (what are intended to be) helpful header comments. For even more
examples, see and/or run helpers.t; that's a regression test
and provides a thorough set of examples of how the helpers work.