When I originally wrote this code I had no idea what POST
would look like so I did a sloppy job, deferring making it
usable. Now that we have some real-world examples in place,
I have a better understanding of what params look like and
how to make tests more readable/maintainable. (Deferring isn't
always bad: one of my early ideas was to separate params using
commas; that would've been a disaster because some JSON values,
such as arrays, include commas).
This commit implements a better way of dealing with POST:
* The main concept is still 'key=value'
* When value is a JSON object (dictionary, array), it
can be quoted.
* Multiple params are simply separated by spaces.
The 3-digit HTTP code is a prominent, readable separator
between POST params and expected results. The parsing
code is a little uglier, but test developers need
never see that. The important thing is that writing
tests is now easier.
* POST params can be empty (this removes the need for a
useless '')
I snuck in one unrelated change: one of the newly-added
tests, .NetworkSettings, was failing when run rootless
(which is how I test on my setup). I made it conditional.
Signed-off-by: Ed Santiago <santiago@redhat.com>
...and a rudimentary set of /auth tests for PR#9589 (disabled).
This simply adds a new start_registry() helper function that
allocates a random unused port, pulls a registry image, creates
a local certificate + random username + random password, and
fires everything up. Since none of this is (yet) used in CI,
this is very low risk.
The only infinitessimally-risky change is using a dedicated
subdirectory of $WORKDIR (instead of $WORKDIR itself) as
the podman root. This fixes a dumb oversight on my part:
the workdir has grown to be used for much more than just
podman root; this change removes clutter and makes it
easier for humans to debug in cases of problems.
Signed-off-by: Ed Santiago <santiago@redhat.com>
While I wasn't looking, some completely unreadable cruft
crept in here, and it's totally my fault: I never knew
you could pass JSON to a GET query. Everyone who DID
know that, did so, but had to URL-escape it into a
completely gobbledygook mess to make curl happy.
Solution: trivial, do the URL-escaping in 't' itself. I
just never realized that was needed.
I'm so sorry. I hope this helps.
Signed-off-by: Ed Santiago <santiago@redhat.com>
It's been a while since I last looked at these; some cruft
has crept in, generating noise and hence unreadable test
results. Clean it up:
* remove pushd/popd in one subtest, replace with 'tar -C'.
(Also remove confusing quotation marks). This removes
spurious directory names from output.
* in like(), show only first line of actual output.
Some commands ('tree', 'generate kube') produce
voluminous multi-line output, which is super useless
and distracting when reading a test run.
* Recognize that some queries will not generate output,
e.g. HEAD requests and some POSTs. Deal with that.
This fixes "curl.result.out: no such file" and "parse
error" warnings.
* In cleanup, 'podman rm -a' and 'rmi -af'; this gets
rid of errors when deleting $WORKDIR. (EBUSY error
when root, EPERM when rootless).
And, the original reason for poking in here: refactor the
wait-for-port part of start_server() into its own helper
function, so we can use it when starting a local registry
in 12-imagesMore. (Ref: #9270)
Signed-off-by: Ed Santiago <santiago@redhat.com>
- apiv2 - the 'ten /info requests' test is flaking often,
taking ~8 seconds (our limit is 7, up from 5 a few weeks
ago). Brent suggested that the first /info call might be
expensive, because it needs to access storage. So, let's
prime it by running one /info outside the timing loop.
And, because even that continues to fail, bump it up
to 10 seconds and file #8076 to track the slowdown.
- toolbox test - WaitForReady() has timed out, even on one
occasion causing a run failure because it failed 3 times.
Solution: bump up timeout from 2s to 5s. Not really great,
but CI systems are underpowered, and it's not unreasonable
that 2s might be too low.
- sdnotify test - add a 'podman wait' between stop & rm.
This may prevent a "cannot rm container as it is running"
race condition.
While working on this, Brent and I noticed a few ways that
test-apiv2 logging can be improved:
- test name: when request is POST, display the jsonified
parameters, not the original input ones. This should
make it much easier to reproduce failures.
- use curl's "--write-out" option to capture http code,
content type, and request time. We were getting the
first two via grep from logged headers; this is cleaner.
And there was no other way to get timing. We now include
the timing as X-Response-Time in the log file.
- abort on *any* curl error, not just 7 (cannot connect).
Any error at all from curl is bad news.
Signed-off-by: Ed Santiago <santiago@redhat.com>
CI discovered that a lot of networking tests are failing; my
fault, for not having run my tests as root on my laptop.
Disable those.
Also: bump up the ten-request time limit, from 5 to 7 seconds.
Looks like something keeps getting slower and slower, but I
guess there's not much we can do about it.
Also: when we get a mismatch response code (e.g. 500 when we
expect 200), dump the response body and skip any subsequent
response checks.
Signed-off-by: Ed Santiago <santiago@redhat.com>
In the new-Cirrus transition, APIv2 tests were inadvertently
disabled. As expected when tests get disabled, they break.
This commit fixes some failing tests, and comments out others
(with big FIXMEs) because I have neither the expertise nor
time to figure out the real problems.
The big change to test-apiv2 is due to a recently-added
test that looks for an '=' sign in json output. My '=' vs '~'
detector completely barfed on that, and there's just no
way to make it work in a bash 'case' statement. So, switch
to an 'if' with 'expr'.
And, unrelated, fix a longstanding (harmless) bug that was
issuing spurious "expected" messages to the test log; those
should've been going to the full results log.
Signed-off-by: Ed Santiago <santiago@redhat.com>
It's not possible to run any of the scripts on distributions which do
have `bash` not in `/bin`. This is being fixed by using `/usr/bin/env
bash` instead.
Signed-off-by: Sascha Grunert <sgrunert@suse.com>
test-apiv2 has two basic comparisons of returned JSON:
equality and likeness ('=' and '~'). When logging failures,
the test runner shows both actual and expected values. When
logging success, for '=' there's no need to show both actual
and expected. But for '~', it can be helpful (for verifying
test correctness) to show the actual returned value.
To be specific:
old: ok ... .MemTotal~[0-9]\+
new: ok ... .MemTotal ('33509068800') ~ [0-9]\+
old: ok ... .[0].State~\(exited\|stopped\)
new: ok ... .[0].State ('exited') ~ \(exited\|stopped\)
The main benefit is that a developer or end user can
easily see precisely what was returned; this can help
confirm that the test is working as intended, and/or
help fine-tune how the test is written.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Add testing for displaying image history and exporting image
Deal with API returning binary (Content-Type =~ 'octet').
When so, set $output to the output of 'file'.
Bug fix: in 't' helper, declare loop var $i as local
to avoid contaminating caller
Signed-off-by: Ed Santiago <santiago@redhat.com>
Signed-off-by: Alex Jia <chuanchang.jia@gmail.com>
apiv2 tests emit TAP-compliant output; recognize it and
highlight it the same way we do BATS tests.
Add anchor links to TAP output, so other tools (e.g.
cirrus-flake-summarize) can link to particular lines
And, remove a "-f" from "wait" in test-apiv2; looks
like there's some version of bash used in some CI VM
that doesn't grok it.
Signed-off-by: Ed Santiago <santiago@redhat.com>
* Allow for descriptive comment in 't' invocations, making it
easier to distinguish similar requests
* Include test file basename (eg 40-pods) in 'ok/not ok' line
* Always symlink $TMPDIR/test-apiv2.log to latest YYMMDDetc file
* Include test result ('ok', 'not ok') in said log
* When curl results are JSON, filter them through jq into log
Signed-off-by: Ed Santiago <santiago@redhat.com>
API v2 has been quiet for a few days, and the test script is
actually passing. Let's take advantage of this opportunity
to get them running in CI.
Requires adding a check for cgroupsv2
Signed-off-by: Ed Santiago <santiago@redhat.com>
Looks like /libpod/pods/create has been fixed to return an
actual pod ID. Extend those tests.
Also, update timeout in the server command: it's now seconds,
not milliseconds.
Also, update FIXME comments in /pods/prune . Still doesn't
work, but clarify what we're seeing.
Also, add a new test that runs ten /info requests and
barfs if it takes more than 5 seconds.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Lots has changed since I first checked this in:
* Switch to new podman system service invocation
* /containers API has changed drastically
* /pods API has some fixes; check for them (e.g.
container-exists is now 409 Conflict, not 500)
* One test ('?invalidparam=x') still doesn't work;
comment it out so we can get everything passing.
Also, some work on the test framework itself:
* Cleaner port-open testing (the bash /dev/tcp check).
* Add a 'podman' function to invoke local podman and
log its output.
The above two allow us to:
* Get rid of stderr special-casing
Furthermore:
* t() no longer needs leading '.'; this allows jq
features such as 'length' and perhaps other filters
* special-case handling of 204 and 304: rfc2616 demands
that they return no message body; assert that it is so.
* new root & rootless helper functions (check server)
* remove the "unlikely to work" message for rootless;
it seems to be working fine
* fix pod tests for rootless
* BUT: add a bolder FIXME because the ID field seems wrong
Signed-off-by: Ed Santiago <santiago@redhat.com>
Initial framework for testing the version 2 (HTTP) API.
Includes a collection of tests for some of the existing
endpoints. Not all tests are currently passing.
Signed-off-by: Ed Santiago <santiago@redhat.com>
