Migrates `test` to the new package looping base command.
Refactors the bulk of the logic to helpers for easier understanding of the flow.
Part of flutter/flutter#83413
Most of the tool operates on packages in general, and the targetting
done currently by the `--plugins` flag is not actually restricted to
plugins, so this makes the name less confusing.
Part of https://github.com/flutter/flutter/issues/83413
Many commands had insufficient failure testing. This adds new tests that
ensure that for every Process call, at least one test fails if a failure
from that process were ignored (with the exception of calls in the
`publish` command, which has a custom process mocking system, so was out
of scope here; it already has more coverage than most tests did though.)
For a few existing failure tests, adds output checks to ensure that they
are testing for the *right* failures.
Other changes:
- Adds convenience constructors to MockProcess for the common cases of a
mock process that just exits with a 0 or 1 status, to reduce test
verbosity.
- Fixes a few bugs that were found by the new tests.
- Minor test cleanup, especially cases where a mock process was being
set up just to make all calls succeed, which is the default as of
recent changes.
To support this command's --machine flag, which moves all normal output into a field in a JSON struct, adds a way of capturing output and providing it to the command subclass on completion.
Part of flutter/flutter#83413
The intent of package-looping commands is that we always want to run them for all the targeted packages, accumulating failures, and then report them all at the end, so we don't end up with the problem of only finding errors one at a time. However, some of them were using `exitOnError: true` for the underlying commands, causing the first error to be fatal to the test.
This PR:
- Removes all use of `exitOnError: true` from the package-looping commands. (It's still used in `format`, but fixing that is a larger issue, and less important due to the way `format` is currently structured.)
- Fixes the mock process runner used in our tests to correctly simulate `exitOrError: true`; the fact that it didn't was hiding this problem in test (e.g., `drive-examples` had a test that asserted that all failures were summarized correctly, which passed because in tests it was behaving as if `exitOnError` were false)
- Adjusts the mock process runner to allow setting a list of mock result for a specific executable instead of one result for all calls to anything, in order to fix some tests that were broken by the two changes above and unfixable without this (e.g., a test of a command that had been calling one executable with `exitOnError: true` then another with ``exitOnError: false`, where the test was asserting things about the second call failing, which only worked because the first call's failure wasn't actually checked). To limit the scope of the PR, the old method of setting a single result for all calls is still supported for now as a fallback.
- Fixes the fact that the mock `run` and `runAndStream` had opposite default exit code behavior when no mock process was set (since that caused me a lot of confusion while fixing the above until I figured it out).
- Updates the remaining tests (other than one that still needs to be
converted to the new base command, which will be fixed then) that
aren't using runCapturingPrint to do so to reduce test log spam.
- Simplifies and standardizes the matcher used for ToolExit in tests.
Add a summary to the end of successful runs for everything using the new looping base command, similar to what we do for summarizing failures. This will make it easy to manually check results for PRs that we know should be changing the set of run packages (adding a new package, adding a new test type to a package, adding a new test type to the tool), as well as spot-checking when we see unexpected results (e.g., looking back and why a PR didn't fail CI when we discover that it should have).
To support better surfacing skips, this restructures the return value of `runForPackage` to have "skip" as one of the options. As a result of it being a return value, packages that used `printSkip` to indicate that *parts* of the command were being skipped have been changed to no longer do that.
Fixes https://github.com/flutter/flutter/issues/85626
Significantly restructures drive-examples:
- Migrates it to the new package-looping base command
- Enforces that only one platform is passed, since in practice multiple platforms never actually worked. (The logic is structured so that it will be easy to enable multi-platform if `flutter drive` gains multi-platform support.)
- Fixes the issue where `--ios` and `--android` were semi-broken, by doing explicit device targeting for them rather than relying on the default device being the right kind
- Extracts much of the logic to helpers so it's easier to understand the flow
- Removes support for a legacy integration test file structure that is no longer used
- Adds more test coverage; previously no failure cases were actually tested.
Fixes https://github.com/flutter/flutter/issues/85147
Part of https://github.com/flutter/flutter/issues/83413
Migrates firebase-test-lab to use the new package-looping base command.
Other changes:
- Extracts several helpers to make the main flow easier to follow
- Removes support for finding and running `*_e2e.dart` files, since we no longer use that file structure for integration tests.
Part of https://github.com/flutter/flutter/issues/83413
Switches build-examples to the new base command that handles the boilerplate of looping over target packages.
While modifying the command, also does some minor cleanup:
- Extracts a helper to reduce duplicated details of calling `flutter build`
- Switches the flag for iOS to `--ios` rather than `--ipa` since `ios` is what is actually passed to the build command
- iOS no longer defaults to on, so that it behaves like all the other platform flags
- Passing no platform flags is now an error rather than a silent pass, to ensure that we never accidentally have CI doing a no-op run without noticing.
- Rewords the logging slightly for the versions where the label for what is being built is a platform, not an artifact (which is now everything but Android).
Part of flutter/flutter#83413
Combines the two different aspects of version-checking into a single looping structure based on plugins, using the new base command, rather than one operating on plugins as controlled by the usual flags and the other operating on a git list of changed files.
Also simplifies the determination of the new version by simply checking the file, rather than querying git for the HEAD state of the file. Tests setup is simplified since we no longer need to set up nearly as much fake `git` output.
Minor changes to base commands:
- Move indentation up to PackageLoopingCommand so that it is consistent across commands
- Add a new post-loop command for any cleanup, which is needed by version-check
- Change the way the GitDir instance is managed by the base PluginCommand, so that it's always cached even when not overridden, to reduce duplicate work and code.
Part of https://github.com/flutter/flutter/issues/83413
Switches `java-test` to the new base command that handles the boilerplate of looping over target packages.
Includes test improvements:
- Adds failure tests; previously no failure cases were covered.
- Captures output so test output isn't spammed with command output.
Part of flutter/flutter#83413
Switches `podspec` to the new base command that handles the boilerplate of looping over target packages.
Includes test improvements:
- Adds failure tests; previously no failure cases were covered.
- Captures output using the standard mechanism instead of using a custom
`_print`, simplifying the command code.
Also removes `--skip`, which is redundant with `--exclude`.
Part of flutter/flutter#83413
Switches `analyze` to the new base command that handles the boilerplate
of looping over target packages. This will change the output format
slightly, but shoudn't have any functional change.
Updates tests to use runCapturingPrint so that test run output isn't
mixed with command output.
Part of https://github.com/flutter/flutter/issues/83413
This PR removes a TODO where we used to check if a release should happen against git tags, instead, we check against pub.
Also, when auto-publish and the package is manually released, the CI will continue to fail without this change.
Fixesflutter/flutter#81047
Most of our commands are generally of the form:
```
for (each plugin as defined by the tool flags)
check some things for success or failure
print a summary all of the failing things
exit non-zero if anything failed
```
Currently all that logic not consistent, having been at various points copied and pasted around, modified, in some cases rewritten. There's unnecessary boilerplate in each new command, and there's unnecessary variation that makes it harder both to maintain the tool, and to consume the test output:
- There's no standard format for separating each plugin's run to search within a log
- There's no standard format for the summary at the end
- In some cases commands have been written to ToolExit on failure, which means we don't actually get the rest of the runs
This makes a new base class for commands that follow this structure to use, with shared code for all the common bits. This makes it harder to accidentally write new commands incorrectly, easier to maintain the code, and lets us standardize output so that searching within large logs will be easier.
This ports two commands over as a proof of concept to demonstrate that it works; more will be converted in follow-ups.
Related to https://github.com/flutter/flutter/issues/83413
Rather than taking a list of list of path elements, just accept a list
of Posix-style paths. In practice, the API was already being partially
used that way.
common.dart is a large-and-growing file containing all shared code,
which makes it hard to navigate. To make maintenance easier, this splits
the file (and its test file) into separate files for each major
component or category.
Migrates publish-check and version-check commands to NNBD.
Reworks the version-check flow so that it's more consistent with the other commands: instead of immediately exiting on failure, it checks all plugins, gathers failures, and summarizes all failures at the end. This ensures that we don't have failures in one package temporarily masked by failures in another, so PRs don't need to go through as many check cycles.
Part of https://github.com/flutter/flutter/issues/81912
- Adds macOS support to the `xctest` tool command
- Adds logic to the tool to check for packages that delegate their implementations
to another package, so they can be skipped when running native unit tests
- Updates the tool's unit test utility for writing pubspecs to be able to make
delegated federated implementation references to test it
- Adds initial unit tests to the non-deprecated macOS plugins
- Enables macOS XCTesting in CI
macOS portion of https://github.com/flutter/flutter/issues/82445
Migrates:
- `all_plugins_app`
- `podspecs`
- `firebase-test-lab`
Minor functional changes to `firebase-test-lab` based on issues highlighted by the migration:
- The build ID used in the path is now a) passable, and b) given a fallback value in the path that
isn't "null"
- Flag setup will no longer assume that `$HOME` must be set in the environment.
- Adds a --build-id flag to `firebase-test-lab` instead of hard-coding the use of `CIRRUS_BUILD_ID`.
The default is still `CIRRUS_BUILD_ID` so no CI changes are needed.
Part of https://github.com/flutter/flutter/issues/81912
Eliminates the global test filesystem and global test packages directory, in favor of local versions. This guarantees that each test runs with a clean filesystem state, rather than relying on cleanup. It also simplifies understanding the tests, since everything is done via params and return values instead of needing to know about the magic global variables and which methods mutate them.
- Replaces most explicit use of `fileSystem` with path construction using the `child*` utility methods
- Removes explicit passing of a filesystem to the commands; we're already passing a `Directory` for the
root where the tool operates, and we should never be using a different filesystem than that directory's
filesystem, so passing it was both redundant, and a potential source of test bugs.
- Removes the .cirrus.yml workaround that removed macOS podspecs before linting
- Deletes the dummy macOS podspecs that are no longer needed due to `flutter` fixes
- Adds --use-modular-headers to the lint command to reflect what Flutter Podfiles do
- Fix the actual issues in the podspecs
If a new version is lower than the current checked-in version, and the
transition from the new version to the old version is valid (indicating
that it could have been the previous version), allow the change.
This prevents the version check from failing when reverting a PR.
This is not fool-proof (e.g., it would allow a revert of an
already-published PR); if we have issues in practice we can further
restrict the check (by checking that the new version is the current pub
version, for instance).
Changes to some files (e.g., CI scripts) have the potential to cause
failures in any packages, without changes to those packages themselves.
This updates the --run-on-changed-packages to consider all packages as
changed if any of those files are changed, to avoid issues where a
change that changes both some repo-level files and some package-specific
files only run presubmit tests on the packages that are directly
changed, causing post-submit-only failures.
Fixes https://github.com/flutter/flutter/issues/82965
- Updates dependencies to null-safe versions
- Migrates common.dart (which doesn't depend on anything)
- Migrates common_tests.dart and its one dependency, utils.dart
- Adds build_runner for Mockito mock generation
- Adds a new utility methods for getting arguments that handle both the casting and the removal of nullability to address a common problematic pattern while migrating code.
- Converts all files, not just the migrated ones, to those new helpers.
Migrating common.dart and utils.dart should unblock a command-by-command migration to null safety.
Reverts the separate of podspect lints into a step that doesn't do a Flutter upgrade
(https://github.com/flutter/plugins/pull/3700) because without that step we had a
version of Dart too old to run null-safe tooling.
First step of https://github.com/flutter/flutter/issues/81912
the skip-confirmation flag will add a --force flag to pub publish, it will also let users to skip the y/n question when pushing tags to remote.
Fixesflutter/flutter#79830
Until the infrastructure limitations that led to flutter/flutter#82032 are fixed, run basic Windows tests via GitHub Actions (which do have VS available) so that we aren't missing all Windows coverage.
