- Adds a minimal unit test to url_launcher_linux as a proof of concept. This uses almost exactly the same CMake structure as the Windows version that was added recently.
- Adds Linux support for unit tests to `native-test`, sharing almost all of the existing Windows codepath.
- Fixes the fact that it it was running the debug version of the unit tests, but `build-examples` only builds release. (On other platforms we run debug unit tests, but on those platforms the test command internally builds the requested unit tests, so the mismatch doesn't matter.)
- Enables the new test in CI.
Also opportunistically fixes some documentation in `native_test_command.dart` that wasn't updated as more platform support was added.
Linux portion of https://github.com/flutter/flutter/issues/82445
The build-examples command was filtering what it attempted to build by plugin platform, which means it never does anything for non-plugin packages. flutter/packages has steps that run this command, which suggests it used to work and regressed at some point, but nobody noticed; this will re-enable those builds so that we are getting CI coverage that the examples in flutter/packages build.
Mostly fixes https://github.com/flutter/flutter/issues/88435 (needs a flutter/packages tool pin roll to pick this up)
Moves `publish` to PackageLoopingCommand, giving it the same
standardized output and summary that is used by most other commands in
the tool.
Adds minor new functionality to the base command to allow it to handle
the specific needs of publish:
- Allows fully customizing the set of packages to loop over, to support
--all-changed
- Allows customization of a few more strings so the output better
matches the functionality (e.g., using 'published' instead of 'ran' in
the summary lines).
Fixes https://github.com/flutter/flutter/issues/83413
Eliminates the remaining logic from tool_runner.sh, completing the goal of migrating repository tooling off of bash (both to make maintenance easier, and to better support Windows both locally and in CI). Its branch-based logic is now part of the tool itself, via a new `--packages-for-branch` flag (which is hidden in help since it's only useful for CI).
Part of https://github.com/flutter/flutter/issues/86113
Implements support for `--windows` in `native-test`, for unit tests only. The structure of the new code has most of the new functionality in a generic utility for running GoogleTest test binaries, so that it can be trivially extended to Linux support in a follow-up once the Linux test PoC has landed.
This runs the recently-added `url_launcher_windows` unit test. However, it's not yet run in CI since it needs LUCI bringup; that will be done one this support is in place.
Requires new logic to check if a plugin contains native code, and some new test utility plumbing to generate plugins whose pubspecs indicate that they only contain Dart code to test it, to allow filtering filtering out the FFI-based Windows plugins.
Part of flutter/flutter#82445
Replaces the `publish`-command-specific `--package` flag with support for `--packages`, and unifies the flow with the existing looping for `--all-changed`.
This better aligns the command's API with the rest of the commands, and reduces divergence in the two flows (e.g., `--package` would attempt to publish and fail if the package was already published, whereas now using `--packages` will use the flow that pre-checks against `pub.dev`). It also sets up a structure that will allow easily converting it to the new base package looping command that most other commands now use, which will be done in a follow-up.
Since all calls now attempt to contact `pub.dev`, the tests have been adjusted to always mock the HTTP client so they will be hermetic.
Part of https://github.com/flutter/flutter/issues/83413
We never want a plugin to be published without tagging the release, so there's no reason to support the added complexity of these flags. Similarly, once someone has confirmed publishing, we don't want to give them an opt-out for doing the tag.
This allows building UWP plugin examples with `build-examples --winuwp`. As with previous pre-stable-template desktop support, this avoids the issue of unstable app templates by running `flutter create` on the fly before trying to build, so a template that will bitrot doesn't need to be checked in.
Also adds no-op "support" for `drive-examples --winuwp`, with warnings about it not doing anything. This is to handle the fact that the LUCI recipe is shared between Win32 and UWP, and didn't conditionalize `drive`. Rather than change that, then change it back later, this just adds the no-op support now (since changing the tooling is much easier than changing LUCI recipes currently).
This required some supporting tool changes:
- Adds the ability to check for the new platform variants in a pubspec
- Adds the ability to write test pubspecs that include variants, for testing
Part of https://github.com/flutter/flutter/issues/82817
Replaces almost all of the `TestProcessRunner`, which was specific to the `publish` tests, with the repo-standard `RecordingProcessRunner` (which now has most of the capabilities these tests need). This finishes aligning these tests with the rest of the repository tests, so they will be easier to maintain as part of the overall repository.
To support this, `RecordingProcessRunner` was modified slightly to return a succeeding, no-output process by default for `start`. That makes it consistent with its existing `run` behavior, so is a good change in general.
The error summary for a CHANGELOG validation failure was written when
the only thing being checked was that the versions matched, but now
there are other ways to fail as well (i.e., leaving NEXT). This fixes
the summary message to be more generic so that it doesn't mislead people
who hit validation failures.
While adding the test for this message, I discovered that almost all of
the tests were actually talking to pub.dev, causing their behavior to in
some cases depend on whether a package with that name happened to have
been published, and if so what its version was. In order to make the
tests hermetic and predictable, this fixes that by making all tests use
a mock HTTP client.
Replaces the use of an actual git repository on the filesystem with mock git output and an in-memory filesystem.
This:
- makes the tests more hermetic.
- simplifies the setup of some tests considerably, avoiding the need to run the command once to set up the expected state before running a second time for the intended test.
- eliminates some of the special handling in the test's custom process runner (making it easier to eliminate in a PR that will follow after).
Also adds some output checking in a couple of tests that didn't have enough to ensure that they were necessarily testing the right thing (e.g., testing that a specific thing didn't happen, but not checking that the publish step that could have caused that thing to happen even ran at all).
Finishes the migration of tool tests to `runCapturingPrint`. This makes
the tests much less verbose, and makes it match the rest of the tool
tests.
Also adds the use of `printError` for error output, now that it's
trivial to do so.
Packages are the primary conceptual object in the tool, but currently they are represented simply as Directory (or occasionally a path string). This introduces an object for packages and:
- moves a number of existing utility methods into it
- sweeps the code for the obvious cases of using `Directory` to represent a package, especially in method signatures and migrates them
- notes a few places where we should migrate later, to avoid ballooning the size of the PR
There are no doubt other cases not caught in the sweep, but this gives us a foundation both for new code, and to migrate incrementally toward as we find existing code that was missed.
The mock process runner used in most of the tests had poor handling of
stdout/stderr:
- By default it would return the `List<int>` output from the mock
process, which was never correct since the parent process runner
interface always sets encodings, thus the `dynamic` should always be
of type `String`
- The process for setting output on a `MockProcess` was awkward, since
it was based on a `Stream<Lint<int>>`, even though in every case what
we actually want to do is just set the full output to a string.
- A hack was at some point added (presumably due to the above issues)
to bypass that flow at the process runner level, and instead return a
`String` set there. That meant there were two ways of setting output
(one of which that only worked for one of the ways of running
processes)
- That hack wasn't updated when the ability to return multiple mock
processes instead of a single global mock process was added, so the
API was even more confusing, and there was no way to set different
output for different processes.
This changes the test APIs so that:
- `MockProcess` takes stdout and stderr as strings, and internally
manages converting them to a `Stream<List<int>>`.
- `RecordingProcessRunner` correctly decodes and returns the output
streams when constructing a process result.
It also removes the resultStdout and resultStderr hacks, as well as the
legacy `processToReturn` API, and converts all uses to the new
structure, which is both simpler to use, and clearly associates output
with specific processes.
Ensures that the full relative path in the 'repository' link is correct,
not just the last segment. This ensure that path-level errors (e.g.,
linking to the group directory rather than the package itself for
app-facing packages) are caught.
Also fixes the errors that this improved check catches, including
several cases where a previously unfederated package wasn't fixed when
it was moved to a subdirectory.
The federated plugin spec calls for implementation packages to include an `implements` entry in the `plugins` section of the `pubspec.yaml` indicating what app-facing package it implements. Most of the described behaviors of the `flutter` tool aren't implemented yet, and the pub.dev features have `default_plugin` as a backstop, so we haven't noticed that they are mostly missing (or in one case, incorrect).
To better future-proof the plugins, and to provide a better example to people looking at our plugins as examples of federation, this adds a CI check to make sure that we are correctly adding it, and fixes all of the missing/incorrect values it turned up.
Fixes https://github.com/flutter/flutter/issues/88222
Adds a new `lint-android` command to run `gradlew lint` on Android plugins.
Also standardizes the names of the Cirrus tasks that run all the build and platform-specific (i.e., not Dart unit test) tests for each platform, as they were getting unnecessarily long and complex in some cases.
Fixes https://github.com/flutter/flutter/issues/87071
Adds integration test support for Android to `native-test`. Also fixes
an issue where the existing unit test support was not honoring
`--no-unit`.
Fixes https://github.com/flutter/flutter/issues/86490
The special "NEXT" entry in a CHANGELOG should never be present in a
commit that bumped the version. This validates that this is true even if
the CHANGELOG would be correct for a non-version-change state, to catch
someone incorrectly resolving a merge conflict by leaving both parts of
the conflict, rather than folding the 'NEXT' entry's list into the new
version's notes.
Fixes https://github.com/flutter/flutter/issues/85584
Removes the `build_all_plugins_app.sh` bash script, in support of the goal of eliminating all use of bash from the repository (for maintainability, and for better Windows compatibility).
- The exclusion list moves to a config file, match other recent repo changes
- The exclusion logging moves into the tool itself, consistent with the tool doing more logging of skipped and excluded plugins
- The bulk of the logic moves to a Cirrus task template. This was done instead of rewriting the script in Dart, even though it will mean more work for alternate CI support (e.g., bringing this up on a Windows LUCI bot), because breaking it into components makes it easier to pinpoint failures from the CI UI rather than having all the steps smashed together.
Makes commands that use the package-looping base command track and
report exclusions. This will make it much easier to debug/audit
situations where tests aren't running when expected (e.g., when enabling
a new type of test for a package that previously had to be explicitly
excluded from that test to avoid failing for having no tests, but
forgetting to remove the package from the exclusion list).
Also fixes a latent issue with using different exclusion lists on
different commands in a single CI task when using sharding could cause
unexpected failures due to different sets of plugins being included for
each step (e.g., build+drive with an exclude list on drive could
potentially try to drive a plugin that hadn't been built in that shard)
by sharding before filtering out excluded packages.
Adds testing for sharding in general, as there was previously none.
Currently the tool accepts `--custom-analysis` to allow a list of packages for which custom `analysis_options.yaml` are allowed, and `--exclude` to exclude a set of packages when running a command against all, or all changed, packages. This results in these exception lists being embedded into CI configuration files (e.g., .cirrus.yaml) or scripts, which makes them harder to maintain, and harder to re-use in other contexts (local runs, new CI systems).
This adds support for both flags to accept paths to YAML files that contain the lists, so that they can be maintained separately, and with inline comments about the reasons things are on the lists.
Also updates the CI to use this new support, eliminating those lists from `.cirrus.yaml` and `tool_runner.sh`
Fixes https://github.com/flutter/flutter/issues/86799
Creates a new `native-test` command that will be used to run native unit and UI/integration tests for all platforms over time. This replaces both `xctest` and `java-test`.
For CI we can continue to run each platform separately for clarity, but the combined command makes it easier to use (and remember how to use) for local development, as well as avoiding the need to introduce several new commands for desktop testing as support for that is added to the tool.
Fixes https://github.com/flutter/flutter/issues/84392
Fixes https://github.com/flutter/flutter/issues/86489
If a package supports Android, it will now report failure instead of
skip if no tests run. This matches the new behavior of drive-examples,
and is intended to prevent recurrance of situations where we are
silently failing to run tests because of, e.g., tests being in the wrong
directory.
Also fixes a long-standing but unnoticed problem where if a
run tried to run more than one package's tests, it would hang
forever (although on the bots it doesn't seem to time out, just
end logs abruptly) due to a logic error in the call to configure
gcloud.
Fixesflutter/flutter#86732
To prep for making a combined command to run native tests across different platforms, rework `xctest`:
- Split analyze out into a new `xcode-analyze` command:
- Since the analyze step runs a new build over everything with different flags, this is only a small amount slower than the combined version
- This makes the logic easier to follow
- This allows us to meaningfully report skips, to better notice missing tests.
- Add the ability to target specific test bundles (RunnerTests or RunnerUITests)
To share code between the commands, this extracts a new `Xcode` helper class.
Part of https://github.com/flutter/flutter/issues/84392 and https://github.com/flutter/flutter/issues/86489
Currently each type of check handles its output in isolation, which
creates confusing output when the last check succeeds but an earlier
check fails, since the end of the output will just be a success message.
This makes the output follow the same basic approach as the package
looper commands, where all failures are collected, and then a final
summary is presented at the end, so the last message will always reflect
the important details.
It also adopts the colorized output now used by most other commands.
Allows `format` to run successfully on Windows:
- Ensures that no calls exceed the command length limit.
- Allows specifying a `java` path to make it easier to run without a system Java (e.g., by pointing to the `java` binary in an Android Studio installation).
- Adds clear error messages when `java` or `clang-format` is missing since it's very non-obvious what's wrong otherwise.
Bumps the version, which I intended to do in the previous PR but apparently didn't push to the PR.
The purpose of this PR is to make running all unit tests on Windows pass (vs failing a large portion of the tests as currently happens). This does not mean that the commands actually work when run on Windows, or that Windows support is tested, only that it's possible to actually run the tests themselves. This is prep for actually supporting parts of the tool on Windows in future PRs.
Major changes:
- Make the tests significantly more hermetic:
- Make almost all tools take a `Platform` constructor argument that can be used to inject a mock platform to control what OS the command acts like it is running on under test.
- Add a path `Context` object to the base command, whose style matches the `Platform`, and use that almost everywhere instead of the top-level `path` functions.
- In cases where Posix behavior is always required (such as parsing `git` output), explicitly use the `posix` context object for `path` functions.
- Start laying the groundwork for actual Windows support:
- Replace all uses of `flutter` as a command with a getter that returns `flutter` or `flutter.bat` as appropriate.
- For user messages that include relative paths, use a helper that always uses Posix-style relative paths for consistent output.
This bumps the version since quite a few changes have built up, and having a cut point before starting to make more changes to the commands to support Windows seems like a good idea.
Part of https://github.com/flutter/flutter/issues/86113
- Adds unit tests, as there are currently none.
- Adds more graceful failure handling.
- Adds an internal ignore list to skip files that don't need to be
formatted that showed up during local testing.
- Adds a note explaining that it's intentially not using the new base
command due to performance issues.
Makes validating that there are no unexpected analysis_options.yaml
files part of the per-package loop, rather than a pre-check, so that it
only runs against the target packages. This makes it easier to run
locally on specific packages, since it's not necessary to pass the allow
list for every package when targetting just one package.
* [flutter_plugin_tools] Work around banner in drive-examples
Strip off any unexpected output before parsing `flutter devices
--machine` output. Works around a bug in the Flutter tool that can
result in banners being printed in `--machine` mode.
Fixes https://github.com/flutter/flutter/issues/86052
Migrates `test` to the new package looping base command.
Refactors the bulk of the logic to helpers for easier understanding of the flow.
Part of flutter/flutter#83413
Most of the tool operates on packages in general, and the targetting
done currently by the `--plugins` flag is not actually restricted to
plugins, so this makes the name less confusing.
Part of https://github.com/flutter/flutter/issues/83413
Many commands had insufficient failure testing. This adds new tests that
ensure that for every Process call, at least one test fails if a failure
from that process were ignored (with the exception of calls in the
`publish` command, which has a custom process mocking system, so was out
of scope here; it already has more coverage than most tests did though.)
For a few existing failure tests, adds output checks to ensure that they
are testing for the *right* failures.
Other changes:
- Adds convenience constructors to MockProcess for the common cases of a
mock process that just exits with a 0 or 1 status, to reduce test
verbosity.
- Fixes a few bugs that were found by the new tests.
- Minor test cleanup, especially cases where a mock process was being
set up just to make all calls succeed, which is the default as of
recent changes.
To support this command's --machine flag, which moves all normal output into a field in a JSON struct, adds a way of capturing output and providing it to the command subclass on completion.
Part of flutter/flutter#83413
The intent of package-looping commands is that we always want to run them for all the targeted packages, accumulating failures, and then report them all at the end, so we don't end up with the problem of only finding errors one at a time. However, some of them were using `exitOnError: true` for the underlying commands, causing the first error to be fatal to the test.
This PR:
- Removes all use of `exitOnError: true` from the package-looping commands. (It's still used in `format`, but fixing that is a larger issue, and less important due to the way `format` is currently structured.)
- Fixes the mock process runner used in our tests to correctly simulate `exitOrError: true`; the fact that it didn't was hiding this problem in test (e.g., `drive-examples` had a test that asserted that all failures were summarized correctly, which passed because in tests it was behaving as if `exitOnError` were false)
- Adjusts the mock process runner to allow setting a list of mock result for a specific executable instead of one result for all calls to anything, in order to fix some tests that were broken by the two changes above and unfixable without this (e.g., a test of a command that had been calling one executable with `exitOnError: true` then another with ``exitOnError: false`, where the test was asserting things about the second call failing, which only worked because the first call's failure wasn't actually checked). To limit the scope of the PR, the old method of setting a single result for all calls is still supported for now as a fallback.
- Fixes the fact that the mock `run` and `runAndStream` had opposite default exit code behavior when no mock process was set (since that caused me a lot of confusion while fixing the above until I figured it out).
- Updates the remaining tests (other than one that still needs to be
converted to the new base command, which will be fixed then) that
aren't using runCapturingPrint to do so to reduce test log spam.
- Simplifies and standardizes the matcher used for ToolExit in tests.
Add a summary to the end of successful runs for everything using the new looping base command, similar to what we do for summarizing failures. This will make it easy to manually check results for PRs that we know should be changing the set of run packages (adding a new package, adding a new test type to a package, adding a new test type to the tool), as well as spot-checking when we see unexpected results (e.g., looking back and why a PR didn't fail CI when we discover that it should have).
To support better surfacing skips, this restructures the return value of `runForPackage` to have "skip" as one of the options. As a result of it being a return value, packages that used `printSkip` to indicate that *parts* of the command were being skipped have been changed to no longer do that.
Fixes https://github.com/flutter/flutter/issues/85626
Significantly restructures drive-examples:
- Migrates it to the new package-looping base command
- Enforces that only one platform is passed, since in practice multiple platforms never actually worked. (The logic is structured so that it will be easy to enable multi-platform if `flutter drive` gains multi-platform support.)
- Fixes the issue where `--ios` and `--android` were semi-broken, by doing explicit device targeting for them rather than relying on the default device being the right kind
- Extracts much of the logic to helpers so it's easier to understand the flow
- Removes support for a legacy integration test file structure that is no longer used
- Adds more test coverage; previously no failure cases were actually tested.
Fixes https://github.com/flutter/flutter/issues/85147
Part of https://github.com/flutter/flutter/issues/83413
Migrates firebase-test-lab to use the new package-looping base command.
Other changes:
- Extracts several helpers to make the main flow easier to follow
- Removes support for finding and running `*_e2e.dart` files, since we no longer use that file structure for integration tests.
Part of https://github.com/flutter/flutter/issues/83413
Switches build-examples to the new base command that handles the boilerplate of looping over target packages.
While modifying the command, also does some minor cleanup:
- Extracts a helper to reduce duplicated details of calling `flutter build`
- Switches the flag for iOS to `--ios` rather than `--ipa` since `ios` is what is actually passed to the build command
- iOS no longer defaults to on, so that it behaves like all the other platform flags
- Passing no platform flags is now an error rather than a silent pass, to ensure that we never accidentally have CI doing a no-op run without noticing.
- Rewords the logging slightly for the versions where the label for what is being built is a platform, not an artifact (which is now everything but Android).
Part of flutter/flutter#83413
Combines the two different aspects of version-checking into a single looping structure based on plugins, using the new base command, rather than one operating on plugins as controlled by the usual flags and the other operating on a git list of changed files.
Also simplifies the determination of the new version by simply checking the file, rather than querying git for the HEAD state of the file. Tests setup is simplified since we no longer need to set up nearly as much fake `git` output.
Minor changes to base commands:
- Move indentation up to PackageLoopingCommand so that it is consistent across commands
- Add a new post-loop command for any cleanup, which is needed by version-check
- Change the way the GitDir instance is managed by the base PluginCommand, so that it's always cached even when not overridden, to reduce duplicate work and code.
Part of https://github.com/flutter/flutter/issues/83413
Switches `java-test` to the new base command that handles the boilerplate of looping over target packages.
Includes test improvements:
- Adds failure tests; previously no failure cases were covered.
- Captures output so test output isn't spammed with command output.
Part of flutter/flutter#83413
