Adds a `--log-timing` flag that can be passed to cause package-looping
commands to log relative start time and per-package elapsed time, to
help with future efforts to improve CI times (e.g., finding unusually
slow packages, or designing changes to sharding).
Adds this flag to the CI scripts to enable timing there.
The build-examples command was filtering what it attempted to build by plugin platform, which means it never does anything for non-plugin packages. flutter/packages has steps that run this command, which suggests it used to work and regressed at some point, but nobody noticed; this will re-enable those builds so that we are getting CI coverage that the examples in flutter/packages build.
Mostly fixes https://github.com/flutter/flutter/issues/88435 (needs a flutter/packages tool pin roll to pick this up)
Packages are the primary conceptual object in the tool, but currently they are represented simply as Directory (or occasionally a path string). This introduces an object for packages and:
- moves a number of existing utility methods into it
- sweeps the code for the obvious cases of using `Directory` to represent a package, especially in method signatures and migrates them
- notes a few places where we should migrate later, to avoid ballooning the size of the PR
There are no doubt other cases not caught in the sweep, but this gives us a foundation both for new code, and to migrate incrementally toward as we find existing code that was missed.
Makes commands that use the package-looping base command track and
report exclusions. This will make it much easier to debug/audit
situations where tests aren't running when expected (e.g., when enabling
a new type of test for a package that previously had to be explicitly
excluded from that test to avoid failing for having no tests, but
forgetting to remove the package from the exclusion list).
Also fixes a latent issue with using different exclusion lists on
different commands in a single CI task when using sharding could cause
unexpected failures due to different sets of plugins being included for
each step (e.g., build+drive with an exclude list on drive could
potentially try to drive a plugin that hadn't been built in that shard)
by sharding before filtering out excluded packages.
Adds testing for sharding in general, as there was previously none.
The purpose of this PR is to make running all unit tests on Windows pass (vs failing a large portion of the tests as currently happens). This does not mean that the commands actually work when run on Windows, or that Windows support is tested, only that it's possible to actually run the tests themselves. This is prep for actually supporting parts of the tool on Windows in future PRs.
Major changes:
- Make the tests significantly more hermetic:
- Make almost all tools take a `Platform` constructor argument that can be used to inject a mock platform to control what OS the command acts like it is running on under test.
- Add a path `Context` object to the base command, whose style matches the `Platform`, and use that almost everywhere instead of the top-level `path` functions.
- In cases where Posix behavior is always required (such as parsing `git` output), explicitly use the `posix` context object for `path` functions.
- Start laying the groundwork for actual Windows support:
- Replace all uses of `flutter` as a command with a getter that returns `flutter` or `flutter.bat` as appropriate.
- For user messages that include relative paths, use a helper that always uses Posix-style relative paths for consistent output.
This bumps the version since quite a few changes have built up, and having a cut point before starting to make more changes to the commands to support Windows seems like a good idea.
Part of https://github.com/flutter/flutter/issues/86113
To support this command's --machine flag, which moves all normal output into a field in a JSON struct, adds a way of capturing output and providing it to the command subclass on completion.
Part of flutter/flutter#83413
Add a summary to the end of successful runs for everything using the new looping base command, similar to what we do for summarizing failures. This will make it easy to manually check results for PRs that we know should be changing the set of run packages (adding a new package, adding a new test type to a package, adding a new test type to the tool), as well as spot-checking when we see unexpected results (e.g., looking back and why a PR didn't fail CI when we discover that it should have).
To support better surfacing skips, this restructures the return value of `runForPackage` to have "skip" as one of the options. As a result of it being a return value, packages that used `printSkip` to indicate that *parts* of the command were being skipped have been changed to no longer do that.
Fixes https://github.com/flutter/flutter/issues/85626
Most of our commands are generally of the form:
```
for (each plugin as defined by the tool flags)
check some things for success or failure
print a summary all of the failing things
exit non-zero if anything failed
```
Currently all that logic not consistent, having been at various points copied and pasted around, modified, in some cases rewritten. There's unnecessary boilerplate in each new command, and there's unnecessary variation that makes it harder both to maintain the tool, and to consume the test output:
- There's no standard format for separating each plugin's run to search within a log
- There's no standard format for the summary at the end
- In some cases commands have been written to ToolExit on failure, which means we don't actually get the rest of the runs
This makes a new base class for commands that follow this structure to use, with shared code for all the common bits. This makes it harder to accidentally write new commands incorrectly, easier to maintain the code, and lets us standardize output so that searching within large logs will be easier.
This ports two commands over as a proof of concept to demonstrate that it works; more will be converted in follow-ups.
Related to https://github.com/flutter/flutter/issues/83413
