* Enable os3 client by default, via feature flag
* Added changelog
* more explicit UnsupportedSearchException logging
* add os3 modules to assembly, plugin providers and dependencies
* Add more dependencies to assembly
* switch os3 storage module packaging from assembly to shade
* Fix aliases impl in OS3 storage module
* fix manifest
* remove not used assembly.xml
* remove duplicate feature flag
* disable storage module 2 for testing
* fix errors found in full backend tests
* fix index block setting
* fix closed keyword (not used in graylog source)
* use old mechanism for resolving if index exists
* safer index deletion in ClientOS
* remove datanode testing instance from os2 storage module
* fix counts adapter and its test
* remove forgotten service provider
* add index closed exception mapping
* fix status parsing on timeout
* resilient CountsIT#totalThrowsElasticsearchExceptionIfIndexDoesNotExist test
* Index status enum(open/closed), naming conversion left to the storage module
* adapted comment
* code cleanup
---------
Co-authored-by: Matthias Oesterheld <matthias.oesterheld@graylog.com>
* add openrewrite infrastructure and necessary junit jupiter api deps
* update openrewrite docs and remove configLocation from plugin config, so we can override it
* initial openrewrite migration to junit5, work in progress
* replace rule with extension, use mongocollections instead of the instance
* fix parameterized test
* replace jukito with guice-extension for junit5 compatibility
* yet more mongocollections injection
* manually create MongoCollections where we need a custom objectmapper for subtypes
* fix license header
* replace junit4 conditional runner with enabledif static methods, only used in a single test anyway
* fix resource name collision in class loader test
* how did this ever compile?
* use explicit temp dir in tests, otherwise we'd need to create the directories
* add extension that tracks search instances and properly calls their clean up methods according to test lifecycle
* add comment explaining how the closing of testable search instances works with the extension
* use convert rule to extension
* convert rule to extension
* remove old import
* convert SkipDefaultIndexTemplateWatcher to junit5 extension api
* migrate from @Rule to @SearchInstance
* convert remaining @Rule to @SearchInstance
* clean up imports
* replace legacy RestoreSystemProperties rules with custom extension
* migrate all remaining junit4 references
* replace legacy system-rules dep with more modern one for system stream capturing
* remove unused junit4 import
* remove junit5-system-extensions, causes junit engine crashes
* remove legacy junit4 dependency for good :party:
* remove unnecessary openrewrite doc
* move forbiddenapi back out of profile, to avoid it being disabled when other profiles are explicitly given
* suppress forbidden api check for search instance detection
* remove jukito + vintage engine and adapt usage of legacy junit4 apis
* add core junit4 classes Test and Assert to forbidden apis
* replace forbidden junit4 api with junit6 equivalent
* remove guava testlib as it pulls in junit4, replaced FakeTicker with a simpler custom implementation for tests
* update testcontainers to 2.0.2 to get rid of the junit4 transitive dependency
needed to remove all ExternalResources
* migrated MongoDBInstance to MongoDBTestService (basically identically anyway)
* inlined some unnecessary interfaces
removed unused imports of junit4 classes
* migrate junit4 api usage
mockito with junit5 is much stricter about argument matcher types, so we need to adapt to Set -> Collection signature change
* actually start the container now that MongoDBInstance.java is gone
* improve test code
* simplify test case
* simplify test case
* simplify test case
* simplify test case
* simplify test case
* simplify test case
* forgot to remove the commented junit4 superclass
* rename MongoDBTestService#create methods to #createStarted and make start() private
* don't close the per-class mongodb instance for @Nested junit test classes
otherwise the first afterAll callback will destroy the instance and subsequent tests will fail
* first batch of changes to remove the matrix test engine
we are migrating matrix tests into github actions and controlling versions externally
this change starts moving the relevant container management code into the extension
this is work in progress, most tests will fail and fixtures aren't imported yet.
more sweeping changes need to be done to avoid constructor injection due to lifecycle calling order issues
* implement class-lifecycle container handling, and adjust inputcreationIT
* have junit close the backends when their store goes out of context, convert more tests
* we no longer need a vm shutdown hook, junit's extension takes care of the closing
* allow setting the mongodb and search server versions using system properties
* add support for requiring data node in tests, if we test functionality only present for that backend
* fix wrong check for data node and skipping tests, i've borked it
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method and fix warnigs
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method and fix warnings
* load mongodb fixtures and pass through feature flags to allow running plugin tests
* Speed up MavenPackager by by 7 minutes when building JARs
We don't need to create the assemblies.
* Use a single container network for all tests
We cache MongoDB and OpenSearch services so we can't use a new network
on every test. Otherwise, the server would be in a different network
when using cached services.
* More verbose logging for class-lifecycle server backends
* Exit server backend startup early when seeing an unknown host error
* Log MongoDB version in ContainerizedGraylogBackend
* Return the version string instead of the enum
* Fix warning in ContainerizedGraylogBackend
* Don't create a new MongoDB instance for every test
* don't use constructor argument for test base class, it must be static through the lifecycle method
* convert remaining tests to use lifecycle injection
* Introduce #cachedInstance to SearchServerBuilder
Allows the creation of uncached search instances.
* Add MongoDBInstance#instanceId
* Add SearchServerInstance#instancdId
* Stop server early if MongoDB is not reachable
* Fix caching issues in services provider
Disable multi-layered caches and only cache at the service layer.
* We need the server assembly
* Remove connection failure detector again
Exiting early is problematic because a retry might fix the problem.
Add check for NoClassDefFoundError
* make error message nicer
* tests require data node, so skip them on other search versions
* test only runs on data node
* remove custom junit engine
rename annotation to FullBackendTest
remove deprecated annotation options
always start mail and webhook services, they are tiny and fast, simplifies this further (and cuts down on container starts)
some warning cleanup
* re-enable maven packager, committed this by mistake
* Only allow FullBackendTest annotation in full-backend-tests
* Let WebInterfaceAssetsResourceWithPrefixIT run in CLASS lifecycle
The test needs custom configuration.
* Don't allow VM lifecycle with customized config or feature flags
* test with custom config must run in class lifecycle
* Remove commented cache key parts
* Shutdown CLASS services with non-empty config or feature flags
We don't need services for custom configurations to stay around.
* Improve Guice error detection on server startup
* Replace MongodbServer with MongoDBVersion
* Remove unused MongoDBVersion#PATTERN
* Remove unused ContainerVersions
* Extract search server and MongoDB version provider
* Remove unused imports
* Remove SearchServerInstance#searchServer method
Also convert SearchServer from an enum to a POJO with some constants.
* Move SearchServer to org.graylog.testing.storage
* Change log level in GraylogBackendExtension
* Add EnabledIfSearchServer annotation and condition
* Switch DatanodeOpensearchProxyIT to EnabledIfSearchServer
* Switch more tests to EnabledIfSearchServer annotation
* Remove GraylogBackendConfiguration#onlyOnDataNode
* Fix EnabledIfSearchServerCondition class name
* Remove unused DatanodeDevInstance and its builder and provider
* Ensure Maven packaging before starting backend services
Also move check for GRAYLOG_IT_SKIP_PACKAGING environment variable into
the MavenPackager to avoid duplication.
* Set Data Node version to Version.CURRENT_CLASSPATH
* Improve log message in ContainerizedGraylogBackend
* Fix EnabledIfSearchServerCondition logic
Don't check the presence of a version when distribution is DATANODE.
* Add Javadoc to EnabledIfSearchServer
* Fix DatanodeInstance to build the local image
* Don't set GRAYLOG_ELASTICSEARCH_VERSION env when running Data Node
Fixes a startup error where the search version validator failed.
* Pass missing env to DatanodeInstance
* Remove backend services from cache when they get closed
* Adjust name for local Data Node image to be consistent with the server
* Adjust log message in MavenPackager when skipping
* Handle Exception and AssertError in GraylogRestApi#waitFor
Otherwise, we would exit the wait loop when the predicate throws an
exception. (e.g., when we get a 404 instead of a 200)
* Add log().ifError() in WebInterfaceAssetsResourceBase
* take the cluster id from the imported licenses instead of manually setting it through a fixture
* Remove unused GraylogBackendExtension#resolveFixtures method
* Fix Data Node SearchVersionRange checks
This was hardcoded to 5.2.0 because that's the version we used in our
tests. (regardless of which actual version was running)
We are now building the version from the Version.CURRENT_CLASSPATH.
* Relax Data Node search server check
* Pass MavenProjectDirProvider into services provider
* Add the include-frontend flag to the services cache key
* Revert "Add the include-frontend flag to the services cache key"
This reverts commit c25b1a5b3a.
* Revert "Pass MavenProjectDirProvider into services provider"
This reverts commit 37001edcc9.
* Remove MavenProjectDirProviderWithFrontend
* Rename ContainerMatrixElasticsearchBaseTest to SearchServerBaseTest
* Remove unused NodeContainerFactory#containerFileExists
* Don't memoize customization config in customization config provider
It makes testing really hard and requires a server restart to change the
values.
* Fix signal handling in webhook.js test server
Stopping webhook server container was blocking for a long time.
* Add EnabledIfMongoDB annotation
* Move MavenPackager call up into ContainerizedGraylogBackend
That way we will build the JAR files at the very beginning and with the
correct MavenProjectDirProvider. (at least for the first test that runs)
* Adjust MavenPackager logs
* Make completebackend.api.Views#getAll more robust
* Silence MavenPackager
* Use defaultIfBlank instead of requireNonNullElse
* Also use defaultIfBlank for MongoDB version
* Remove unused service image properties
We can add them back if we should implement image selection.
* Cleanup GraylogBackendExtension
* Rename ConfigurationParameter to Env in GraylogBackendConfiguration
* Prevent importLicenses=false in VM lifecycle
* Add Javadoc to GraylogBackendConfiguration
* Move GraylogBackendConfiguration and FullBackendTest annotations
Those were the last files in the containermatrix package.
* More Javadoc
* Remove services from cache before stopping them
* Log if validation fails
* Introduce Server and Data Node product configuration
This replaces the low-level provider configurations for Maven and plugin
JAR files.
* Don't implement AutoClosable for ContainerizedGraylogBackendServicesProvider
We never close it anywhere. Convert the instance method to a static
cleanup method.
* Remove mavenProjectDirProvider, pluginJarsProvider, datanodePluginJarsProvider
* Fix duplicate BeforeAll methods
* Fix logging in completebackend.apis.Views
* Re-enable DbEntitiesCatalogProviderTest
* Use SearchServer.DEFAULT_VERSION for the default distribution
* Remove unused RunningGraylogBackend
* Check composable index templates feature flag in AdaptersES7
Fixes IndicesIT with Elasticsearch 7 and OpenSearch 1.x because they've
always been using composable index templates.
* Remove unused RunningElasticsearchInstanceES7 class
* Add sleep to ensure updatedAt differs when reacquiring lock
* Make EnabledIfSearchServer annotation repeatable
* Improve wording in EnabledIfSearchServerCondition
* Bump default heap in SearchServerBuilder to 1 GB
* Don't set "elasticsearch_version" and let the server detect the version
* Adjust IndexRangesCleanupIT to handle AssertionError in retry loop
* Remove unneccessary warning supression in DatanodeProvisioningIT
* Remove unused testClass param in GraylogBackendExtension#createBackend
---------
Co-authored-by: Bernd Ahlers <bernd@graylog.com>
Co-authored-by: Bernd Ahlers <bernd@users.noreply.github.com>
Co-authored-by: Othello Maurer <othello@graylog.com>
* Wrap test methods in awaitility in DatanodeOpensearchProxyIT
* Wait for the actual messages to be present instead of relying on the message count which could be wrong depending on the order of other tests
* Improve the waiting on index creation
Co-authored-by: Jan Heise <jan.heise@graylog.com>
