* removal of classes that went to Jadconfig
adding new steps to the server package
* adding missing license
* adding the config descriptions from server.conf to the annotations
* adding the config descriptions from server.conf to the annotations
* Update feature-flag.config
* jadconfig 1.0.0
* setting a valid default for the node id file in tests
* fixing dependency
* fix validation + fix test
* fixing test
* fixing test
* fix directories
* fix test
* fix test
* fix test
* fix test
* adding missing doc annotations
* merge master & fix missing doc annotation
* Adapt section headings for documentation
* Removed documentation test (will be replaced by annotation processor)
---------
Co-authored-by: Jan Heise <jan.heise@graylog.com>
* removal of classes that went to Jadconfig
adding new steps to the server package
* adding missing license
* adding the config descriptions from server.conf to the annotations
* adding the config descriptions from server.conf to the annotations
* Update feature-flag.config
* jadconfig 1.0.0
* setting a valid default for the node id file in tests
* fixing dependency
* fix validation + fix test
* fixing test
* fixing test
* fix directories
* fix test
* fix test
* fix test
* fix test
* adding missing doc annotations
* merge master & fix missing doc annotation
* Adapt section headings for documentation
---------
Co-authored-by: Tomas Dvorak <tomas.dvorak@graylog.com>
* add openrewrite infrastructure and necessary junit jupiter api deps
* update openrewrite docs and remove configLocation from plugin config, so we can override it
* initial openrewrite migration to junit5, work in progress
* replace rule with extension, use mongocollections instead of the instance
* fix parameterized test
* replace jukito with guice-extension for junit5 compatibility
* yet more mongocollections injection
* manually create MongoCollections where we need a custom objectmapper for subtypes
* fix license header
* replace junit4 conditional runner with enabledif static methods, only used in a single test anyway
* fix resource name collision in class loader test
* how did this ever compile?
* use explicit temp dir in tests, otherwise we'd need to create the directories
* add extension that tracks search instances and properly calls their clean up methods according to test lifecycle
* add comment explaining how the closing of testable search instances works with the extension
* use convert rule to extension
* convert rule to extension
* remove old import
* convert SkipDefaultIndexTemplateWatcher to junit5 extension api
* migrate from @Rule to @SearchInstance
* convert remaining @Rule to @SearchInstance
* clean up imports
* replace legacy RestoreSystemProperties rules with custom extension
* migrate all remaining junit4 references
* replace legacy system-rules dep with more modern one for system stream capturing
* remove unused junit4 import
* remove junit5-system-extensions, causes junit engine crashes
* remove legacy junit4 dependency for good :party:
* remove unnecessary openrewrite doc
* move forbiddenapi back out of profile, to avoid it being disabled when other profiles are explicitly given
* suppress forbidden api check for search instance detection
* remove jukito + vintage engine and adapt usage of legacy junit4 apis
* add core junit4 classes Test and Assert to forbidden apis
* replace forbidden junit4 api with junit6 equivalent
* remove guava testlib as it pulls in junit4, replaced FakeTicker with a simpler custom implementation for tests
* update testcontainers to 2.0.2 to get rid of the junit4 transitive dependency
needed to remove all ExternalResources
* migrated MongoDBInstance to MongoDBTestService (basically identically anyway)
* inlined some unnecessary interfaces
removed unused imports of junit4 classes
* migrate junit4 api usage
mockito with junit5 is much stricter about argument matcher types, so we need to adapt to Set -> Collection signature change
* actually start the container now that MongoDBInstance.java is gone
* improve test code
* simplify test case
* simplify test case
* simplify test case
* simplify test case
* simplify test case
* simplify test case
* forgot to remove the commented junit4 superclass
* rename MongoDBTestService#create methods to #createStarted and make start() private
* don't close the per-class mongodb instance for @Nested junit test classes
otherwise the first afterAll callback will destroy the instance and subsequent tests will fail
* Hot reloading of datanode certificates
* Differentiate between certificate change and certificate renewal in datanode events
* fixed log message
* added changelog
* code cleanup
* code cleanup and typos
* remove event handling as this also triggers the `OpensearchConfigurationChangeEvent`
* Fix DatanodeRestApiProxy
* make keystore a singleton to respect synchronized calls
---------
Co-authored-by: Matthias Oesterheld <33032967+moesterheld@users.noreply.github.com>
Co-authored-by: Matthias Oesterheld <matthias.oesterheld@graylog.com>
* first batch of changes to remove the matrix test engine
we are migrating matrix tests into github actions and controlling versions externally
this change starts moving the relevant container management code into the extension
this is work in progress, most tests will fail and fixtures aren't imported yet.
more sweeping changes need to be done to avoid constructor injection due to lifecycle calling order issues
* implement class-lifecycle container handling, and adjust inputcreationIT
* have junit close the backends when their store goes out of context, convert more tests
* we no longer need a vm shutdown hook, junit's extension takes care of the closing
* allow setting the mongodb and search server versions using system properties
* add support for requiring data node in tests, if we test functionality only present for that backend
* fix wrong check for data node and skipping tests, i've borked it
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method and fix warnigs
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method
* convert constructor injection to lifecycle method and fix warnings
* load mongodb fixtures and pass through feature flags to allow running plugin tests
* Speed up MavenPackager by by 7 minutes when building JARs
We don't need to create the assemblies.
* Use a single container network for all tests
We cache MongoDB and OpenSearch services so we can't use a new network
on every test. Otherwise, the server would be in a different network
when using cached services.
* More verbose logging for class-lifecycle server backends
* Exit server backend startup early when seeing an unknown host error
* Log MongoDB version in ContainerizedGraylogBackend
* Return the version string instead of the enum
* Fix warning in ContainerizedGraylogBackend
* Don't create a new MongoDB instance for every test
* don't use constructor argument for test base class, it must be static through the lifecycle method
* convert remaining tests to use lifecycle injection
* Introduce #cachedInstance to SearchServerBuilder
Allows the creation of uncached search instances.
* Add MongoDBInstance#instanceId
* Add SearchServerInstance#instancdId
* Stop server early if MongoDB is not reachable
* Fix caching issues in services provider
Disable multi-layered caches and only cache at the service layer.
* We need the server assembly
* Remove connection failure detector again
Exiting early is problematic because a retry might fix the problem.
Add check for NoClassDefFoundError
* make error message nicer
* tests require data node, so skip them on other search versions
* test only runs on data node
* remove custom junit engine
rename annotation to FullBackendTest
remove deprecated annotation options
always start mail and webhook services, they are tiny and fast, simplifies this further (and cuts down on container starts)
some warning cleanup
* re-enable maven packager, committed this by mistake
* Only allow FullBackendTest annotation in full-backend-tests
* Let WebInterfaceAssetsResourceWithPrefixIT run in CLASS lifecycle
The test needs custom configuration.
* Don't allow VM lifecycle with customized config or feature flags
* test with custom config must run in class lifecycle
* Remove commented cache key parts
* Shutdown CLASS services with non-empty config or feature flags
We don't need services for custom configurations to stay around.
* Improve Guice error detection on server startup
* Replace MongodbServer with MongoDBVersion
* Remove unused MongoDBVersion#PATTERN
* Remove unused ContainerVersions
* Extract search server and MongoDB version provider
* Remove unused imports
* Remove SearchServerInstance#searchServer method
Also convert SearchServer from an enum to a POJO with some constants.
* Move SearchServer to org.graylog.testing.storage
* Change log level in GraylogBackendExtension
* Add EnabledIfSearchServer annotation and condition
* Switch DatanodeOpensearchProxyIT to EnabledIfSearchServer
* Switch more tests to EnabledIfSearchServer annotation
* Remove GraylogBackendConfiguration#onlyOnDataNode
* Fix EnabledIfSearchServerCondition class name
* Remove unused DatanodeDevInstance and its builder and provider
* Ensure Maven packaging before starting backend services
Also move check for GRAYLOG_IT_SKIP_PACKAGING environment variable into
the MavenPackager to avoid duplication.
* Set Data Node version to Version.CURRENT_CLASSPATH
* Improve log message in ContainerizedGraylogBackend
* Fix EnabledIfSearchServerCondition logic
Don't check the presence of a version when distribution is DATANODE.
* Add Javadoc to EnabledIfSearchServer
* Fix DatanodeInstance to build the local image
* Don't set GRAYLOG_ELASTICSEARCH_VERSION env when running Data Node
Fixes a startup error where the search version validator failed.
* Pass missing env to DatanodeInstance
* Remove backend services from cache when they get closed
* Adjust name for local Data Node image to be consistent with the server
* Adjust log message in MavenPackager when skipping
* Handle Exception and AssertError in GraylogRestApi#waitFor
Otherwise, we would exit the wait loop when the predicate throws an
exception. (e.g., when we get a 404 instead of a 200)
* Add log().ifError() in WebInterfaceAssetsResourceBase
* take the cluster id from the imported licenses instead of manually setting it through a fixture
* Remove unused GraylogBackendExtension#resolveFixtures method
* Fix Data Node SearchVersionRange checks
This was hardcoded to 5.2.0 because that's the version we used in our
tests. (regardless of which actual version was running)
We are now building the version from the Version.CURRENT_CLASSPATH.
* Relax Data Node search server check
* Pass MavenProjectDirProvider into services provider
* Add the include-frontend flag to the services cache key
* Revert "Add the include-frontend flag to the services cache key"
This reverts commit c25b1a5b3a.
* Revert "Pass MavenProjectDirProvider into services provider"
This reverts commit 37001edcc9.
* Remove MavenProjectDirProviderWithFrontend
* Rename ContainerMatrixElasticsearchBaseTest to SearchServerBaseTest
* Remove unused NodeContainerFactory#containerFileExists
* Don't memoize customization config in customization config provider
It makes testing really hard and requires a server restart to change the
values.
* Fix signal handling in webhook.js test server
Stopping webhook server container was blocking for a long time.
* Add EnabledIfMongoDB annotation
* Move MavenPackager call up into ContainerizedGraylogBackend
That way we will build the JAR files at the very beginning and with the
correct MavenProjectDirProvider. (at least for the first test that runs)
* Adjust MavenPackager logs
* Make completebackend.api.Views#getAll more robust
* Silence MavenPackager
* Use defaultIfBlank instead of requireNonNullElse
* Also use defaultIfBlank for MongoDB version
* Remove unused service image properties
We can add them back if we should implement image selection.
* Cleanup GraylogBackendExtension
* Rename ConfigurationParameter to Env in GraylogBackendConfiguration
* Prevent importLicenses=false in VM lifecycle
* Add Javadoc to GraylogBackendConfiguration
* Move GraylogBackendConfiguration and FullBackendTest annotations
Those were the last files in the containermatrix package.
* More Javadoc
* Remove services from cache before stopping them
* Log if validation fails
* Introduce Server and Data Node product configuration
This replaces the low-level provider configurations for Maven and plugin
JAR files.
* Don't implement AutoClosable for ContainerizedGraylogBackendServicesProvider
We never close it anywhere. Convert the instance method to a static
cleanup method.
* Remove mavenProjectDirProvider, pluginJarsProvider, datanodePluginJarsProvider
* Fix duplicate BeforeAll methods
* Fix logging in completebackend.apis.Views
* Re-enable DbEntitiesCatalogProviderTest
* Use SearchServer.DEFAULT_VERSION for the default distribution
* Remove unused RunningGraylogBackend
* Check composable index templates feature flag in AdaptersES7
Fixes IndicesIT with Elasticsearch 7 and OpenSearch 1.x because they've
always been using composable index templates.
* Remove unused RunningElasticsearchInstanceES7 class
* Add sleep to ensure updatedAt differs when reacquiring lock
* Make EnabledIfSearchServer annotation repeatable
* Improve wording in EnabledIfSearchServerCondition
* Bump default heap in SearchServerBuilder to 1 GB
* Don't set "elasticsearch_version" and let the server detect the version
* Adjust IndexRangesCleanupIT to handle AssertionError in retry loop
* Remove unneccessary warning supression in DatanodeProvisioningIT
* Remove unused testClass param in GraylogBackendExtension#createBackend
---------
Co-authored-by: Bernd Ahlers <bernd@graylog.com>
Co-authored-by: Bernd Ahlers <bernd@users.noreply.github.com>
Co-authored-by: Othello Maurer <othello@graylog.com>
* enforce DB 7.0 and remove 5.0 references
* bump mongo version for datanode
* add links to Mongo upgrade docs
* more upgrade guidance
---------
Co-authored-by: Anton Ebel <anton.ebel@graylog.com>
* Replace WithBeanGetterTest with AutoValueValidatorTest
We want to ensure that auto-value adds the validation annotations to the
generated class fields.
* Remove all WithBeanGetter annotation usages
- Fix classes where the constructor became private because of the
builder usage
- Fix wrong "$AutoValue" references
* Remove auto-value-javabean dependency
* Restore InputCreateRequest#create method
* Revert some JsonSeserialize conversions
The changes introduced a breaking change in InputCreateRequest, so we
play safe for now.
* Remove JsonDeserialize from ShardRouting
* Convert CreateEntityRequest to a POJO to fix validation issues
* Check that CreateEntityRequest#title is not null
* Fix botched merge
* Explicitly set JsonProperty names to fix parsing issue
* Remove unused files
* Use explicit values for JsonProperty annotations
Removing the WithBeanGetter annotation resulted in some de/serialization
errors. We prevent those by using explicit annotation values.
* Fix wrong property name
* Add PMD rule to check for missing values in JsonProperty annotations
* Remove newly introduced WithBeanGetter usages
* Add missing license header
* Fix "$AutoValue_*" usage
* use os 2.19.0-rc4 in data node
* use 2.18.0 version of repository-s3 plugin
* fix s3 plugin version
* use repository-s3 from build
* remove expired certificates from data node truststore
* use ga release
* disable system indices temporarily because replicas are not set correctly in single-node installations (see https://github.com/opensearch-project/security/issues/4739)
* remove query-insights plugin
* do not use shaded lucene classes to avoid running into UnsupportedOperationException - if the singleton getter is not called from known Lucene classes in the lucene `VectorizationProvider` called by `DirectoryReader#open`
* use OS 2.19.0 shaded clients
* use patched repository-s3 (see
https://github.com/opensearch-project/OpenSearch/issues/17363)
* add lucene-backward-codecs
* use OS 2.19.1
* update to 2.19.3
* correct testcontainers version
* change order of deleting os plugins
* temporarily use 2.19.2
* temporarily use 2.19.2
* switch back to 2.19.3, temporarily disable remote reindexing it
* Changelog
* update containermatrix os server to 2.19.3
* adjust hdfs plugin hash
* Better error message in test
* add more verbose failure message
* change default_number_of_indices for test OS
* add setting only for 2.19.3
---------
Co-authored-by: Tomas Dvorak <tomas.dvorak@graylog.com>
* Remove maven.compiler.* properties from plugin archetype
They are already defined in org.graylog:graylog-parent
* Maven: update java compiler version from 17 to 21
* Update java in Github workflows
* Fix deprecated usage of URL constructor
* Fix deprecated usage of Thread#id
* Catch IllegalArgumentException when URLs are not absolute
* Remove PMD plugin for lack of java 21 support
* Update to Java 21
* Inherig compiler source/target settings from parent
* Use Java 21 in test containers
* Fix URL deprecation error for Java 21
* Setup Mockito Java Agent for surefire
See "Explicitly setting up instrumentation for inline mocking (Java 21+)"
at https://javadoc.io/doc/org.mockito/mockito-core/latest/org.mockito/org/mockito/Mockito.html#0.3
* Switch dependency plugin to test phase and fix argLine
* Get dependency properties in test-compile phase
* Add changelog entry and adjust UPGRADING.md
---------
Co-authored-by: Othello Maurer <othello@graylog.com>
