* Disable automatic CSR in preflight mode
* added changelog
* run automatic cert renewal for datanode only if preflight is finished
---------
Co-authored-by: Matthias Oesterheld <33032967+moesterheld@users.noreply.github.com>
* Fix repeated opensearch startup during HEALTH_CHECK_FAILED
* Set network.publish_host to hostname for all datanodes
* Verify CA subject name in DatanodeProvisioningIT
* add `DataNodeStatus` and check that data node is `CONNECTED && AVAILABLE` to avoid race condition between ProvisioningPeriodical and NodeHeartbeat
* Simplify TruststoreCreator
---------
Co-authored-by: Matthias Oesterheld <matthias.oesterheld@graylog.com>
* use dependency injection for process and state machine tracer
* fix forbidden annotations
* move removal logic from tracer to process
* add state machine exception handling
* call reset via statemachine, not process
* do not start os process during configure
* temporary fix of tests
* move start/stop process triggers to state machine
* use state machine instead of process
* fix watchdog test
* move removal tracer test to process and remove tracer
* set statemachine for tracer directly
* trigger `PREPARED` instead of `STARTING` after provisioning, startup is handled by Graylog
* add restart transition for reconfiguring test data node
* add missing transition
* make `fire` with errorEvent private
* remove static process variable
* set tracer in `createNew`
* provide default implementation
* revert to default attempts_count
* fix replicas for data stream system indices
fixes#19010
* move `isManagerNode()` to process, only create metrics ism config on manager node
fixes#18830
* cl
* fix cl
* Automate datanode configuration documentation
* work around reflection usage errors in datanode documentation generation
* fixed bindings for datanode
* rename state machine to DataNodeStateMachine to make clear it is not only handling the process anymore
* initial refactoring of packages and classes:
clearly separate processing and opensearch-related classes
* Make ManageableProcess independent from OpensearchStateMachine
* Async cluster restart during remote reindex migration, configuration cleanup
* Safer and faster way to restart and wait for datanode cluster during remote reindex
---------
Co-authored-by: Matthias Oesterheld <33032967+moesterheld@users.noreply.github.com>
* remove first directory compatibility check and steps, rename DIRECTORY_COMPATIBILITY_CHECK_PAGE2 to DIRECTORY_COMPATIBILITY_CHECK_PAGE
* skip ca and cert renewal steps if these exist
rename wrongly named methods
* skip provisioning steps in in-place and remote reindexing migration if data nodes have already been provisioned
only provision data nodes which haven't been provisioned yet
* fix in-place provisioning skip for provisioned and started data nodes
* add directory readable check to compatibility check
run compatibility check in migration trigger (not visible in UI yet)
add retry trigger on failed check
implement directoryCompatibilityCheckOK in MigrationActionsImpl
add directory check button label in UI
* add hostname to CompatibilityResult, run compatibility check for all data nodes on migration action, fix tests
* setting the whitelist separate from the host that we execute the command against. needs to be two settings because the whitelist can contain wildcards
* adding the whitelist to the UI
* fix import, fix text
* fix test
* changing whitelist -> allowlist
* fixing merge conflicts
* Start container services in paralel, bind opensearch from fs instead of copy
* code cleanup
* read-write fs for opensearch in docker, fs bind for plugins
* disable debug logs in integration tests
* preinstall s3 repository plugin for opensearch, bind os distro as read only in docker
* Print unassigned shards if waiting for opensearch fails with yellow status
* Update WaitingRestOperation.java
* add indexsetconfig for replicas to data stream creation
* set number of replicas for data stream backing indices and ism system indices automatically to either indexset default settings or based on the number of available data nodes
* Remove unused opensearch plugins dir references, add datanode directories to status response
* use default config from configuration
---------
Co-authored-by: Jan Heise <jan.heise@graylog.com>
Co-authored-by: Matthias Oesterheld <33032967+moesterheld@users.noreply.github.com>
Co-authored-by: Matthias Oesterheld <matthias.oesterheld@graylog.com>
* do not write/check for lock file in data dir during preflight test
* remove unnecessary directory read check, handled by validator
* add lockfile check to startup
* remove unused exception
* fix compatibility check url
* add more information and show data node list on restart step
* ignore PROCESS_STOPPED event when in WAITING_FOR_CONFIGURATION state to avoid ugly exception
* fix test and show MigrationDatanodeList
* Configure build to create SBOM files for backend and frontend
* Update yarn.lock file
* Reorder imports to please the linter
* Don't modify yarn.lock to avoid conflicts
* Move CycloneDxWebpackPlugin to webpack/core.js
* Build SBOM for the vendor bundle
* Sync yarn.lock
* Add formatter for ISM policy rollover
* Use duration for retention in data node configuration
* License headers
* Fix forbidden error
* Add size formatter, and ability to set rollover minPrimaryShardSize
* Use simple enum size and time formatters with all options
* License headers
* Add IsmTemplate support
---------
Co-authored-by: Matthias Oesterheld <33032967+moesterheld@users.noreply.github.com>
* Fix matrix tests configurations, calculate unique variants for flags and additional params
* don't run cleanup when the server instance has been already closed
* update opensearch in integration tests to 2.12.0
* Migrating `CountsAdapter`.
* Starting to migrate `ClusterAdapter`.
* Migrating usage of `PlainJsonApi`.
* Adding license header.
* Migrating `FieldMappingApi`.
* Fixing up node/host name resolution.
* Migrating `QuerySuggestions` to new client.
* Migrating `StreamsForFieldRetriever` adapter.
* Providing `RestClient` for low-level queries.
* Removing injections of `RestClient` in other adapters.
* Using rest client in data node setup code too.
* Outputting response for debugging.
* SearchJob is aware of executing node. (#18436)
* Allow plugins to add search types to event execution (#18372)
* add event modifier state, allow plugins to add search types for aggregations
* use better variable name
* ensure eventModifierState is not null
* add test for eventModifier state callback
add missing NonNull annotation
properly initialize autovalue
* use binder explicitly to ensure it's not missing if no one is using it
* add changelog
* add license header
* fix invalid test matcher
* use ImmutableMap.of for consistency
* Bump express from 4.18.2 to 4.18.3 in /graylog2-web-interface (#18446)
Bumps [express](https://github.com/expressjs/express) from 4.18.2 to 4.18.3.
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/master/History.md)
- [Commits](https://github.com/expressjs/express/compare/4.18.2...4.18.3)
---
updated-dependencies:
- dependency-name: express
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump @types/react in /graylog2-web-interface/packages/graylog-web-plugin (#18447)
Bumps [@types/react](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react) from 18.2.60 to 18.2.61.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react)
---
updated-dependencies:
- dependency-name: "@types/react"
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Allow overriding `events` search type pagination through execution state. (#18438)
* Allow overriding pagination through execution state.
* Adding missing builder methods.
* Fixing retrieval of cluster settings.
* Retrieving unquoted string fuer settings value.
* Returning `null` if absent.
* Using search instead of counts endpoint to avoid overly large `GET`.
* Improving test.
* Using multi-search to avoid GET becoming too large.
* Including jackson in assembly.
* Reusing `OpensearchClient#search`.
* Including missing client deps in assembly.
* Handle search request failures properly
* Performing suggestions aggregation as msearch.
* Fixing up extraction of corrections.
* Fixing up bucket extraction for numeric values.
* Removing unused object mapper.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jan Heise <jan.heise@graylog.com>
Co-authored-by: luk-kaminski <lukasz.kaminski@graylog.com>
Co-authored-by: Kay Roepke <kroepke@googlemail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
