* Create Data Stream template, prepare ism policy
* add ism
* create ism and collect jvm heap as first metric
* change lookup and term query to use data streams
* start moving data stream functionality to graylog
* move data stream management into graylog
* code cleanup, add tests
* better validation, clean up after tests
* additional clean up after IT
* don't remove `index-management` plugin from datanode
* inject ObjectMapper
* change number of replicas from default 1 to 0 for test
* set `number_of_replicas` to 0 for .opendistro-ism-config in test
* replace javax with jakarta classes
* fix integration test
* remove data streams on test cleanup
* set high priority for data stream template
* ensure password_secret meets length requirement
* update CL
* update test
* adjusted wording so it references the data node
---------
Co-authored-by: Jan Heise <jan.heise@graylog.com>
* Parser for opensearch/elasticsearch data directories
* add opensearch 1.3 and elasticsearch 6.8.0 to indexer parser
* fixed paths
* code cleanup
* remove explicit dependency version of jackson smile
* Preflight check for index dir compatibility
* indexer dir parser fixed for empty dirs
* added changelog
* rest controller for indices directory parser
* require auth for indices compatibility rest api
* catch incompatibility exceptions of indices dir parsing
* code cleanup
* better dir detection handling
---------
Co-authored-by: Matthias Oesterheld <33032967+moesterheld@users.noreply.github.com>
- Read the "max_map_count" from the /proc system instead of calling the
/sbin/sysctl command
- Only run check on Linux operating systems (avoids warning on FreeBSD)
- Throw "PreflightCheckException" instead of "RuntimeException"
- Don't use "System.out.println()" for logging
- Enhance changelog entry to help users to understand the fix
Refs https://github.com/Graylog2/graylog2-server/pull/17436
* publish data node status
* add paginated datanode service
* changelog
* adjust changelog
* adjust NodeImplTest
* refactor initial separation of server node and data node
* fix backward compatibility
* refactor separation of server node and data node
* add timezone
* add new service tests
* add entity tests
* inject DataNodeDto NodeService
* service, rest & permissions for data node management
* add tests
* add reset data node event
* add states and transitions for node removal
add node removal functionality
* changelog
* add datanode permissions to permission list
* add audit event types to list
* add custom thread factory
* add node removal integration test
refactor rest calls in ITs
* wait for primary shard node availability
* refactor duplicate code
* simplify node retrieval
* add `@OnlyInSecuredNode` annotation and request filter
move delete endpoint to management controller and secure it
adjust IT
change javadoc
* stabilize tests
* using the hostname as the default node-name, if none was specified explicitly
* using the hostname as the default node-name, if none was specified explicitly
* using the hostname as the default node-name, if none was specified explicitly
* nodeName is either set explicitly in the conf file or will be the result of the hostname setting otherwise
* small refactoring
* Fix file permissions for key material #17009
* added changelog
* correct file permissions for private key used during preflight CSR
* better http and transport keystore files generation
* removed changelog
---------
Co-authored-by: Jan Heise <jan.heise@graylog.com>
* Include local reverse lookups for loopback addresses in CSR.
* Add reverse lookups for local loopback-addresses to CSR.
* Suppressing forbidden method invocation warning.
* writing opensearch.yml instead of env vars
* appending to an existing original config file to be able to check in default config options in git
* remove original config file, always create/write config
* removed unnecessary imports
* refactor datanode directories checks, use configuration validators
* comments, error messages
* fix datanode config dir in docker
* force image pull strategy for datanode dev container
* remove image pull strategy
* More detailed DirectoryWritableValidator for datanode
* remove commons dependency from UploadedCertFilesSecureConfiguration
* Use random http auth password during preflight web setup
* set initial preflight password once and persist it in mongodb
* refactor structure of the preflight config service, add migration
* add license
* code cleanup
* manually trigger password creation migration in preflight
* Fixing up import.
* fixed preflight auth filter init
* migrations for prefligh boot phase
* code cleanup
* add migration to the PreflightWebModule
* fix injector reference in preflight migrations
* Include logo, make sure that all relevant information is shown at last.
* Consolidating banner logging into single call.
---------
Co-authored-by: Dennis Oelkers <dennis@graylog.com>
* fixing discovery process for new nodes
* only write discovery seeds on start of the opensearch process depending on the list of active nodes.
* fix dependent init
* fix method call to not depend on pre-initialized members
* fixing parameter map creation
* fixing test and using correct parameter
* fixing test, fixing default roles
* fixing tests
* basic auth for datanode rest api if security enabled
* added changelog
* adapted to master changes
* configurable rest admin username in datanode
---------
Co-authored-by: Jan Heise <jan.heise@graylog.com>
* jwt implementation, adding preflight + versionprobe
* adding JWT auth to anomaly detection client, some minor refactorings
* added missing license
* added provider, refactorings
* fixing small bugs
* set smaller default expiration
* using the correct OpenSearch distro for the erchitecture
* added config options
* small refactorings, fixing tests, adding dependencies for running tests from inside IntelliJ
* fixing import to correct Duration class
* removing unnecessary debug output
* fixing tests
* fixing tests
* adding JWT to Cluster test
* fixing path
* fixing path problems
* fix architecture logic for DatanodeContainerizedBackend
* fixing path problems
* fixing JWT settings in tests
* fixing JWT settings in tests
* fixing cluster test
* fixing cluster test
* fixing cluster test
* adding missing jwt token
* small refactoring to force thread-safety on creation
* the datanode uses mongodb - so the closing sequence needs to be adjusted
* moving config changes into config file, keep only change that has to be done by code
* adding breaking change info for JWT and clock synchronization.
* renaming config settings
* renaming config settings
* removed debug settings that are no longer in use
* renamed provider/annotation to be more precise
* lowering log level
* smaller refactorings/renamings
* keep config change in code until PR is merged, remove code in a follow-up PR
* moved changelog file to correct folder
* clock synchronsation
Co-authored-by: Dennis Oelkers <dennis@graylog.com>
* added jwt auth into changes after merge
* upped the timeout
* log curl connection information in DatanodeOpensearchWait
* replacing with jwt auth
* changing wait strategy
* fixing imports
* remove uname/pwd from opensearch rest base url
* propagate jwt token provider to http rest client callback, obtain fresh token every time
* inject IndexerJwtAuthTokenProvider in DatanodeConfigurationProvider
* remove rest uname/pwd from datanode configuration
---------
Co-authored-by: Tomas Dvorak <tomas.dvorak@graylog.com>
Co-authored-by: Dennis Oelkers <dennis@graylog.com>
* Update opensearch to 2.9.0 in matrix integration tests
* Update datanode to 2.9.0 opensearch
* workaround for opensearch number_of_replicas in single-node mode
* trigger the fix of replica numbers in opensearch only for 2.9.0 and higher releases
* better hostname handling for DatanodeClusterIT
* log all request/responses for DatanodeClusterIT
* add response logging to DatanodeClusterIT
* print also logs from node B in DatanodeClusterIT in case of error
* fixed DatanodeClusterIT, cached datanode images, synchronized maven packaging
* better handling of DatanodeClusterIT waiting for cluster status
* code cleanup
* log last recorded exception in case of failed waiting for datanode opensearch
* log state machine transitions
* increase waiting timeout for DatanodeClusterIT
* print shard info in case of datanode cluster failure
* safer shard info debug in DatanodeOpensearchWait
* disable scluster.routing.allocation.disk.threshold for datanode integration tests (causes problems in github actions)
---------
Co-authored-by: Dennis Oelkers <dennis@graylog.com>
