* Create Data Stream template, prepare ism policy
* add ism
* create ism and collect jvm heap as first metric
* change lookup and term query to use data streams
* start moving data stream functionality to graylog
* move data stream management into graylog
* code cleanup, add tests
* better validation, clean up after tests
* additional clean up after IT
* don't remove `index-management` plugin from datanode
* inject ObjectMapper
* change number of replicas from default 1 to 0 for test
* set `number_of_replicas` to 0 for .opendistro-ism-config in test
* replace javax with jakarta classes
* fix integration test
* remove data streams on test cleanup
* set high priority for data stream template
* Update to OpenSearch 2.11.0
* Update to OpenSearch 2.11.0
* reverting test back
* reverting test back
* better handling of waiting for datanode startups in integration tests
---------
Co-authored-by: Tomas Dvorak <tomas.dvorak@graylog.com>
* Parser for opensearch/elasticsearch data directories
* add opensearch 1.3 and elasticsearch 6.8.0 to indexer parser
* fixed paths
* code cleanup
* remove explicit dependency version of jackson smile
* Preflight check for index dir compatibility
* indexer dir parser fixed for empty dirs
* added changelog
* rest controller for indices directory parser
* require auth for indices compatibility rest api
* catch incompatibility exceptions of indices dir parsing
* code cleanup
* better dir detection handling
---------
Co-authored-by: Matthias Oesterheld <33032967+moesterheld@users.noreply.github.com>
* publish data node status
* add paginated datanode service
* changelog
* adjust changelog
* adjust NodeImplTest
* refactor initial separation of server node and data node
* fix backward compatibility
* refactor separation of server node and data node
* add timezone
* add new service tests
* add entity tests
* inject DataNodeDto NodeService
* service, rest & permissions for data node management
* add tests
* add reset data node event
* add states and transitions for node removal
add node removal functionality
* changelog
* add datanode permissions to permission list
* add audit event types to list
* add custom thread factory
* add node removal integration test
refactor rest calls in ITs
* wait for primary shard node availability
* refactor duplicate code
* simplify node retrieval
* add `@OnlyInSecuredNode` annotation and request filter
move delete endpoint to management controller and secure it
adjust IT
change javadoc
* stabilize tests
* Fix file permissions for key material #17009
* added changelog
* correct file permissions for private key used during preflight CSR
* better http and transport keystore files generation
* removed changelog
---------
Co-authored-by: Jan Heise <jan.heise@graylog.com>
* refactor datanode directories checks, use configuration validators
* comments, error messages
* fix datanode config dir in docker
* force image pull strategy for datanode dev container
* remove image pull strategy
* More detailed DirectoryWritableValidator for datanode
* remove commons dependency from UploadedCertFilesSecureConfiguration
* Fixing compilation of single module.
* Adding test case to ensure that unwanted plugins are not included/loaded.
* Adding license header.
* Doing simple web frontend change to trigger reviewbot.
* Fixing linter hint.
* fixing discovery process for new nodes
* only write discovery seeds on start of the opensearch process depending on the list of active nodes.
* fix dependent init
* fix method call to not depend on pre-initialized members
* fixing parameter map creation
* fixing test and using correct parameter
* fixing test, fixing default roles
* fixing tests
* basic auth for datanode rest api if security enabled
* added changelog
* adapted to master changes
* configurable rest admin username in datanode
---------
Co-authored-by: Jan Heise <jan.heise@graylog.com>
* adding datanode dev provider, aligning password/secret/mongoUri usage
* starting mongoDb
* adding providers etc.
* fixing init procedure
* fixing missing license headers
* fixing path
* removing test dependencies
* adding transport, fixed naming
* fixing dependencies
* disabling enormous output
* fixing wrong param
* added hostname as possible config option
* fixed initial cluster name
* fixing dependencies and tests
* fixing dependencies and tests
* fixing dependencies and tests
* fixing test
* trying to load os distri from both possible locations
* fixing paths
* run mvn packager early
* testing full matrix
* adding more debug info
* removing network closing because it's never opened here - so we keep the closing responsibility in one place
* never create a network automatically here
* adding network close
* adding network close
* adding network close
* finding project version and base path consistently
* setting smaller default matrix
* jwt implementation, adding preflight + versionprobe
* adding JWT auth to anomaly detection client, some minor refactorings
* added missing license
* added provider, refactorings
* fixing small bugs
* set smaller default expiration
* using the correct OpenSearch distro for the erchitecture
* added config options
* small refactorings, fixing tests, adding dependencies for running tests from inside IntelliJ
* fixing import to correct Duration class
* removing unnecessary debug output
* fixing tests
* fixing tests
* adding JWT to Cluster test
* fixing path
* fixing path problems
* fix architecture logic for DatanodeContainerizedBackend
* fixing path problems
* fixing JWT settings in tests
* fixing JWT settings in tests
* fixing cluster test
* fixing cluster test
* fixing cluster test
* adding missing jwt token
* small refactoring to force thread-safety on creation
* the datanode uses mongodb - so the closing sequence needs to be adjusted
* moving config changes into config file, keep only change that has to be done by code
* adding breaking change info for JWT and clock synchronization.
* renaming config settings
* renaming config settings
* removed debug settings that are no longer in use
* renamed provider/annotation to be more precise
* lowering log level
* smaller refactorings/renamings
* keep config change in code until PR is merged, remove code in a follow-up PR
* moved changelog file to correct folder
* clock synchronsation
Co-authored-by: Dennis Oelkers <dennis@graylog.com>
* added jwt auth into changes after merge
* upped the timeout
* log curl connection information in DatanodeOpensearchWait
* replacing with jwt auth
* changing wait strategy
* fixing imports
* remove uname/pwd from opensearch rest base url
* propagate jwt token provider to http rest client callback, obtain fresh token every time
* inject IndexerJwtAuthTokenProvider in DatanodeConfigurationProvider
* remove rest uname/pwd from datanode configuration
---------
Co-authored-by: Tomas Dvorak <tomas.dvorak@graylog.com>
Co-authored-by: Dennis Oelkers <dennis@graylog.com>
* Update opensearch to 2.9.0 in matrix integration tests
* Update datanode to 2.9.0 opensearch
* workaround for opensearch number_of_replicas in single-node mode
* trigger the fix of replica numbers in opensearch only for 2.9.0 and higher releases
* better hostname handling for DatanodeClusterIT
* log all request/responses for DatanodeClusterIT
* add response logging to DatanodeClusterIT
* print also logs from node B in DatanodeClusterIT in case of error
* fixed DatanodeClusterIT, cached datanode images, synchronized maven packaging
* better handling of DatanodeClusterIT waiting for cluster status
* code cleanup
* log last recorded exception in case of failed waiting for datanode opensearch
* log state machine transitions
* increase waiting timeout for DatanodeClusterIT
* print shard info in case of datanode cluster failure
* safer shard info debug in DatanodeOpensearchWait
* disable scluster.routing.allocation.disk.threshold for datanode integration tests (causes problems in github actions)
---------
Co-authored-by: Dennis Oelkers <dennis@graylog.com>
* Insecure startup of OS in datanode only if explicitly configured
* Data node can be started without starting OS, just to be discovered by pre-flight mode
* Start datanode jersey only after ssl configuration resolved, listen to change events
* Adapted DatanodeSecuritySetupIT to work with https version of the datanode interface
* simplify hostnames in DatanodeSecuritySetupIT
* code cleanup
* added changelog
* code cleanup
* disable security plugin and perf analyzer in matrix tests opensearch
* Fixed datanode jersey service
* Fixed opensearch instance env
* Opensearch security setup refactored
* code cleanup
* removed commented out code from datanodes JerseyService
* add fqdn to the http cert in the DatanodeSecuritySetupIT
* configure truststore only if the security is enabled
* code cleanup
* reintroduced root certificate finder
* adjusted class and methods visibility of RootCertificateFinder
* Start datanode jersey only after ssl configuration resolved, listen to change events
* Adapted DatanodeSecuritySetupIT to work with https version of the datanode interface
* simplify hostnames in DatanodeSecuritySetupIT
* code cleanup
* added changelog
* code cleanup
* disable security plugin and perf analyzer in matrix tests opensearch
* Fixed datanode jersey service
* Fixed opensearch instance env
* add fqdn to the http cert in the DatanodeSecuritySetupIT
* first version adding gl CA
* adding custom trutstore
* fixed SSLContext
* fixed SSLContext for connectivity check
* fix comments
* fixed imports
* reverting altnames tmp change
* reverting altnames tmp change
* fixed preflightconfig, fixed hostname-verifier
* added credentials to target string, set hostname
* added credentials from target string
* only add TrustManager if https is requested
* fixing tests
* add another alt name
* change creation of restBaseUrl
* DatanodeSecuritySetupIT adaptation for stable hostname of the datanode
* trying to fix naming/binding for docker
* adding another alt name
* added s to http
* changing exception
* more debug output
* change test
* change test
* adding debug
* switch back to http
* more debugging
* more debugging
* added exception to retryer
* fixed boolean eval
* going to the next step on FAILED, too
* fixing if clause
* cleanup
* cleanup
* cleanup
* cleanup
* cleanup
* cleanup
* cleanup
* cleanup
* removed wrong credentials provider
* added name verifying
* fix alt name handling
* removed unneccessary hostname verifier
* reverted DefaultX509TrustManager
* removed link to DefaultX509TrustManager because it's used in different contexts, singleton for SSL related stuff so we don't instantiate it all the time
* full rollback DefaultX509TrustManager
* reverted change on logging
* revert VersionProbe, unnecessary changes
* DatanodeSecuritySetupIT fixed, separated bind address and hostname
* changed to lambda
* changed to lambda
* small refactorings in csrsigner
* added test for CustomCAX509TrustManager
* added binding for CaService into Preflight
* fixing bindings
* fixing bindings
---------
Co-authored-by: Tomas Dvorak <tomas.dvorak@graylog.com>
* New configuration handling. Incomplete manual config ends with error. Empty config allows to try different config options.
* Trying to reflect functionality responsibilities with better naming
* Change ConfigProvider, so that it stops using PEM files, and starts using PKCS12 files.
Refactoring, splitting the functionality to different classes.
Some config changes that make things work in IDEA.
Reusable truststore.
Truststore stores only root certificates.
* Simpler root certificate finding
* Dir Sync does not remove cert files
* Removed unnecessary default config changes. Cleared confusing certificate copying and unnecessary certificate duplication in 2 folders
* Truststore path created outside of TruststoreCreator. Root certificate found in a different way, without considering its place in the certificate chain
* Fixed log message
* Integration test fixed
* Certs are not an exception in FullDirSync. Introduced second config location as a step towards final configuration. Certificates are moved between locations, as it was done previously
* Fixed a test. Minor refactoring.
* Config location has no default value. It is checked if it is provided, as it is required
* Synchronize opensearch config to a user-defined directory (fixes#15223)
* fixed check naming
* fixed opensearch configuration location
* simplified Environment constructor
* point gc logs of datanodes opensearch to /tmp for now
* full dir sync for opensearch config
* Fixed full dir sync when removing non-empty dir
* simpified recursive directory deletion
