Commit Graph

19 Commits

Author SHA1 Message Date
Jan Heise
65f60bd606 adding a validation method (#16840) 2023-10-06 14:44:42 +02:00
Jan Heise
38b9e9066b DataNode: Simplifying hostname usage Part 2 (#16741)
* simplifying hostname usage part2

* added suppress message

* removed comment
2023-09-29 08:09:33 +02:00
Jan Heise
15d3924130 using the machine's hostname as nodename if none was provided in the config (#16607) 2023-09-22 15:20:59 +02:00
Jan Heise
0364446c1b removing obsolete single-node setting (#16542) 2023-09-19 13:41:54 +02:00
Tomas Dvorak
70d6edc8a2 basic auth for datanode rest api if security enabled (#16446)
* basic auth for datanode rest api if security enabled

* added changelog

* adapted to master changes

* configurable rest admin username in datanode

---------

Co-authored-by: Jan Heise <jan.heise@graylog.com>
2023-09-15 20:52:14 +02:00
Jan Heise
cc52dcb5b0 DataNode: using JWT for authentication/authorisation (#16334)
* jwt implementation, adding preflight + versionprobe

* adding JWT auth to anomaly detection client, some minor refactorings

* added missing license

* added provider, refactorings

* fixing small bugs

* set smaller default expiration

* using the correct OpenSearch distro for the erchitecture

* added config options

* small refactorings, fixing tests, adding dependencies for running tests from inside IntelliJ

* fixing import to correct Duration class

* removing unnecessary debug output

* fixing tests

* fixing tests

* adding JWT to Cluster test

* fixing path

* fixing path problems

* fix architecture logic for DatanodeContainerizedBackend

* fixing path problems

* fixing JWT settings in tests

* fixing JWT settings in tests

* fixing cluster test

* fixing cluster test

* fixing cluster test

* adding missing jwt token

* small refactoring to force thread-safety on creation

* the datanode uses mongodb - so the closing sequence needs to be adjusted

* moving config changes into config file, keep only change that has to be done by code

* adding breaking change info for JWT and clock synchronization.

* renaming config settings

* renaming config settings

* removed debug settings that are no longer in use

* renamed provider/annotation to be more precise

* lowering log level

* smaller refactorings/renamings

* keep config change in code until PR is merged, remove code in a follow-up PR

* moved changelog file to correct folder

* clock synchronsation

Co-authored-by: Dennis Oelkers <dennis@graylog.com>

* added jwt auth into changes after merge

* upped the timeout

* log curl connection information in DatanodeOpensearchWait

* replacing with jwt auth

* changing wait strategy

* fixing imports

* remove uname/pwd from opensearch rest base url

* propagate jwt token provider to http rest client callback, obtain fresh token every time

* inject IndexerJwtAuthTokenProvider in DatanodeConfigurationProvider

* remove rest uname/pwd from datanode configuration

---------

Co-authored-by: Tomas Dvorak <tomas.dvorak@graylog.com>
Co-authored-by: Dennis Oelkers <dennis@graylog.com>
2023-09-12 11:48:44 +02:00
Jan Heise
af73ea391b DataNode: adding a default config location to avoid NPEs (#16381)
* adding a default

* fixing default paths

* fixing default paths
2023-09-06 11:07:02 +02:00
Tomas Dvorak
18e326b8a1 make cluster initial nodes in datanode configurable (#16146)
* make cluster initial nodes in datanode configurable

* added changelog

* removed forgotten debug logging

* returned opensearch_network_host config for containerized datanode tests
2023-08-10 14:29:48 +02:00
luk-kaminski
e7025a9da8 Insecure startup of OS in datanode only if explicitly configured (#16108)
* Insecure startup of OS in datanode only if explicitly configured

* Data node can be started without starting OS, just to be discovered by pre-flight mode
2023-08-07 17:21:08 +02:00
Jan Heise
34d5b47d2a Include Graylog CA into places where we connect to the DataNode (#15925)
* first version adding gl CA

* adding custom trutstore

* fixed SSLContext

* fixed SSLContext for connectivity check

* fix comments

* fixed imports

* reverting altnames tmp change

* reverting altnames tmp change

* fixed preflightconfig, fixed hostname-verifier

* added credentials to target string, set hostname

* added credentials from target string

* only add TrustManager if https is requested

* fixing tests

* add another alt name

* change creation of restBaseUrl

* DatanodeSecuritySetupIT adaptation for stable hostname of the datanode

* trying to fix naming/binding for docker

* adding another alt name

* added s to http

* changing exception

* more debug output

* change test

* change test

* adding debug

* switch back to http

* more debugging

* more debugging

* added exception to retryer

* fixed boolean eval

* going to the next step on FAILED, too

* fixing if clause

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* removed wrong credentials provider

* added name verifying

* fix alt name handling

* removed unneccessary hostname verifier

* reverted DefaultX509TrustManager

* removed link to DefaultX509TrustManager because it's used in different contexts, singleton for SSL related stuff so we don't instantiate it all the time

* full rollback DefaultX509TrustManager

* reverted change on logging

* revert VersionProbe, unnecessary changes

* DatanodeSecuritySetupIT fixed, separated bind address and hostname

* changed to lambda

* changed to lambda

* small refactorings in csrsigner

* added test for CustomCAX509TrustManager

* added binding for CaService into Preflight

* fixing bindings

* fixing bindings

---------

Co-authored-by: Tomas Dvorak <tomas.dvorak@graylog.com>
2023-07-21 12:59:55 +02:00
luk-kaminski
a8f29b0431 New configuration handling. Incomplete manual config ends with error.… (#15784)
* New configuration handling. Incomplete manual config ends with error. Empty config allows to try different config options.

* Trying to reflect functionality responsibilities with better naming
2023-07-04 13:56:22 +02:00
Tomas Dvorak
bc56d397c3 Datanode config separation (#15701)
* Different variants of security configuration (from file, from mongo, insecure).
Encryption/decryption pattern used in practice.

* Reverting temporary changes

* More precise certificate presence check

* Added unit tests for testHasCert() method

* Datanode and opensearch configuration refactored

* code cleanup

* code cleanup

* code cleanup

* Node preflight config service refactored

* restbaseurl moved to OpensearchConfiguration

---------

Co-authored-by: Lukasz Kaminski <lukasz.kaminski@graylog.com>
2023-06-06 14:44:06 +02:00
luk-kaminski
f9dc221627 Change ConfigProvider, so that it stops using PEM files, and starts u… (#15460)
* Change ConfigProvider, so that it stops using PEM files, and starts using PKCS12 files.
Refactoring, splitting the functionality to different classes.
Some config changes that make things work in IDEA.
Reusable truststore.
Truststore stores only root certificates.

* Simpler root certificate finding

* Dir Sync does not remove cert files

* Removed unnecessary default config changes. Cleared confusing certificate copying and unnecessary certificate duplication in 2 folders

* Truststore path created outside of TruststoreCreator. Root certificate found in a different way, without considering its place in the certificate chain

* Fixed log message

* Integration test fixed

* Certs are not an exception in FullDirSync. Introduced second config location as a step towards final configuration. Certificates are moved between locations, as it was done previously

* Fixed a test. Minor refactoring.

* Config location has no default value. It is checked if it is provided, as it is required
2023-05-16 08:38:03 +02:00
Tomas Dvorak
c36a63495d Opensearch dist dir autodetection (#15216)
* Opensearch dist dir autodetection

* Add arm64 as supported architecture for datanode

* Added changelog
2023-04-14 09:53:33 +02:00
Tomas Dvorak
8501714ee8 Update opensearch in datanode to 2.5.0 (#15198)
* Update opensearch in datanode to 2.5.0

* Added changelog
2023-04-13 08:21:14 +02:00
Tomas Dvorak
a84cad55a0 Adapted default configuration, fixed logging, extended datanode.conf template (#15056)
Co-authored-by: Bernd Ahlers <bernd@users.noreply.github.com>
2023-04-06 17:48:46 +02:00
Jan Heise
e4db603eb6 DataNode: config option to enforce single-node type (#15070)
* adds a config option to force single-node type so that (at least on macOS) the bootstrap checks won't run and fail

* add changelog

* reworded message
2023-03-28 14:30:10 +02:00
Bernd Ahlers
120fae5221 Refactor data-node download and assembly (#14963)
* Refactor data-node download and assembly

- Download OpenSearch aarch64 distribution in addition to x64
- Extract OpenSearch distribution into "target/opensearch/*" instead of
  "bin/download".
- Include OpenSearch distributions in "dist/" instead of "bin/" in
  assembly

* Fix data-node test container creation

* Adjust OpenSearch location for architecture dependent locations

* If the configured location exists, use it

Otherwise, check for architecture dependent distribution directories.

* Switch default config file to locations that work with the tarball

* Rename datanode.jar to graylog-datanode.jar

* Rename datanodectl to graylog-datanode.sh

* Include graylog-datanode.sh in assembly as bin/graylog-datanode

* Add a jvm.options config file

* Add a production log4j2.xml config file

* Use bundled JVM if it exists

* fixed docker image for datanode integration tests

---------

Co-authored-by: Tomas Dvorak <tomas.dvorak@graylog.com>
2023-03-28 10:49:49 +02:00
Tomas Dvorak
21e55cbcb6 Initial Data Node version PR (#14417)
Co-authored-by: Jan Heise <jan.heise@graylog.com>
2023-02-27 13:11:24 +01:00