diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6d81b58..8c42731 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -67,7 +67,7 @@ jobs: sed -i -e 's/\(xpack.monitoring.elasticsearch.username:\) elastic/\1 logstash_system/g' -e 's/\(xpack.monitoring.elasticsearch.password:\) changeme/\1 testpasswd/g' logstash/config/logstash.yml sed -i 's/\(password =>\) "changeme"/\1 "testpasswd"/g' logstash/pipeline/logstash.conf - sed -i -e 's/\(elasticsearch.username:\) elastic/\1 kibana_system/g' -e 's/\(elasticsearch.password:\) changeme/\1 testpasswd/g' kibana/config/kibana.yml + sed -i 's/\(elasticsearch.password:\) changeme/\1 testpasswd/g' kibana/config/kibana.yml sed -i -e 's/\(elasticsearch.password:\) changeme/\1 testpasswd/g' -e 's/\(secret_management.encryption_keys:\)/\1 [test-encrypt]/g' extensions/enterprise-search/config/enterprise-search.yml sed -i 's/\(password:\) changeme/\1 testpasswd/g' extensions/apm-server/config/apm-server.yml sed -i 's/\(password:\) changeme/\1 testpasswd/g' extensions/metricbeat/config/metricbeat.yml @@ -288,7 +288,7 @@ jobs: sed -i -e 's/\(xpack.monitoring.elasticsearch.username:\) elastic/\1 logstash_system/g' -e 's/\(xpack.monitoring.elasticsearch.password:\) changeme/\1 testpasswd/g' logstash/config/logstash.yml sed -i 's/\(password =>\) "changeme"/\1 "testpasswd"/g' logstash/pipeline/logstash.conf - sed -i -e 's/\(elasticsearch.username:\) elastic/\1 kibana_system/g' -e 's/\(elasticsearch.password:\) changeme/\1 testpasswd/g' kibana/config/kibana.yml + sed -i 's/\(elasticsearch.password:\) changeme/\1 testpasswd/g' kibana/config/kibana.yml # Run Elasticsearch and wait for its availability diff --git a/README.md b/README.md index 1860a11..088d76b 100644 --- a/README.md +++ b/README.md @@ -48,12 +48,11 @@ own_. [sherifabdlnaby/elastdocker][elastdocker] is one example among others of p * [Windows](#windows) * [macOS](#macos) 1. [Usage](#usage) - * [Version selection](#version-selection) - * [Bringing up the stack](#bringing-up-the-stack) - * [Cleanup](#cleanup) * [Initial setup](#initial-setup) * [Setting up user authentication](#setting-up-user-authentication) * [Injecting data](#injecting-data) + * [Cleanup](#cleanup) + * [Version selection](#version-selection) 1. [Configuration](#configuration) * [How to configure Elasticsearch](#how-to-configure-elasticsearch) * [How to configure Kibana](#how-to-configure-kibana) @@ -110,65 +109,40 @@ instructions from the [documentation][mac-filesharing] to add more locations. ## Usage -### Version selection - -This repository tries to stay aligned with the latest version of the Elastic stack. The `main` branch tracks the current -major version (8.x). - -To use a different version of the core Elastic components, simply change the version number inside the `.env` file. If -you are upgrading an existing stack, please carefully read the note in the next section. - -**:warning: Always pay attention to the [official upgrade instructions][upgrade] for each individual component before -performing a stack upgrade.** - -Older major versions are also supported on separate branches: - -* [`release-7.x`](https://github.com/deviantony/docker-elk/tree/release-7.x): 7.x series -* [`release-6.x`](https://github.com/deviantony/docker-elk/tree/release-6.x): 6.x series (End-of-life) -* [`release-5.x`](https://github.com/deviantony/docker-elk/tree/release-5.x): 5.x series (End-of-life) - -### Bringing up the stack - -Clone this repository onto the Docker host that will run the stack, then start services locally using Docker Compose: - -```console -$ docker-compose up -``` - -You can also run all services in the background (detached mode) by adding the `-d` flag to the above command. - **:warning: You must rebuild the stack images with `docker-compose build` whenever you switch branch or update the -version of an already existing stack.** +[version](#version-selection) of an already existing stack.** -If you are starting the stack for the very first time, please read the section below attentively. +### Initial setup -### Cleanup - -Elasticsearch data is persisted inside a volume by default. - -In order to entirely shutdown the stack and remove all persisted data, use the following Docker Compose command: +Clone this repository onto the Docker host that will run the stack, then start the Elasticsearch service locally using +Docker Compose: ```console -$ docker-compose down -v +$ docker-compose up elasticsearch -d ``` -## Initial setup +We will start the rest of the Elastic components _after_ completing the initial setup described in this section. These +steps only need to be performed _once_. -### Setting up user authentication +**:warning: Starting with Elastic v8.0.0, it is no longer possible to run Kibana using the bootstraped privileged +`elastic` user. If you are starting the stack for the very first time, you MUST initialize a password for the [built-in +`kibana_system` user][builtin-users] to be able to start and access Kibana. Please read the section below attentively.** -*:information_source: Refer to [How to disable paid features](#how-to-disable-paid-features) to disable authentication.* +#### Setting up user authentication + +*:information_source: Refer to [Security settings in Elasticsearch][es-security] to disable authentication.* The stack is pre-configured with the following **privileged** bootstrap user: * user: *elastic* * password: *changeme* -Although all stack components work out-of-the-box with this user, we strongly recommend using the unprivileged [built-in -users][builtin-users] instead for increased security. +For increased security, we will reset this bootstrap password, and generate a set of passwords to be used by +unprivileged [built-in users][builtin-users] within components of the Elastic stack. 1. Initialize passwords for built-in users - The commands below generate random passwords for all 6 built-in users. Take note of them. + The commands below generate random passwords for the `elastic` and `kibana_system` users. Take note of them. ```console $ docker-compose exec -T elasticsearch bin/elasticsearch-reset-password --batch --user elastic @@ -178,33 +152,17 @@ users][builtin-users] instead for increased security. $ docker-compose exec -T elasticsearch bin/elasticsearch-reset-password --batch --user kibana_system ``` - ```console - $ docker-compose exec -T elasticsearch bin/elasticsearch-reset-password --batch --user logstash_system - ``` - - ```console - $ docker-compose exec -T elasticsearch bin/elasticsearch-reset-password --batch --user beats_system - ``` - - ```console - $ docker-compose exec -T elasticsearch bin/elasticsearch-reset-password --batch --user apm_system - ``` - - ```console - $ docker-compose exec -T elasticsearch bin/elasticsearch-reset-password --batch --user remote_monitoring_user - ``` - -1. Unset the bootstrap password (_optional_) - - Remove the `ELASTIC_PASSWORD` environment variable from the `elasticsearch` service inside the Compose file - (`docker-compose.yml`). It is only used to initialize the keystore during the initial startup of Elasticsearch. + If the need for it arises (e.g. if you want to [collect monitoring information][ls-monitoring] through Beats and + other components), feel free to repeat this operation at any time for the rest of the [built-in + users][builtin-users]. 1. Replace usernames and passwords in configuration files - Use the `kibana_system` user inside the Kibana configuration file (`kibana/config/kibana.yml`) in place of the - existing `elastic` user. + Replace the password of the `kibana_system` user inside the Kibana configuration file (`kibana/config/kibana.yml`) + with the password generated in the previous step. - Replace the password for the `elastic` user inside the Logstash pipeline file (`logstash/pipeline/logstash.conf`). + Replace the password of the `elastic` user inside the Logstash pipeline file (`logstash/pipeline/logstash.conf`) + with the password generated in the previous step. *:information_source: Do not use the `logstash_system` user inside the Logstash **pipeline** file, it does not have sufficient permissions to create indices. Follow the instructions at [Configuring Security in Logstash][ls-security] @@ -212,16 +170,27 @@ users][builtin-users] instead for increased security. See also the [Configuration](#configuration) section below. -1. Restart Kibana and Logstash to apply changes +1. Unset the bootstrap password (_optional_) + + Remove the `ELASTIC_PASSWORD` environment variable from the `elasticsearch` service inside the Compose file + (`docker-compose.yml`). It is only used to initialize the keystore during the initial startup of Elasticsearch, and + is ignored on subsequent runs. + +1. Start Kibana and Logstash ```console - $ docker-compose restart kibana logstash + $ docker-compose up -d ``` + The `-d` flag runs all services in the background (detached mode). + + On subsequent runs of the Elastic stack, it is sufficient to execute the above command in order to start all + components. + *:information_source: Learn more about the security of the Elastic stack at [Secure the Elastic Stack][sec-cluster].* -### Injecting data +#### Injecting data Give Kibana about a minute to initialize, then access the Kibana web UI by opening in a web browser and use the following credentials to log in: @@ -244,6 +213,33 @@ $ cat /path/to/logfile.log | nc -c localhost 5000 You can also load the sample data provided by your Kibana installation. +### Cleanup + +Elasticsearch data is persisted inside a volume by default. + +In order to entirely shutdown the stack and remove all persisted data, use the following Docker Compose command: + +```console +$ docker-compose down -v +``` + +### Version selection + +This repository stays aligned with the latest version of the Elastic stack. The `main` branch tracks the current major +version (8.x). + +To use a different version of the core Elastic components, simply change the version number inside the `.env` file. If +you are upgrading an existing stack, please carefully read the note in the next section. + +**:warning: Always pay attention to the [official upgrade instructions][upgrade] for each individual component before +performing a stack upgrade.** + +Older major versions are also supported on separate branches: + +* [`release-7.x`](https://github.com/deviantony/docker-elk/tree/release-7.x): 7.x series +* [`release-6.x`](https://github.com/deviantony/docker-elk/tree/release-6.x): 6.x series (End-of-life) +* [`release-5.x`](https://github.com/deviantony/docker-elk/tree/release-5.x): 5.x series (End-of-life) + ## Configuration *:information_source: Configuration is not dynamically reloaded, you will need to restart individual components after @@ -404,6 +400,7 @@ instead of `elasticsearch`.* [elk-stack]: https://www.elastic.co/what-is/elk-stack [xpack]: https://www.elastic.co/what-is/open-x-pack [paid-features]: https://www.elastic.co/subscriptions +[es-security]: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html [trial-license]: https://www.elastic.co/guide/en/elasticsearch/reference/current/license-settings.html [license-mngmt]: https://www.elastic.co/guide/en/kibana/current/managing-licenses.html [license-apis]: https://www.elastic.co/guide/en/elasticsearch/reference/current/licensing-apis.html @@ -420,6 +417,7 @@ instead of `elasticsearch`.* [builtin-users]: https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-users.html [ls-security]: https://www.elastic.co/guide/en/logstash/current/ls-security.html +[ls-monitoring]: https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html [sec-cluster]: https://www.elastic.co/guide/en/elasticsearch/reference/current/secure-cluster.html [connect-kibana]: https://www.elastic.co/guide/en/kibana/current/connect-to-elasticsearch.html diff --git a/kibana/config/kibana.yml b/kibana/config/kibana.yml index 0e1dc60..456b3f7 100644 --- a/kibana/config/kibana.yml +++ b/kibana/config/kibana.yml @@ -9,5 +9,5 @@ monitoring.ui.container.elasticsearch.enabled: true ## X-Pack security credentials # -elasticsearch.username: elastic +elasticsearch.username: kibana_system elasticsearch.password: changeme