How to deploy elasticsearch with docker swarm?

Question

I create 3 virtual machine use docker-machine,there are:

NAME       ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER     ERRORS
cluster    -        virtualbox   Running   tcp://192.168.99.101:2376           v18.09.5   
cluster2   -        virtualbox   Running   tcp://192.168.99.102:2376           v18.09.5   
master     -        virtualbox   Running   tcp://192.168.99.100:2376           v18.09.5 

and then I create a docker swarm in master machine:

 docker-machine ssh master "docker swarm init ----advertise-addr 192.168.99.100"

and in cluster and cluster2 join master:

docker-machine ssh cluster "docker swarm join --advertise-addr 192.168.99.101 --token xxxx 192.168.99.100:2377"

docker-machine ssh cluster2 "docker swarm join --advertise-addr 192.168.99.102 --token xxxx 192.168.99.100:2377"

the docker node ls info:

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
r4a6y9wie4zp3pl4wi4e6wqp8     cluster             Ready               Active                                  18.09.5
sg9gq6s3k6vty7qap7co6eppn     cluster2            Ready               Active                                  18.09.5
xb6telu8cn3bfmume1kcektkt *   master              Ready               Active              Leader              18.09.5

there is deploy config swarm.yml:

version: "3.3"

services:
  elasticsearch:
    image: elasticsearch:7.0.0
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      - cluster.name=elk
      - network.host=_eth1:ipv4_
      - network.bind_host=_eth1:ipv4_
      - network.publish_host=_eth1:ipv4_
      - discovery.seed_hosts=192.168.99.100,192.168.99.101
      - cluster.initial_master_nodes=192.168.99.100,192.168.99.101
      - bootstrap.memory_lock=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    networks:
      - backend
    deploy:
      mode: replicated
      replicas: 3
      #endpoint_mode: dnsrr
      restart_policy:
        condition: none
      resources:
        limits:
          cpus: "1.0"
          memory: "1024M"
        reservations:
          memory: 20M
networks:
  backend:
    # driver: overlay
    # attachable: true

i pull elasticsearch image to virtual machie:

docker-machine ssh master "docker image pull elasticsearch:7.0.0"
docker-machine ssh cluster "docker image pull elasticsearch:7.0.0"
docker-machine ssh cluster2 "docker image pull elasticsearch:7.0.0"

before run i run this command fix some elasticearch bootstrap error:

docker-machine ssh master "sudo sysctl -w vm.max_map_count=262144"
docker-machine ssh cluster "sudo sysctl -w vm.max_map_count=262144"
docker-machine ssh cluster2 "sudo sysctl -w vm.max_map_count=262144"

and then i run `docker stack deploy -c swarm.yml es, the elasticsearch cluster cannot work.

docker-machine ssh master
docker service logs es_elasticsearch -f 

show:

es_elasticsearch.1.uh1x0s9qr7mb@cluster    | {"type": "server", "timestamp": "2019-04-25T16:28:47,143+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elk", "node.name": "e8dba5562417",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [192.168.99.100, 192.168.99.101] to bootstrap a cluster: have discovered []; discovery will continue using [192.168.99.100:9300, 192.168.99.101:9300] from hosts providers and [{e8dba5562417}{Jy3t0AAkSW-jY-IygOCjOQ}{z7MYIf5wTfOhCX1r25wNPg}{10.255.0.46}{10.255.0.46:9300}{ml.machine_memory=1037410304, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }
es_elasticsearch.2.swswlwmle9e9@cluster2    | {"type": "server", "timestamp": "2019-04-25T16:28:47,389+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elk", "node.name": "af5d88a04b42",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [192.168.99.100, 192.168.99.101] to bootstrap a cluster: have discovered []; discovery will continue using [192.168.99.100:9300, 192.168.99.101:9300] from hosts providers and [{af5d88a04b42}{zhxMeNMAQN2evKDlsA33qA}{fpYPTvJ6STmyqrgxlMkD_w}{10.255.0.47}{10.255.0.47:9300}{ml.machine_memory=1037410304, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }
es_elasticsearch.3.x8ouukovhh80@master    | {"type": "server", "timestamp": "2019-04-25T16:28:48,818+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elk", "node.name": "0e7e4d96b31a",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [192.168.99.100, 192.168.99.101] to bootstrap a cluster: have discovered []; discovery will continue using [192.168.99.100:9300, 192.168.99.101:9300] from hosts providers and [{0e7e4d96b31a}{Xs9966RjTEWvEbuj4-ySYA}{-eV4lvavSHq6JhoW0qWu6A}{10.255.0.48}{10.255.0.48:9300}{ml.machine_memory=1037410304, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }

I guess the cluster formation failed may be due to network configuration error. I don't know how to fix it, I try many times modify the config, fail and fail again.

Answer 1

try, this is working :) docker-compose.yml

version: "3.7"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
    hostname: "{{.Node.Hostname}}"
    environment:
      - node.name={{.Node.Hostname}}
      - cluster.name=my-cluster
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
      - discovery.seed_hosts=elasticsearch
      - cluster.initial_master_nodes=node1,node2,node3
      - node.ml=false
      - xpack.ml.enabled=false
      - xpack.monitoring.enabled=false
      - xpack.security.enabled=false
      - xpack.watcher.enabled=false
      - bootstrap.memory_lock=false
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    deploy:
      mode: global
      endpoint_mode: dnsrr
      resources:
        limits:
          memory: 4G
  nginx:
    image: nginx:1.17.1-alpine
    ports:
      - 9200:9200
    deploy:
      mode: global
    command: |
      /bin/sh -c "echo '
      user nobody nogroup;
      worker_processes auto;
      events {
        worker_connections 1024;
      }
      http {
        client_max_body_size 4g;
        resolver 127.0.0.11 ipv6=off;
        server {
          listen *:9200;
          location / {
            proxy_set_header Connection keep-alive;
            set $$url http://elasticsearch:9200;
            proxy_pass $$url;
            proxy_set_header  Host $$http_host;
            proxy_set_header  X-Real-IP $$remote_addr;
            proxy_set_header  X-Forwarded-For $$proxy_add_x_forwarded_for;
          }
        }
      }' | tee /etc/nginx/nginx.conf && nginx -t && nginx -g 'daemon off;'"

volumes:
  elasticsearch-data:

Answer 2

Trying to manually specify all the specific IP's and bindings is tricky because of the swarm overlaying network. Instead, simply make your ES nodes discoverable and let Swarm take care of the node discovery and communication. To make them discoverable, we can use a predictable name like the Swarm node hostname.

Try change your environment settings in the swarm.yml file as follows:

    environment:
    - network.host=0.0.0.0
    - discovery.seed_hosts=elasticsearch #Service name, to let Swarm handle discovery
    - cluster.initial_master_nodes=master,cluster,cluster2 #Swarm nodes host names
    - node.name={{.Node.Hostname}} #To create a predictable node name

This of course assumes that we already known the swarm hostnames, which you pointed out in the screenshot above. Without knowing these values, we would have no way of having a predictable set of node names to look for. In that case, you could create 1 ES node entry with a particular node name, and then another entry which references the first entry's node name as the cluster.initial_master_nodes.

Answer 3

In my experience https://github.com/shazChaudhry/docker-elastic works perfectly, and just one file from the entire repo is enough. I downloaded https://github.com/shazChaudhry/docker-elastic/blob/master/docker-compose.yml and removed the logstash bits, I didn't need that. Then added the following to .bashrc

export ELASTICSEARCH_HOST=$(hostname)
export ELASTICSEARCH_PASSWORD=foobar
export ELASTICSEARCH_USERNAME=elastic
export ELASTIC_VERSION=7.4.2
export INITIAL_MASTER_NODES=$ELASTICSEARCH_HOST

And docker stack deploy --compose-file docker-compose.yml elastic works.

Answer 4

Use dnsrr mode without ports. Expose elasticsearch with nginx ;) See my docker-compose.yml

Answer 5

Ideas I gleaned from Ahmet Vehbi Olgaç 's docker-compose.yml, which worked for me:

1. Use deployment / mode: global. This will cause the swarm to deploy one replica to each swarm worker, for each node that is configured like this.

2. Use deployment / endpoint_mode: dnsrr. This will let all containers in the swarm access the nodes by the service name.

3. Use hostname: {{.Node.Hostname}} or a similar template-based expression. This ensures a unique name for each deployed container.

4. Use environment / node.name={{.Node.Hostname}}. Again, you can vary the pattern. The point is that each es node should get a unique name.

5. Use cluster.initial_master_nodes=*hostname1*,*hostname2*,.... Assuming you know the hostnames of your docker worker machines. Use whatever pattern you used in #3, but substitute out the whole hostname, and include all the hostnames.
   
   If you don't know your hostnames, you can do what Andrew Cachia's answer suggests: set up one container (do not replicate it) to act solely as the master seed and give it a predictable hostname, then have all other nodes refer to that node as the master seed. However, this introduces a single point of failure.