ES学习

发布日期: 2020-12-22

作者: 阿钟

文章字数: 3.2k

阅读时长: 16 分

阅读次数:

Elasticsearch 的安装与简单配置

¶依赖环境

运行 ES 需要安装并配置 Java
- 配置$JAVA_HOME
各个版本对 Java 的依赖
- ES5需要 Java8以上的版本
- ES 从6.5开始支持 Java11
- http://www.elastic.co/support/matrix#matrix_jvm
- 7.0开始，内置了 Java 环境

¶下载安装

¶1、下载

我们可以到Elasticsearch 的官方下载地址下载需要的软件，也可以通过包管理器安装的方式或者 docker 容器启动：

¶2、安装

这里我选择下载 linux 版本然后上传到 aliyun 服务器

[root@izwz920kp0myp15p982vp4z ~]# cd 
/usr/local/software/elasticsearch/
[root@izwz920kp0myp15p982vp4z elasticsearch]# ll
总用量 534476
-rw-r--r-- 1 root root 296477546 4月  23 17:00 elasticsearch-7.6.2-linux-x86_64.tar.gz
-rw-r--r-- 1 root root 249555386 3月  31 23:38 kibana-7.6.2-linux-x86_64.tar.gz

解压到当前目录

[root@izwz920kp0myp15p982vp4z elasticsearch]# tar -zxvf elasticsearch-7.6.2-linux-x86_64.tar.gz
[root@izwz920kp0myp15p982vp4z elasticsearch]# cd elasticsearch-7.6.2
[root@izwz920kp0myp15p982vp4z elasticsearch-7.6.2]# ll
总用量 564
drwxr-xr-x  2 root root   4096 3月  26 14:36 bin
drwxr-xr-x  2 root root   4096 3月  26 14:36 config
drwxr-xr-x  9 root root   4096 3月  26 14:36 jdk
drwxr-xr-x  3 root root   4096 3月  26 14:36 lib
-rw-r--r--  1 root root  13675 3月  26 14:28 LICENSE.txt
drwxr-xr-x  2 root root   4096 3月  26 14:36 logs
drwxr-xr-x 38 root root   4096 3月  26 14:37 modules
-rw-r--r--  1 root root 523209 3月  26 14:36 NOTICE.txt
drwxr-xr-x  2 root root   4096 3月  26 14:36 plugins
-rw-r--r--  1 root root   8164 3月  26 14:28 README.asciidoc

进入解压后的 ES 目录，可以看到以下目录结构：

目录	相关文件	描述
bin		所有运行的脚本文件，包括启动 ES，安装插件。运行统计数据等
config	如 elasticsearch.yml 等	集群配置文件，user，role based 相关配置
JDK		Java 运行环境。从7.0开始 ES 自动集成 JDK 到安装文件中
data	path.data	包含了 ES 所有的相关数据文件
lib		Java 类库
logs	path.log	日志文件
modules		包含所有 ES 模块
plugins		包含所有已安装插件

¶3、JVM 配置

修改 JVM，在 option/jvm.options
- 7.1 下载的默认设置时1GB
配置的建议
- Xmx 和 Xms设置成一样
- Xmx 不要超过机器内存的50%
- 不要超过30GB - https://www.elastic.co/blog/a-heap-of-touble

¶4、尝试启动 ES

[root@izwz920kp0myp15p982vp4z elasticsearch-7.6.2]# bin/elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2020-04-23T17:47:45,074][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [izwz920kp0myp15p982vp4z] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
	at ... ...

启动失败。看到 es 不能以 root 用户启动。

¶5、创建新用户

创建用户elasticsearch输入，新的密码zhonghongpeng并赋予权限

[root@izwz920kp0myp15p982vp4z elasticsearch-7.6.2]# adduser elasticsearch
[root@izwz920kp0myp15p982vp4z elasticsearch-7.6.2]# passwd elasticsearch
更改用户 elasticsearch 的密码 。
新的 密码：
重新输入新的 密码：
passwd：所有的身份验证令牌已经成功更新。
[root@izwz920kp0myp15p982vp4z elasticsearch-7.6.2]#

¶6、用户内存过小问题

切换用户再次启动，还是遇到问题max virtual memory areas vm.max_map_count [65530] is too low用户虚拟内存太小

[root@izwz920kp0myp15p982vp4z elasticsearch-7.6.2]# su elasticsearch
[elasticsearch@izwz920kp0myp15p982vp4z elasticsearch-7.6.2]$ bin/elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2020-04-23T17:53:32,277][INFO ][o.e.e.NodeEnvironment    ] [izwz920kp0myp15p982vp4z] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [32.3gb], net total_space [39.2gb], types [rootfs]
... ... 
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
... ...

再次切换回 root 用户，并执行以下执行进行修改，然后编辑/etc/sysctl.conf在最后一行添加vm.max_map_count = 262144保证永久生效

[elasticsearch@izwz920kp0myp15p982vp4z config]$ su - root
密码：
上一次登录：四 4月 23 17:55:21 CST 2020从 219.137.74.3pts/0 上
[root@izwz920kp0myp15p982vp4z ~]# sysctl -a|grep vm.max_map_count
vm.max_map_count = 65530
[root@izwz920kp0myp15p982vp4z ~]# sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144
[root@izwz920kp0myp15p982vp4z ~]# sysctl -a|grep vm.max_map_count
vm.max_map_count = 262144
[root@izwz920kp0myp15p982vp4z ~]# vi /etc/sysctl.conf

切换回elasticsearch用户再次启动，启动成功！

[root@izwz920kp0myp15p982vp4z ~]# su elasticsearch
[elasticsearch@izwz920kp0myp15p982vp4z elasticsearch]$ cd /usr/local/software/elasticsearch/elasticsearch-7.6.2
[elasticsearch@izwz920kp0myp15p982vp4z elasticsearch-7.6.2]$ ./bin/elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2020-04-23T18:32:39,304][INFO ][o.e.e.NodeEnvironment    ] [izwz920kp0myp15p982vp4z] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [32.3gb], net total_space [39.2gb], types [rootfs]
[2020-04-23T18:32:39,309][INFO ][o.e.e.NodeEnvironment    ] [izwz920kp0myp15p982vp4z] heap size [1007.3mb], compressed ordinary object pointers [true]
[2020-04-23T18:32:39,461][INFO ][o.e.n.Node               ] [izwz920kp0myp15p982vp4z] node name [izwz920kp0myp15p982vp4z], node ID [AZUJJ0XhR0qwGYFxZ0O0xQ], cluster name [elasticsearch]
[2020-04-23T18:32:39,462][INFO ][o.e.n.Node               ] [izwz920kp0myp15p982vp4z] version[7.6.2], pid[17383], build[default/tar/ef48eb35cf30adf4db14086e8aabd07ef6fb113f/2020-03-26T06:34:37.794943Z], OS[Linux/3.10.0-693.2.2.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.2/13.0.2+8]
[2020-04-23T18:32:39,462][INFO ][o.e.n.Node               ] [izwz920kp0myp15p982vp4z] JVM home [/usr/local/software/elasticsearch/elasticsearch-7.6.2/jdk]
[2020-04-23T18:32:39,463][INFO ][o.e.n.Node               ] [izwz920kp0myp15p982vp4z] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-10878863762573310172, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/local/software/elasticsearch/elasticsearch-7.6.2, -Des.path.conf=/usr/local/software/elasticsearch/elasticsearch-7.6.2/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]

在 ssh 终端中测试连接，成功！

[root@izwz920kp0myp15p982vp4z config]# curl localhost:9200
{
  "name" : "izwz920kp0myp15p982vp4z",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "0lHe6Dh7TOKSTX2Jy6PDSw",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

¶7、默认监听接口问题

但是在本地电脑的浏览器中访问不成功：（myecs.com 配置本地 hosts 指向了 aliyun ecs ip）

在本地电脑直接telnet 也连不上，一直无响应：

zhonghongpeng@bogon ~ % telnet 120.24.80.237 9200

看了一下 ES 输出的日志，发现它在监听的接口是127.0.0.1:9200

[2020-04-23T18:23:33,314][INFO ][o.e.t.TransportService   ] [izwz920kp0myp15p982vp4z] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}

修改配置文件./config/elasticsearch.yml

network.host: 0.0.0.0
discovery.seed_hosts: ["0.0.0.0", "[::1]"]

再次启动，绑定接口为0.0.0.0:9200：

[2020-04-23T18:23:33,314][INFO ][o.e.t.TransportService   ] [izwz920kp0myp15p982vp4z] publish_address {172.18.93.184:9200}, bound_addresses {0.0.0.0:9200}

回到本地电脑浏览器测试，成功！

¶插件安装

elasticsearch-plugin list命令可以查看插件列表

[elasticsearch@izwz920kp0myp15p982vp4z elasticsearch-7.6.2]$ bin/elasticsearch-plugin list
[elasticsearch@izwz920kp0myp15p982vp4z elasticsearch-7.6.2]$

install命令可以安装插件，我们来安装一个国际化的分词插件analysis-icu

[elasticsearch@izwz920kp0myp15p982vp4z elasticsearch-7.6.2]$ ./bin/elasticsearch-plugin install analysis-icu
-> Installing analysis-icu
-> Downloading analysis-icu from elastic
[=================================================] 100%
-> Installed analysis-icu
[elasticsearch@izwz920kp0myp15p982vp4z elasticsearch-7.6.2]$ ./bin/elasticsearch-plugin list
analysis-icu

通过浏览器输入http://myecs.com:9200/_cat/plugins也可以显示插件列表：

通过插件机制用户可以自定义拓展其功能：

Discovery Plugin
Analysis Plugin
Security Plugin
Management Plugin
Ingest Plugin
Mapper Plugin
Backup Plugin

Https://elastic.co/guide/en/elasticsearch/plugins/current/intro.html

¶启动集群

elasticsearch -E node.name=节点名称 -E cluster.name=集群名称 -E path.data=节点数据文件名称 -d，其中-E通过 name=value 形式配置参数，-d后台启动，启动之前我们需要配置./config/elasticsearch.yml以下配置包含我们需要启动的节点名称

cluster.initial_master_nodes: ["node1", "node2", "node3"]

然后启动

bin/elasticsearch -E node.name=node1 -E cluster.name=john -E path.data=node1_data -d
bin/elasticsearch -E node.name=node2 -E cluster.name=john -E path.data=node2_data -d
bin/elasticsearch -E node.name=node3 -E cluster.name=john -E path.data=node3_data -d

浏览器访问http://myecs.com:9200/_cat/nodes，可以看到我们启动集群了

Kibana 的安装

¶安装

同 Elasticsearch 下载好安装文件，并解压（注意，如果用 root 用户进行解压，需要对解压之后的文件夹进行 chmod 授权，或者直接用 elasticsearch 用户进行解压，不然elasticsearch 用户启动 kibana 的时候就会因为 kibana需要写入一些文件没有权限导致启动失败）

[root@izwz920kp0myp15p982vp4z elasticsearch]# tar -zxvf kibana-7.6.2-linux-x86_64.tar.gz

修改配置${kibana_home}/config/kibana.yml：

server.host: "0.0.0.0"

如果需要汉化需要配置i18n.locale: "zh-CN"

启动

[elasticsearch@izwz920kp0myp15p982vp4z kibana-7.6.2-linux-x86_64]$ ./bin/kibana
  log   [12:07:17.644] [info][plugins-service] Plugin "case" is disabled.
  log   [12:07:22.407] [info][plugins-system] Setting up [37] plugins: [taskManager,siem,licensing,infra,encryptedSavedObjects,code,usageCollection,metrics,canvas,timelion,features,security,apm_oss,translations,reporting,uiActions,data,navigation,status_page,share,newsfeed,kibana_legacy,management,dev_tools,inspector,expressions,visualizations,embeddable,advancedUiActions,dashboard_embeddable_container,home,spaces,cloud,apm,graph,eui_utils,bfetch]
  log   [12:07:22.408] [info][plugins][taskManager] Setting up plugin
  log   [12:07:22.422] [info][plugins][siem] Setting up plugin
  log   [12:07:22.423] [info][licensing][plugins] Setting up plugin
  log   [12:07:22.425] [info][infra][plugins] Setting up plugin
  log   [12:07:22.426] [info][encryptedSavedObjects][plugins] Setting up plugin
  log   [12:07:22.427] [warning][config][encryptedSavedObjects][plugins] Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml
  log   [12:07:22.431] [info][code][plugins] Setting up plugin
  log   [12:07:22.432] [info][plugins][usageCollection] Setting up plugin
  log   [12:07:22.434] [info][metrics][plugins] Setting up plugin
  log   [12:07:22.434] [info][canvas][plugins] Setting up plugin
  ... ...

访问myecs.com:5601

启动成功！

¶一些样例数据

ES 在 Kibana 的开箱即64用的版本中加入了一些样例数据：电商网站订单、航空公司飞行记录、web 应用的日志。

我们可以点击添加这些数据来打入这些数据到 Elasticsearch 中

asd

通过点击右边的 dashboard 可以发现这三份数据的 dashboard 已经构建好了，我们点击其中一个

进来后看到一些数据面板

¶Dev Tools

下面我们来看左边面板的一个 Dev Tools 工具

它可以方便我们进行 Elasticsearch 的 api 的调用

另外它还提供了一些键盘快捷操作：

¶插件

Docker 的安装

¶相关阅读与安装

安装docker

安装 docker-compose

¶docker-compose命令

启动docker-compose up
停止容器docker-compose down
停止容器并且移除数据docker-compose down -v

¶docker 命令

docker ps
docker stop Name/ContainerId
docker start Name/ContainerId
删除单个容器 docker rm Name/ID -f, –force=false; -l, –link=false Remove the specified link and not the underlying container; -v, –volumes=false Remove the volumes associated to the container
删除所有容器 docker rm docker ps -a -q
停止、启动、杀死、重启一个容器
docker stop Name/ID
docker start Name/ID

docker kill Name/ID
docker restart name/ID

¶Docker-compse 配置文件

version: '2.2'
services:
  cerebro:
    image: lmenezes/cerebro:0.8.3
    container_name: cerebro
    ports:
      - "9000:9000"
    command:
      - -Dhosts.0.host=http://elasticsearch:9200
    networks:
      - es72net
  kibana:
    image: kibana:7.2.0
    container_name: kibana72
    environment:
      #- I18N_LOCALE=zh-CN
      - XPACK_GRAPH_ENABLED=true
      - TIMELION_ENABLED=true
      - XPACK_MONITORING_COLLECTION_ENABLED="true"
    ports:
      - "5601:5601"
    networks:
      - es72net
  elasticsearch:
    image: elasticsearch:7.2.0
    container_name: es72_01
    environment:
      - cluster.name=geektime
      - node.name=es72_01
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.seed_hosts=es72_01,es72_02
      - cluster.initial_master_nodes=es72_01,es72_02
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - es72data1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - es72net
  elasticsearch2:
    image: elasticsearch:7.2.0
    container_name: es72_02
    environment:
      - cluster.name=geektime
      - node.name=es72_02
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.seed_hosts=es72_01,es72_02
      - cluster.initial_master_nodes=es72_01,es72_02
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - es72data2:/usr/share/elasticsearch/data
    networks:
      - es72net


volumes:
  es72data1:
    driver: local
  es72data2:
    driver: local

networks:
  es72net:
    driver: bridge

¶docker 阿里云镜像加速器

¶docker 中的服务

以上准备工作就绪之后执行以下指令启动 docker 容器

/usr/local/software/elasticsearch/docker-es-7.2
[root@izwz920kp0myp15p982vp4z docker-es-7.2]# ls
docker-compose.yaml
[root@izwz920kp0myp15p982vp4z docker-es-7.2]# docker-compose up

启动完成后。我们一共启动了以下三个服务

kibana

端口:5601
elasticsearch

端口:9200
cerebro

端口:9000

可以看到上面6行信息分别表示有一个 elasticsearch集群叫 geektime，有两个节点，有2个索引，分布在4个分片上，有6个文档，总共存储消耗了123.50KB 的磁盘空间。

Logstash 安装

¶Logstash 下载安装

同上面的 Elasticsearch 和 Kibana到官网进行下载，注意 Logstash 并没有集成 Jdk，所以需要先安装 Jdk 并配置$JAVA_HOME 环境变量

[root@izwz920kp0myp15p982vp4z logstash-7.1.0]# pwd
/usr/local/software/elasticsearch/logstash-7.1.0
[root@izwz920kp0myp15p982vp4z logstash-7.1.0]# ./bin/logstash --version
logstash 7.1.0

附：Logstash 参考文档

¶下载最MovieLens最小测试数据集

下载完毕：

[root@izwz920kp0myp15p982vp4z movielens]# pwd
/usr/local/software/elasticsearch/movielens
[root@izwz920kp0myp15p982vp4z movielens]# ll
总用量 980
drwxr-xr-x 2 root root   4096 4月  24 00:09 ml-latest-small
-rw-r--r-- 1 root root   9703 4月  24 00:09 ml-latest-small-README.html
-rw-r--r-- 1 root root 978202 4月  24 00:09 ml-latest-small.zip
-rw-r--r-- 1 root root    166 4月  24 00:09 movies_settings.json

¶Logstash 配置文件

logstash.conf

input {
  file {
    # 指向测试数据csv文件
    path => "/usr/local/software/elasticsearch/movielens/ml-latest-small/movies.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
    separator => ","
    columns => ["id","content","genre"]
  }

  mutate {
    split => { "genre" => "|" }
    remove_field => ["path", "host","@timestamp","message"]
  }

  mutate {

    split => ["content", "("]
    add_field => { "title" => "%{[content][0]}"}
    add_field => { "year" => "%{[content][1]}"}
  }

  mutate {
    convert => {
      "year" => "integer"
    }
    strip => ["title"]
    remove_field => ["path", "host","@timestamp","message","content"]
  }

}
output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "movies"
     document_id => "%{id}"
   }
  stdout {}
}

¶启动 Logstash 抓取测试数据

[root@izwz920kp0myp15p982vp4z logstash-7.1.0]# pwd
/usr/local/software/elasticsearch/logstash-7.1.0
[root@izwz920kp0myp15p982vp4z logstash-7.1.0]# ./bin/logstash -f ../movielens/logstash.conf

转载规则

《003_安装上手》由阿钟采用知识共享署名 4.0 国际许可协议进行许可。