分布式 on F嘉阳

elasticsearch分布式环境搭建和插件安装

Thu, 17 May 2018 18:01:00 +0800

环境搭建

安装包和插件下载

下载elasticsearch

通过官网下载elasticsearch 官网下载页面此处下载的为elasticsearch-6.2.4.tar.gz

下载elasticsearch-head插件

下载elasticsearch-head插件可以在浏览器中以图形化的方式显示下载elasticsearch中的数据 GitHub下载页面

安装node环境

elasticsearch-head插件依赖node环境来启动，可以通过官方node.js安装包安装，也可以通过宝塔自带的PM2管理器安装，此处安装宝塔的PM2管理器

安装后检查当前环境中的node版本，若装好后node命令无法使用可以尝试重启服务器或者让配置环境变量生效

 [root@localhost elasticsearch]# node -v
 v8.11.2

将两个下载好的文件用宝塔上传到CentOS 7环境中并解压

单节点环境配置

elasticsearch单节点配置

配置文件目录为elasticsearch-6.2.4/config 修改elasticsearch.yml

 #虚拟机IP
 network.host: 192.168.79.138
 #es端口
 http.port: 9200
 
 #防止跨源请求阻塞
 http.cors.enabled: true
 http.cors.allow-origin: "*"
 
 #主节点名称
 cluster.name: Fjiayang
 #节点名称
 node.name: master
 #是否为主节点
 node.master: true

若虚拟机或者服务器内存较小，则还要配置jvm.options，将默认1g内存设置为512m或者更小

 -Xms512m
 -Xmx512m

elasticsearch-head插件安装

进入elasticsearch-head安装目录，执行node的安装命令

 [root@localhost elasticsearch]# cd elasticsearch-head-master
 [root@localhost elasticsearch-head-master]# npm install

集群环境配置

集群环境只需要解压多份es压缩包，并按如下对elasticsearch.yml配置文件进行简单配置即可，此处搭建了一个主节点，两个从节点从节点1

 #子节点IP
 network.host: 192.168.79.138
 #子节点端口
 http.port: 8200
 #主节点IP
 discovery.zen.ping.unicast.hosts: ["192.168.79.138"]
 #主节点名称
 cluster.name: Fjiayang
 #子节点名称
 node.name: slave1

从节点2

 #子节点IP
 network.host: 192.168.79.138
 #子节点端口
 http.port: 8300
 #主节点IP
 discovery.zen.ping.unicast.hosts: ["192.168.79.138"]
 #主节点名称
 cluster.name: Fjiayang
 #子节点名称
 node.name: slave2

服务启动

elasticsearch启动

elasticsearch默认不支持root用户启动，故要新建用户并对运行目录授予权限才能启动es

 [root@localhost elasticsearch]# adduser es
 [root@localhost elasticsearch]# passwd es
 更改用户 es 的密码 。
 新的 密码：
 无效的密码： 密码未通过字典检查 - 过于简单化/系统化
 重新输入新的 密码：
 passwd：所有的身份验证令牌已经成功更新。

在root用户下对es目录所有权进行更改

 [root@localhost www]# chown -R es /www/elasticsearch
 [root@localhost www]# ll
 总用量 4
 drwxrwxrwx 5 es www 151 5月 16 21:10 elasticsearch
 drwxr-xr-x 6 root root 4096 5月 7 09:01 Recycle_bin
 drwxr-xr-x. 14 root root 179 5月 7 09:20 server
 drwxrwxrwx. 3 root root 116 5月 6 15:41 wwwlogs
 drwxr-xr-x. 4 root root 38 5月 6 15:41 wwwroot

切换到新建的用户es，用/bin/elasticsearch启动es服务，若加上-d则为后台启动，即/bin/elasticsearch -d

 [root@localhost www]# su es
 [es@localhost elasticsearch]$ cd elasticsearch/elasticsearch-6.2.4
 [es@localhost elasticsearch-6.2.4]$ ls
 bin config data hs_err_pid4546.log lib LICENSE.txt logs modules NOTICE.txt plugins README.textile
 [es@localhost elasticsearch-6.2.4]$ ./bin/elasticsearch -d

如果配置了集群则继续对每个从节点逐一启动即可

elasticsearch-head插件启动

进入elasticsearch-head目录，使用npm run start启动node服务即可

服务检验

首先使用宝塔开放相关防火墙端口，

直接访问es服务，浏览器默认显示为原生json格式，此处使用chrome浏览器的JSON viewer进行了美化

访问elasticsearch-head插件，此处截图为配置好集群后的截图

elasticsearch环境搭建完成！

Hadoop分布式搭建测试

Sat, 21 Apr 2018 18:10:00 +0800

搭建环境

Windows VM虚拟机

虚拟机配置 CPU:7700 单核 RAM:2G SSD:20G OS:CentOS 7.4

环境配置

修改主机名

因为三台虚拟机均为一台虚拟机克隆而成，且源主机名为localhost，先计划将三台主机分别设置为hadoop0，hadoop1，hadoop2 IP、主机名、职能分别为： 192.168.79.129 hadoop0 NameNode DataNode ResourceManager NodeManager 192.168.79.130 hadoop1 DataNode NodeManager 192.168.79.131 hadoop2 DataNode NodeManager

CentOS 7使用hostnamectl set-hostname hadoop0设置主机名，其他几台同样操作，输入后可通过hostnamectl status查看是否修改成功

 [root@hadoop0 ~]# hostnamectl status
 Static hostname: hadoop0
 Icon name: computer-vm
 Chassis: vm
 Machine ID: 4eddbba909c84e159cfa474604412c3c
 Boot ID: df37e835fde94d039153028c9479b68e
 Virtualization: vmware
 Operating System: CentOS Linux 7 (Core)
 CPE OS Name: cpe:/o:centos:centos:7
 Kernel: Linux 3.10.0-693.el7.x86_64
 Architecture: x86-64

由于hadoop0是根节点，用于资源调度，故其需要具有所有DataNode的访问地址和权限，进入/etc/hosts文件，添加如下内容将主机名解析到对应的IP

 192.168.79.129 hadoop0
 192.168.79.130 hadoop1
 192.168.79.131 hadoop2

SSH免密码登录

进入当前用户的根目录/home/hadoop/.ssh/，在该目录下执行ssh-keygen -t rsa生成主机公钥，过程中弹出的除了 Overwrite (y/n)? y选择y确认覆盖之外，其余全为空值，则可生成免密码登录的公钥 【注意】Ubuntu系统操作略有特殊，建议使用CentOS系统进行操作

[hadoop@hadoop0 root]$ cd /home/hadoop/.ssh/
[hadoop@hadoop0 .ssh]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
/home/hadoop/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:RjyGCGedx9xzxXG97KGgjXMIZ/pg7Q+tLU1nJoxIhm8 hadoop@hadoop0
The key's randomart image is:
+---[RSA 2048]----+
| . o. + . oo.o|
| + .oo+ o . ...|
| . o.= o . .|
| . * + . + |
| + S B . o .|
| E B.* = . |
| o +.=.= |
| ++. |
| .oo |
+----[SHA256]-----+

使用ssh-copy-id将本地主机的公钥复制到远程主机的authorized_keys文件上，此时先对本地主机hadoop0授权

[hadoop@hadoop0 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop0
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub"
The authenticity of host 'hadoop0 (fe80::e99d:ad75:28e9:9276%ens33)' can't be established.
ECDSA key fingerprint is SHA256:GEEH9kBVcDuaYpJZLNsaWZA3O47+HdqIbd91fGAWcew.
ECDSA key fingerprint is MD5:cd:df:9e:8c:92:3a:9f:ae:b0:10:9b:b6:25:9c:4d:c6.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
hadoop@hadoop0's password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'hadoop0'"
and check to make sure that only the key(s) you wanted were added.

测试SSH免密码登录成功

 [hadoop@hadoop0 /]$ ssh hadoop0
 Last login: Sat Apr 21 17:18:37 2018
 [hadoop@hadoop0 ~]$ exit
 登出
 Connection to hadoop0 closed.

对另外两台主机访问授权 hadoop1

 [hadoop@hadoop0 root]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop1
 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub"
 The authenticity of host 'hadoop1 (192.168.79.130)' can't be established.
 ECDSA key fingerprint is SHA256:GEEH9kBVcDuaYpJZLNsaWZA3O47+HdqIbd91fGAWcew.
 ECDSA key fingerprint is MD5:cd:df:9e:8c:92:3a:9f:ae:b0:10:9b:b6:25:9c:4d:c6.
 Are you sure you want to continue connecting (yes/no)? yes
 /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
 /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
 hadoop@hadoop1's password:
 
 Number of key(s) added: 1
 
 Now try logging into the machine, with: "ssh 'hadoop1'"
 and check to make sure that only the key(s) you wanted were added.

hadoop2

 [hadoop@hadoop0 root]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop2
 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub"
 The authenticity of host 'hadoop2 (192.168.79.131)' can't be established.
 ECDSA key fingerprint is SHA256:GEEH9kBVcDuaYpJZLNsaWZA3O47+HdqIbd91fGAWcew.
 ECDSA key fingerprint is MD5:cd:df:9e:8c:92:3a:9f:ae:b0:10:9b:b6:25:9c:4d:c6.
 Are you sure you want to continue connecting (yes/no)? yes
 /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
 /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
 hadoop@hadoop2's password:
 
 Number of key(s) added: 1
 
 Now try logging into the machine, with: "ssh 'hadoop2'"
 and check to make sure that only the key(s) you wanted were added.

对另外两台主机做类似操作，即仅对本地主机授权，无需对hadoop0主机授权，操作后可查看authorized_keys文件内容，当前主机名即可

环境变量

三台主机均由之前的单节点主机克隆而来，故宝塔、java、hadoop环境是自带的 Java8

 [hadoop@hadoop0 .ssh]$ java -version
 java version "1.8.0_121"
 Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
 Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
 [hadoop@hadoop0 .ssh]$ javac -version
 javac 1.8.0_121

hadoop

 [hadoop@hadoop0 .ssh]$ cd /www/hadoop
 [hadoop@hadoop0 hadoop]$
 [hadoop@hadoop0 hadoop]$ ls
 hadoop-2.6.0-cdh5.7.0 hadoop-2.6.0-cdh5.7.0.tar.gz testFile tmp
 [hadoop@hadoop0 hadoop]$ source ~/.bash_profile

$HADOOP_HOME

 [hadoop@hadoop0 hadoop]$ echo $HADOOP_HOME
 /www/hadoop/hadoop-2.6.0-cdh5.7.0

hadoop配置

若之前配置过hadoop，建议将原目录删除后重新解压，主节点和从节点的目录都要删除干净，否则会导致大量问题 配置主节点core-site.xml

 <configuration>
 <property>
 <name>fs.defaultFS</name>
 <value>hdfs://192.168.79.129:8020</value>
 </property>
 </configuration>

配置HDFS文件存放目录和节点数hdfs-site.xml

 <property>
 <name>dfs.namenode.name.dir</name>
 <value>/www/hadoop/app/tmp/dfs/name</value>
 </property>
 <property>
 <name>dfs.datanode.data.dir</name>
 <value>/www/hadoop/app/tmp/dfs/data</value>
 </property>

配置YARN的yarn.xml，删除单节点的配置，添加如下字段

 <configuration>
 
 <!-- Site specific YARN configuration properties -->
 <property>
 <name>yarn.nodemanager.aux-services</name>
 <value>mapreduce_shuffle</value>
 </property>
 
 	<property>
 		<name>yarn.log-aggregation-enable</name>
 		<value>true</value>
 	</property>
 
 <!--ResourceManager主机名-->
 <property>
 		<name>yarn.resourcemanager.hostname</name>
 		<value>hadoop0</value>
 	</property>
 
 </configuration>

配置从节点slave，默认slave为localhost

 [hadoop@hadoop0 hadoop]$ cat slaves
 localhost

加入host配置

 [hadoop@hadoop0 hadoop]$ cat slaves
 hadoop0
 hadoop1
 hadoop2

额外配置，启用jobhistory

 <configuration>
 <property>
 <name>mapreduce.framework.name</name>
 <value>yarn</value>
 </property>
 
 <property>
 		<name>mapreduce.jobhistory.address</name>
 		<value>192.168.79.129:10020</value>
 		<description>MapReduce JobHistory Server IPC host:port</description>
 	</property>
 
 	<property>
 		<name>mapreduce.jobhistory.webapp.address</name>
 		<value>192.168.79.129:19888</value>
 		<description>MapReduce JobHistory Server Web UI host:port</description>
 	</property>
 
 	<property>
 		<name>mapreduce.jobhistory.done-dir</name>
 		<value>/history/done</value>
 	</property>
 
 	<property>
 		<name>mapreduce.jobhistory.intermediate-done-dir</name>
 		<value>/history/done_intermediate</value>
 	</property>
 
 </configuration>

【注意】配置好后将单节点时的/www/hadoop/hadoop-2.6.0-cdh5.7.0/logs文件夹删除或者直接重新解压hadoop的压缩包防止出现文件夹权限导致的错误

资源分发

将主节点的所有配置分发到从节点，避免重复配置。在主节点执行如下命令即可

 scp -r /www/hadoop hadoop@hadoop1:/www
 scp -r /www/hadoop hadoop@hadoop2:/www

分发完成

去根节点查看文件是否已经传到，分别执行source ~/.bash_profile以启用hadoop命令

注意分发前执行一次hadoop命令是报未找到命令，说明从节点的hadoop旧根目录是删除干净的，分发完成后再次执行命令就有效了

NameNode格式化

格式化只对主节点执行即可

 hdfs namenode -format

格式化成功

hadoop分布式启动

主节点启动，在/sbin目录下，执行./start-all.sh

 [hadoop@hadoop0 bin]$ cd ../sbin
 [hadoop@hadoop0 sbin]$ ./start-all.sh
 This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
 18/04/21 22:16:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 Starting namenodes on [hadoop0]

从节点的datanode和namenode逐一启动

 hadoop0: starting namenode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop0.out
 hadoop1: starting datanode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop1.out
 hadoop0: starting datanode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop0.out
 hadoop2: starting datanode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop2.out
 Starting secondary namenodes [0.0.0.0]

主节点secondarynamenode启动

 0.0.0.0: starting secondarynamenode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop0.out
 18/04/21 22:16:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

启动yarn和resourcemanager、nodemanager

 starting yarn daemons
 starting resourcemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-hadoop0.out
 hadoop1: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop1.out
 hadoop2: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop2.out
 hadoop0: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop0.out

hadoop分布式启动验证

命令行方式

使用jps查看进程，以下进程少一个都不行，若缺少，查看另一篇文章寻找解决办法hadoop分布式搭建的一些坑

[[hadoop分布式搭建的一些坑]]

启动主节点后从节点的HDFS会自动启动，进入从节点，使用jps查看，有两个进程即成功

 [hadoop@hadoop2 hadoop]$ jps
 29876 Jps
 29575 DataNode
 29692 NodeManager

webUI方式

使用webUI确认所有进程成功启动，查看HDFS节点

查看YARN

已经有三个从节点注册到YARN

分布式集群测试

分布式文件测试

上传测试文件再查看节点情况初始节点为空

 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ ls
 bin cloudera examples include libexec logs README.txt share
 bin-mapreduce1 etc examples-mapreduce1 lib LICENSE.txt NOTICE.txt sbin src
 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -ls /
 18/04/21 22:33:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

新建数据目录

 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -mkdir /data
 18/04/21 22:33:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -ls /
 18/04/21 22:33:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 Found 1 items
 drwxr-xr-x - hadoop supergroup 0 2018-04-21 22:33 /data

上传小文件

 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -put README.txt /data
 18/04/21 22:34:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -ls /data
 18/04/21 22:34:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 Found 1 items
 -rw-r--r-- 3 hadoop supergroup 1366 2018-04-21 22:34 /data/README.txt

上传大文件

 [hadoop@hadoop0 hadoop]$ ll
 总用量 304284
 drwxrwxr-x 3 hadoop hadoop 17 4月 21 22:16 app
 drwxrwxrwx 15 www www 253 4月 21 22:08 hadoop-2.6.0-cdh5.7.0
 -rwxrwxrwx 1 www www 311585484 3月 31 23:06 hadoop-2.6.0-cdh5.7.0.tar.gz
 drwxr-xr-x 2 www www 301 4月 11 06:31 testFile
 [hadoop@hadoop0 hadoop]$ hadoop fs -put hadoop-2.6.0-cdh5.7.0.tar.gz /data
 18/04/21 22:35:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 [hadoop@hadoop0 hadoop]$ hadoop fs -ls /data
 18/04/21 22:36:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 Found 2 items
 -rw-r--r-- 3 hadoop supergroup 1366 2018-04-21 22:34 /data/README.txt
 -rw-r--r-- 3 hadoop supergroup 311585484 2018-04-21 22:36 /data/hadoop-2.6.0-cdh5.7.0.tar.gz

通过webUI查看文件情况

分布式MapReduce测试

测试MapReduce，进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce目录，执行pi运算hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 2 3 【注意】由于此处是虚拟机，不建议将后面的精度设置过大，否则会报IO错误 进入YARN的web界面查看

运行成功

分布式日志离线处理测试

上传日志数据 [hadoop@hadoop0 testFile]$ hadoop fs -put access.log /data 18/04/21 23:42:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable 执行

 hadoop jar hadoopstudy-0.6-jar-with-dependencies.jar com.fjy.hadoop.hadoop.project.LogApp /data/access.log /logaccess/out

Web查看工作详情

查看结果

题外话

其实同时运行三台虚拟机是很吃配置的，特别是内存，对于hadoop来说内存太小很多都会报异常，因此本次的测试机器配置还可以的，三台虚拟机均运行在480G SATA3的固态上 CPU:7700 4core 8thread RAM:16G DDR4 Dual Channal SSD:128+480 nvme+sata3 HDD:9T sata3 Eth:1000M I219V 但在实际运行中依然顶着内存瓶颈，多次使用了系统盘的交换分区