Hadoop on F嘉阳

HBase伪分布式安装

Sat, 15 Dec 2018 19:18:00 +0800

Hadoop伪分布式安装

基础环境

Hadoop：版本2.7.3，下载地址
HBase：版本1.2.4，下载地址
JDK8

jdk 安装命令

$ yum -y install java-devel java

查看当前系统jre和jdk路径，输入update-alternatives --config java和update-alternatives --config javac

$ update-alternatives --config java

共有 1 个提供“java”的程序。

 选项 命令
-----------------------------------------------
*+ 1 java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/bin/java)

$ update-alternatives --config javac

共有 1 个提供“javac”的程序。

 选项 命令
-----------------------------------------------
*+ 1 java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/bin/javac)

查看环境

$ ll
总用量 311132
-rw-------. 1 root root 1241 8月 11 10:53 anaconda-ks.cfg
-rw-r--r--. 1 root root 214092195 8月 26 2016 hadoop-2.7.3.tar.gz
-rw-r--r--. 1 root root 104497899 10月 26 2016 hbase-1.2.4-bin.tar.gz
$ java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
$ javac -version
javac 1.8.0_191

免密登录

新建hadoop用户，忽略密码强度提示

$ adduser hadoop
$ passwd hadoop

为方便远程连接和学习，同时授予root权限，输入whereis sudoers查看权限控制文件位置

$ whereis sudoers
sudoers: /etc/sudoers /etc/sudoers.d /usr/libexec/sudoers.so /usr/share/man/man5/sudoers.5.gz

vi /etc/sudoers修改内容为

root ALL=(ALL) ALL
hadoop ALL=(ALL) ALL

切换至hadoop用户，使用 ssh-keygen -t rsa生成私钥信息，位置在/home/hadoop/，并且忽略所有提示输入的信息

查看生成的文件，该SSH文件夹为隐藏文件夹，故用ls -a查看，可见自动生成了.ssh目录和id_rsa,id_rsa.pub两个文件

$ ls /home/hadoop/.ssh/
id_rsa id_rsa.pub

复制公钥到.ssh目录

$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys

免密测试

$ ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is SHA256:3UEr3lx2jznmN3FL3SflViR05IZe6AWweb3TeSsfX0M.
ECDSA key fingerprint is MD5:0f:31:c1:cc:54:69:7f:d8:8d:7c:8c:22:95:2e:03:4e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Last login: Sat Dec 15 11:14:27 2018
$ exit
登出
Connection to localhost closed.

安装Hadoop

解压文件

$ tar -zxvf hadoop-2.7.3.tar.gz

修改配置文件

$ vi /hadoop-2.7.3/etc/hadoop/hadoop-env.sh

配置Java路径

# The java implementation to use.
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64

修改HDFS配置

$ vi /hadoop-2.7.3/etc/hadoop/hdfs-site.xml

配置内容

<configuration>
	<property>
	 <name>dfs.replication</name>
	 <!-- 配置副本数量-->
	 <value>1</value>
	</property>
	<property>
	 <name>dfs.namenode.name.dir</name>
	 <value>file:/home/hadoop/hadoop_data/dfs/name</value>
	</property>
	<property>
	 <name>dfs.namenode.data.dir</name>
	 <value>file:/home/hadoop/hadoop_data/dfs/data</value>
	</property>
</configuration>

手动创建相关目录

$ mkdir -p /home/hadoop/hadoop_data/dfs/name
$ mkdir -p /home/hadoop/hadoop_data/dfs/data

配置core-site.xml，内容为

<configuration>
 <property>
 <name>fs.defaultFS</name>
 <!-- 配置主机IP-->
 <value>hdfs://0.0.0.0:8020</value>
 </property>

 <!-- 配置临时文件位置-->
 <property>
 <name>hadoop.tmp.dir</name>
 <value>file:/home/hadoop/hadoop_data</value>
 </property>
</configuration>

放行端口并重载

$ firewall-cmd --zone=public --add-port=8020/tcp --add-port=50070/tcp --permanent
$ firewall-cmd --reload

启动Hadoop

初始化

进入/hadoop-2.7.3/bin目录，使用如下语句初始化Hadoop，该操作仅执行一次即可，以后使用无需重复执行，否则执行后会删除当前hadoop上的所有文件

$ hadoop namenode -format

滚动均为info级别的日志并且status 0表示已初始化成功

...
18/12/15 17:07:32 INFO namenode.FSImage: Allocated new BlockPoolId: BP-99731851-127.0.0.1-1544864852133
18/12/15 17:07:32 INFO common.Storage: Storage directory /home/hadoop/hadoop_data/dfs/name has been successfully formatted.
18/12/15 17:07:32 INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/hadoop_data/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
18/12/15 17:07:32 INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/hadoop_data/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 353 bytes saved in 0 seconds.
18/12/15 17:07:32 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
18/12/15 17:07:32 INFO util.ExitUtil: Exiting with status 0
18/12/15 17:07:32 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
************************************************************/

启动HDFS

进入/hadoop-2.7.3/sbin目录输入./start-dfs.sh启动HDFS服务

$ ./start-dfs.sh
Starting namenodes on [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is SHA256:3UEr3lx2jznmN3FL3SflViR05IZe6AWweb3TeSsfX0M.
ECDSA key fingerprint is MD5:0f:31:c1:cc:54:69:7f:d8:8d:7c:8c:22:95:2e:03:4e.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting namenode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-namenode-localhost.localdomain.out
localhost: starting datanode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-datanode-localhost.localdomain.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-secondarynamenode-localhost.localdomain.out

进程验证，若出现如下四个进程表示HDFS服务启动成功

$ jps
28930 Jps
28821 SecondaryNameNode
28663 DataNode
28541 NameNode

配置hdfs命令

使用vi ~/.bash_profile进入配置文件，按如下配置

export HADOOP_HOME=/home/hadoop/hadoop-2.7.3
export PATH=$HADOOP_HOME/bin:$PATH

使配置文件生效source ~/.bash_profile 输入hdfs回车测试命令是否生效，若出现如下画面则表示命令已经生效

$ hdfs
Usage: hdfs [--config confdir] [--loglevel loglevel] COMMAND
 where COMMAND is one of:
 dfs run a filesystem command on the file systems supported in Hadoop.
 classpath prints the classpath
 namenode -format format the DFS filesystem
 secondarynamenode run the DFS secondary namenode
 namenode run the DFS namenode
 journalnode run the DFS journalnode
 zkfc run the ZK Failover Co
 ...

命令测试

$ hadoop fs -mkdir /test
$ hadoop fs -ls /
Found 1 items
drwxr-xr-x - hadoop supergroup 0 2018-12-15 17:14 /test

HBase伪分布式搭建

HBase安装

解压安装包，目录如下

$ ll
总用量 320
drwxr-xr-x. 4 hadoop hadoop 4096 1月 29 2016 bin
-rw-r--r--. 1 hadoop hadoop 122439 10月 26 2016 CHANGES.txt
drwxr-xr-x. 2 hadoop hadoop 178 1月 29 2016 conf
drwxr-xr-x. 12 hadoop hadoop 4096 10月 26 2016 docs
drwxr-xr-x. 7 hadoop hadoop 80 10月 26 2016 hbase-webapps
-rw-rw-r--. 1 hadoop hadoop 261 10月 26 2016 LEGAL
drwxrwxr-x. 3 hadoop hadoop 8192 12月 15 11:24 lib
-rw-rw-r--. 1 hadoop hadoop 130696 10月 26 2016 LICENSE.txt
-rw-rw-r--. 1 hadoop hadoop 42025 10月 26 2016 NOTICE.txt
-rw-r--r--. 1 hadoop hadoop 1477 12月 27 2015 README.txt

拷贝Hadoop配置文件到/conf目录

$ cp /home/hadoop/hadoop-2.7.3/etc/hadoop/hdfs-site.xml .
$ cp /home/hadoop/hadoop-2.7.3/etc/hadoop/core-site.xml .

修改hbase-env.sh文件，增加JDK路径

# The java implementation to use. Java 1.7+ required.
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64

由于使用的是jdk8，主要注释内存配置

# Configure PermSize. Only needed in JDK7. You can safely remove it for JDK8+
#export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m"
#export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m"

另外打开文件最后zookeeper配置，设置为使用内置的zookeeper

# Tell HBase whether it should manage it's own instance of Zookeeper or not.
export HBASE_MANAGES_ZK=true

修改hbase-site.xml文件，内容如下

<configuration>
	<property>
		<name>hbase.rootdir</name>
 <!-- 配置文件根目录-->
		<value>hdfs://localhost:8020/hbase</value>
	</property>
	<property>
		<name>hbase.zookeeper.property.dataDir</name>
		<value>/home/hadoop/hadoop_data/zookeeper</value>
	</property>
	<property>
		<name>hbase.cluster.distributed</name>
 <!-- 配置以集群方式运行-->
		<value>true</value>
	</property>
</configuration>

参考文档

启动HBase

在/bin目录下执行./start-hbase.sh启动HBase

$ ./start-hbase.sh 
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
localhost: starting zookeeper, logging to /home/hadoop/hbase-1.2.4/bin/../logs/hbase-hadoop-zookeeper-localhost.localdomain.out
localhost: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
starting master, logging to /home/hadoop/hbase-1.2.4/bin/../logs/hbase-hadoop-master-localhost.localdomain.out
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
starting regionserver, logging to /home/hadoop/hbase-1.2.4/bin/../logs/hbase-hadoop-1-regionserver-localhost.localdomain.out

由于虚拟机只分配了单核单线程，GC警告暂时忽略

进程验证，确认启动成功

$ jps
28821 SecondaryNameNode
28663 DataNode
29911 HRegionServer
29785 HMaster
29724 HQuorumPeer
28541 NameNode
30303 Jps

进入HBase命令行测试

$ ./hbase shell
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hbase-1.2.4/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.2.4, r67592f3d062743907f8c5ae00dbbe1ae4f69e5af, Tue Oct 25 18:10:20 CDT 2016

hbase(main):001:0> status
1 active master, 0 backup masters, 1 servers, 0 dead, 2.0000 average load

hbase(main):002:0>

查看其是否自动在HDFS上创建目录/hbase

$ hadoop fs -ls /
Found 2 items
drwxr-xr-x - hadoop supergroup 0 2018-12-15 18:43 /hbase
drwxr-xr-x - hadoop supergroup 0 2018-12-15 17:14 /test

HBase 学习笔记

Sat, 15 Dec 2018 12:06:00 +0800

概述

HBase 简介

Hbase是一个分布式的、面向列的开源数据库
- 列式数据库：将列值统一存储
- 行式数据库：关系型数据库，将每一行值统一存储
Hbase在Hadoop之上提供了类似于Bigtable的能力
- Bigtable是压缩、高性能、高可扩展性的基于Google GFS文件系统的数据库，用于存储大规模的结构化数据，在扩展性和性能上有很大的优势
Hbase不同于一般的关系数据库，它适合非结构化数据存储
- 非结构化数据：不能用二维表进行存储的数据，例如图片、文档等

Hbase地位

Apache基金会顶级项目
基于Hadoop核心HDFS系统进行数据存储，类似于Hive
可以存储超大数据并适合用来进行大数据的的实时查询

Hbase与HDFS

Hbase建立在Hadoop文件系统上，利用了Hadoop文件系统的容错能力
Hbase提供对数据随机实时读/写访问功能
Hbase内部使用哈希表存储索引，可将HDFS文件中的数据进行快速查找

适用场景

瞬间写入量很大，常用数据库不好支持或者需要很高的成本支撑
数据需要长久保存，且数据量会持久增长到比较大的场景
Hbase不适合用于有join、多级索引、表关系复杂的数据模型

存储模型与原理

CAP定理

分布式系统的CAP理论：理论首先把分布式系统中的三个特性进行了如下归纳：

一致性（C）：所有节点在同一时间具有相同的数据
可用性（A）：保证每个请求不管成功或者失败（包含节点宕机）都有响应，但不保证获取数据的正确性
分区容错性（P）：系统中任意信息的丢失或失败不会影响系统的继续运作。系统如果不能在时限内达成数据一致性，必须就当前操作在C和A之间做出选择。

Hbase为CP特性

ACID

ACID的定义：

Atomic原子性:所有的步骤要么全部完成要么一个也不会完成
Consistent一致性: 通过各种途径包括外键约束等任何写入数据库的数据都是有效的，不能发生表与表之间存在外键约束
Isolated隔离性: 一个未完成事务不会影响另外一个未完成事务
Durable持久性: 一旦一个事务被提交，它应该持久保存

HBase非强ACID

HBase概念

字段	概念	备注
NameSpace	类似RDBMS的“数据库”
Table	表名必须是能用在文件路径里的合法名字	原因：每个表都会映射为HDFS的文件
Row	在表里，每一行代表一个数据对象，每一行都以一个行键（Row Key）来唯一标识的，行键没有特定的数据类型，以二进制的字节进行存储
Row Key	唯一标识一行记录，不可改变，只能删除
Column	由Column family和Column qualifier组成，由（：）进行分隔	例如：family：qualifier
Column family	在定义HBase表的时候需要提前设置好列族，表中所有的列都需要组织在列族里面	同一个列族的成员有相同的前缀，物理上，一个列族的成员都是储存在一起，用于存储优化
Column qualifier（列限定符）	列族中的数据通过列标识来进行映射，可以理解为一个键值对，Column qualifier就是Key	类似具体的列名
Cell	每一个行键，列族和列标识共同组成的一个单元	无特定的数据类型，以二进制字节存储
Timestamp	每个值都有一个Timestamp，作为该值特定版本的标识符	读未指定版本则返回最新版，写未指定时间则使用当前时间，HBase默认保留3个版本数据

HBase与传统关系数据库的区别

	Hbase	RDBMS
数据库大小	PB	GB、TB
数据类型	Bytes	丰富的数据类型
事务支持	ACID只支持单个Row级别	全面的ACID支持
索引	只支持Row-Key	支持
吞吐量	百万查询/每秒	数千查询/每秒

HBase基础架构

HBase分布式系统简介

HMaster
- HBase主/从集群架构中的中央节点
- 负责将region（HBase中存储最小单元，表格的基本单位）分配给RegionServer，协调RegionServer的负载并维护集群状态
- 维护表和Region的元数据，不参与数据的输入/输出过程
RegionServer
- 维护HMaster分配给他的region，处理对这些region的IO请求
- 负责切分正在运行过程中变得过大的region
Zookeeper
- 集群协调器，保证至少一个节点出于active状态
- HMaster启动将系统表加载到ZK（用于服务发现）
- 提供HBase RegionServer状态信息

HBase原理

HBase写流程

Client先访问zookeeper，得到对应的RegionServer地址
Client对RegionServer发起写请求，RegionServer接受数据写入内存
当MemStore的大小达到一定值后，flush到StoreFile并存储到HDFS

HBase特殊点在于其先写内存再写日志，通过类似MySQL中MVCC机制保证一致性

HBase读流程

Client先访问zookeeper，得到对应的RegionServer地址
Client对RegionServer发起读请求
当RegionServer收到client的读请求后，先扫描自己的Memstore，再扫描BlockCache（加速读内容缓存区）如果还没找到则从StoreFile中读取数据，然后将数据返回给Client

读特点

读过程与HMaster无关，只和ZK有关，有效的减小HMaster负载

HBase模块协作

有关HBase的三个问题

HBase启动时发生了什么
当RegionServer失效后会发生什么
当HMaster失效后会发生什么

HBase启动

HBase启动，注册到zookeeper，等待RegionServer汇报
RegionServer注册到zookeeper，并向HMaster汇报
对各个RegionServer（包括失效的）的数据进行整理，分配Region和meta信息（所有表的索引）

RegionServer失效

HMaster将失效RegionServer上的Region分配到其他节点
HMaster更新HBase：meta表保证数据正常访问

HMaster失效

配置高可用后

出于Backup状态的其他HMaster节点选举出一个转为Active状态

未配置高可用

数据能正常读写，但不能创建删除表，也不能更改表结构，因为涉及meta表更新

hadoop分布式搭建的一些坑

Sun, 22 Apr 2018 10:39:00 +0800

DataNode没启动

这种情况多出现于该机器之前配置过hadoop，没清理干净的情况，即仍有文件目录，导致新hadoop的clusterID与旧的不一致，导致HDFS无法启动，报JavaIO异常

 2018-04-22 00:21:15,868 WARN org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /www/hadoop/app/tmp/dfs/data: namenode clusterID = CID-6c7766c6-b260-42b6-8c03-07dda802c0cf; datanode clusterID = CID-a53df7ad-e593-4a29-ad15-a93486c54776
 2018-04-22 00:21:15,868 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to hadoop0/192.168.79.129:8020. Exiting. 
 java.io.IOException: All specified directories are failed to load.
 	at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
 	at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1394)
 	at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1355)
 	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317)
 	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:228)
 	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:829)
 	at java.lang.Thread.run(Thread.java:745)
 2018-04-22 00:21:15,869 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to hadoop0/192.168.79.129:8020

解决方案

清理所有节点的旧文件存储目录，重新再根节点格式化存储空间

 rm -rf app/
 hdfs namenode -format

重新启动HDFS

 [hadoop@hadoop0 bin]$ cd ../sbin
 [hadoop@hadoop0 sbin]$ ./start-all.sh

查看进程，DataNode已启动

 [hadoop@hadoop0 sbin]$ jps
 37618 DataNode
 37783 SecondaryNameNode
 37932 ResourceManager
 37485 NameNode
 38045 NodeManager
 38365 Jps

NodeManager没启动/启动后进程消失

该情况由于yarn-site.xml文件配置有误，导致从节点找不到主节点，报JavaNet异常

 org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.BindException: Problem binding to [hadoop0:0] java.net.BindException: 无法指定被请求的地址; For more details see: http://wiki.apache.org/hadoop/BindException
 	at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:139)
 	at org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
 	at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:406)
 	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
 	at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
 	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:266)
 	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
 	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:475)
 	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:521)

解决方案

重新配置yarn-site.xml，确保其有下面这一段，并且使用SSH 主机名能够访问该节点

 <!--ResourceManager主机名-->
 <property>
 	<name>yarn.resourcemanager.hostname</name>
 	<value>hadoop0</value>
 </property>

YARN运算IO错误

该错误由于机器内存过小，却执行复杂的运算，导致内存不足，报JavaIO异常，例如在2G内存的机器上执行 hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 4 5运算

 Job Finished in 34.889 seconds
 java.io.FileNotFoundException: File does not exist: hdfs://192.168.79.129:8020/user/hadoop/QuasiMonteCarlo_1524323663509_1454922894/out/reduce-out
 at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1219)
 at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
 at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
 at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1750)
 at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1774)
 at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
 at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
 at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
 at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
 at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

YARN也显示工作失败

解决方案

调整虚拟内存，或者减小运算精度，例如设置精度为hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 2 3则一般不会有问题

机器时间不同步

该问题出现在MapReduce测试时，若主从节点机器时间不一致则会报同步错误

 18/04/21 23:55:20 INFO mapreduce.Job: Task Id : attempt_1524320196806_0003_m_000000_2, Status : FAILED
 Container launch failed for container_1524320196806_0003_01_000004 : org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
 This token is expired. current time is 1524333912535 found 1524326717165
 Note: System times on machines may be out of sync. Check system time and time zones.
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
 at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
 at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:159)
 at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)

解决方案

首先检查各节点的时间主节点

 [hadoop@hadoop0 testFile]$ timedatectl status
 Local time: 六 2018-04-21 23:57:21 CST
 Universal time: 六 2018-04-21 15:57:21 UTC
 RTC time: 六 2018-04-21 15:44:44
 Time zone: Asia/Shanghai (CST, +0800)
 NTP enabled: yes
 NTP synchronized: no
 RTC in local TZ: no
 DST active: n/a

从节点1

[hadoop@hadoop1 hadoop]$ timedatectl status
 Local time: 日 2018-04-22 02:06:49 CST
 Universal time: 六 2018-04-21 18:06:49 UTC
 RTC time: 六 2018-04-21 17:54:14
 Time zone: Asia/Shanghai (CST, +0800)
 NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: no
 DST active: n/a

从节点2

[hadoop@hadoop2 hadoop]$ timedatectl status
 Local time: 日 2018-04-22 02:07:00 CST
 Universal time: 六 2018-04-21 18:07:00 UTC
 RTC time: 日 2018-04-22 04:00:48
 Time zone: Asia/Shanghai (CST, +0800)
 NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: no
 DST active: n/a

三个节点显然时间不一致，使用ntpdate工具同步各机器时间

[root@hadoop2 hadoop]# yum -y install ntp ntpdate
已加载插件：fastestmirror
base | 3.6 kB 00:00:00
epel/x86_64/metalink | 7.7 kB 00:00:00
epel | 4.7 kB 00:00:00
extras | 3.4 kB 00:00:00
updates | 3.4 kB 00:00:00
(1/4): epel/x86_64/updateinfo | 916 kB 00:00:01
(2/4): extras/7/x86_64/primary_db | 185 kB 00:00:01
(3/4): epel/x86_64/primary_db | 6.3 MB 00:00:01
(4/4): updates/7/x86_64/primary_db | 6.9 MB 00:00:03
Determining fastest mirrors
 * base: centos.ustc.edu.cn
 * epel: mirror01.idc.hinet.net
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
软件包 ntp-4.2.6p5-25.el7.centos.2.x86_64 已安装并且是最新版本
软件包 ntpdate-4.2.6p5-25.el7.centos.2.x86_64 已安装并且是最新版本
无须任何处理
[root@hadoop2 hadoop]# ntpdate cn.pool.ntp.org
22 Apr 12:02:55 ntpdate[36396]: step time server 108.59.2.24 offset 35618.521668 sec

将时间偏差写入机器时间

[root@hadoop2 hadoop]# hwclock --systohc
[root@hadoop2 hadoop]# date
2018年 04月 22日 星期日 12:03:33 CST

问题解决

Hadoop分布式搭建测试

Sat, 21 Apr 2018 18:10:00 +0800

搭建环境

Windows VM虚拟机

虚拟机配置 CPU:7700 单核 RAM:2G SSD:20G OS:CentOS 7.4

环境配置

修改主机名

因为三台虚拟机均为一台虚拟机克隆而成，且源主机名为localhost，先计划将三台主机分别设置为hadoop0，hadoop1，hadoop2 IP、主机名、职能分别为： 192.168.79.129 hadoop0 NameNode DataNode ResourceManager NodeManager 192.168.79.130 hadoop1 DataNode NodeManager 192.168.79.131 hadoop2 DataNode NodeManager

CentOS 7使用hostnamectl set-hostname hadoop0设置主机名，其他几台同样操作，输入后可通过hostnamectl status查看是否修改成功

 [root@hadoop0 ~]# hostnamectl status
 Static hostname: hadoop0
 Icon name: computer-vm
 Chassis: vm
 Machine ID: 4eddbba909c84e159cfa474604412c3c
 Boot ID: df37e835fde94d039153028c9479b68e
 Virtualization: vmware
 Operating System: CentOS Linux 7 (Core)
 CPE OS Name: cpe:/o:centos:centos:7
 Kernel: Linux 3.10.0-693.el7.x86_64
 Architecture: x86-64

由于hadoop0是根节点，用于资源调度，故其需要具有所有DataNode的访问地址和权限，进入/etc/hosts文件，添加如下内容将主机名解析到对应的IP

 192.168.79.129 hadoop0
 192.168.79.130 hadoop1
 192.168.79.131 hadoop2

SSH免密码登录

进入当前用户的根目录/home/hadoop/.ssh/，在该目录下执行ssh-keygen -t rsa生成主机公钥，过程中弹出的除了 Overwrite (y/n)? y选择y确认覆盖之外，其余全为空值，则可生成免密码登录的公钥 【注意】Ubuntu系统操作略有特殊，建议使用CentOS系统进行操作

[hadoop@hadoop0 root]$ cd /home/hadoop/.ssh/
[hadoop@hadoop0 .ssh]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
/home/hadoop/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:RjyGCGedx9xzxXG97KGgjXMIZ/pg7Q+tLU1nJoxIhm8 hadoop@hadoop0
The key's randomart image is:
+---[RSA 2048]----+
| . o. + . oo.o|
| + .oo+ o . ...|
| . o.= o . .|
| . * + . + |
| + S B . o .|
| E B.* = . |
| o +.=.= |
| ++. |
| .oo |
+----[SHA256]-----+

使用ssh-copy-id将本地主机的公钥复制到远程主机的authorized_keys文件上，此时先对本地主机hadoop0授权

[hadoop@hadoop0 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop0
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub"
The authenticity of host 'hadoop0 (fe80::e99d:ad75:28e9:9276%ens33)' can't be established.
ECDSA key fingerprint is SHA256:GEEH9kBVcDuaYpJZLNsaWZA3O47+HdqIbd91fGAWcew.
ECDSA key fingerprint is MD5:cd:df:9e:8c:92:3a:9f:ae:b0:10:9b:b6:25:9c:4d:c6.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
hadoop@hadoop0's password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'hadoop0'"
and check to make sure that only the key(s) you wanted were added.

测试SSH免密码登录成功

 [hadoop@hadoop0 /]$ ssh hadoop0
 Last login: Sat Apr 21 17:18:37 2018
 [hadoop@hadoop0 ~]$ exit
 登出
 Connection to hadoop0 closed.

对另外两台主机访问授权 hadoop1

 [hadoop@hadoop0 root]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop1
 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub"
 The authenticity of host 'hadoop1 (192.168.79.130)' can't be established.
 ECDSA key fingerprint is SHA256:GEEH9kBVcDuaYpJZLNsaWZA3O47+HdqIbd91fGAWcew.
 ECDSA key fingerprint is MD5:cd:df:9e:8c:92:3a:9f:ae:b0:10:9b:b6:25:9c:4d:c6.
 Are you sure you want to continue connecting (yes/no)? yes
 /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
 /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
 hadoop@hadoop1's password:
 
 Number of key(s) added: 1
 
 Now try logging into the machine, with: "ssh 'hadoop1'"
 and check to make sure that only the key(s) you wanted were added.

hadoop2

 [hadoop@hadoop0 root]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop2
 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub"
 The authenticity of host 'hadoop2 (192.168.79.131)' can't be established.
 ECDSA key fingerprint is SHA256:GEEH9kBVcDuaYpJZLNsaWZA3O47+HdqIbd91fGAWcew.
 ECDSA key fingerprint is MD5:cd:df:9e:8c:92:3a:9f:ae:b0:10:9b:b6:25:9c:4d:c6.
 Are you sure you want to continue connecting (yes/no)? yes
 /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
 /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
 hadoop@hadoop2's password:
 
 Number of key(s) added: 1
 
 Now try logging into the machine, with: "ssh 'hadoop2'"
 and check to make sure that only the key(s) you wanted were added.

对另外两台主机做类似操作，即仅对本地主机授权，无需对hadoop0主机授权，操作后可查看authorized_keys文件内容，当前主机名即可

环境变量

三台主机均由之前的单节点主机克隆而来，故宝塔、java、hadoop环境是自带的 Java8

 [hadoop@hadoop0 .ssh]$ java -version
 java version "1.8.0_121"
 Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
 Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
 [hadoop@hadoop0 .ssh]$ javac -version
 javac 1.8.0_121

hadoop

 [hadoop@hadoop0 .ssh]$ cd /www/hadoop
 [hadoop@hadoop0 hadoop]$
 [hadoop@hadoop0 hadoop]$ ls
 hadoop-2.6.0-cdh5.7.0 hadoop-2.6.0-cdh5.7.0.tar.gz testFile tmp
 [hadoop@hadoop0 hadoop]$ source ~/.bash_profile

$HADOOP_HOME

 [hadoop@hadoop0 hadoop]$ echo $HADOOP_HOME
 /www/hadoop/hadoop-2.6.0-cdh5.7.0

hadoop配置

若之前配置过hadoop，建议将原目录删除后重新解压，主节点和从节点的目录都要删除干净，否则会导致大量问题 配置主节点core-site.xml

 <configuration>
 <property>
 <name>fs.defaultFS</name>
 <value>hdfs://192.168.79.129:8020</value>
 </property>
 </configuration>

配置HDFS文件存放目录和节点数hdfs-site.xml

 <property>
 <name>dfs.namenode.name.dir</name>
 <value>/www/hadoop/app/tmp/dfs/name</value>
 </property>
 <property>
 <name>dfs.datanode.data.dir</name>
 <value>/www/hadoop/app/tmp/dfs/data</value>
 </property>

配置YARN的yarn.xml，删除单节点的配置，添加如下字段

 <configuration>
 
 <!-- Site specific YARN configuration properties -->
 <property>
 <name>yarn.nodemanager.aux-services</name>
 <value>mapreduce_shuffle</value>
 </property>
 
 	<property>
 		<name>yarn.log-aggregation-enable</name>
 		<value>true</value>
 	</property>
 
 <!--ResourceManager主机名-->
 <property>
 		<name>yarn.resourcemanager.hostname</name>
 		<value>hadoop0</value>
 	</property>
 
 </configuration>

配置从节点slave，默认slave为localhost

 [hadoop@hadoop0 hadoop]$ cat slaves
 localhost

加入host配置

 [hadoop@hadoop0 hadoop]$ cat slaves
 hadoop0
 hadoop1
 hadoop2

额外配置，启用jobhistory

 <configuration>
 <property>
 <name>mapreduce.framework.name</name>
 <value>yarn</value>
 </property>
 
 <property>
 		<name>mapreduce.jobhistory.address</name>
 		<value>192.168.79.129:10020</value>
 		<description>MapReduce JobHistory Server IPC host:port</description>
 	</property>
 
 	<property>
 		<name>mapreduce.jobhistory.webapp.address</name>
 		<value>192.168.79.129:19888</value>
 		<description>MapReduce JobHistory Server Web UI host:port</description>
 	</property>
 
 	<property>
 		<name>mapreduce.jobhistory.done-dir</name>
 		<value>/history/done</value>
 	</property>
 
 	<property>
 		<name>mapreduce.jobhistory.intermediate-done-dir</name>
 		<value>/history/done_intermediate</value>
 	</property>
 
 </configuration>

【注意】配置好后将单节点时的/www/hadoop/hadoop-2.6.0-cdh5.7.0/logs文件夹删除或者直接重新解压hadoop的压缩包防止出现文件夹权限导致的错误

资源分发

将主节点的所有配置分发到从节点，避免重复配置。在主节点执行如下命令即可

 scp -r /www/hadoop hadoop@hadoop1:/www
 scp -r /www/hadoop hadoop@hadoop2:/www

分发完成

去根节点查看文件是否已经传到，分别执行source ~/.bash_profile以启用hadoop命令

注意分发前执行一次hadoop命令是报未找到命令，说明从节点的hadoop旧根目录是删除干净的，分发完成后再次执行命令就有效了

NameNode格式化

格式化只对主节点执行即可

 hdfs namenode -format

格式化成功

hadoop分布式启动

主节点启动，在/sbin目录下，执行./start-all.sh

 [hadoop@hadoop0 bin]$ cd ../sbin
 [hadoop@hadoop0 sbin]$ ./start-all.sh
 This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
 18/04/21 22:16:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 Starting namenodes on [hadoop0]

从节点的datanode和namenode逐一启动

 hadoop0: starting namenode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop0.out
 hadoop1: starting datanode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop1.out
 hadoop0: starting datanode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop0.out
 hadoop2: starting datanode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop2.out
 Starting secondary namenodes [0.0.0.0]

主节点secondarynamenode启动

 0.0.0.0: starting secondarynamenode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop0.out
 18/04/21 22:16:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

启动yarn和resourcemanager、nodemanager

 starting yarn daemons
 starting resourcemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-hadoop0.out
 hadoop1: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop1.out
 hadoop2: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop2.out
 hadoop0: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop0.out

hadoop分布式启动验证

命令行方式

使用jps查看进程，以下进程少一个都不行，若缺少，查看另一篇文章寻找解决办法hadoop分布式搭建的一些坑

[[hadoop分布式搭建的一些坑]]

启动主节点后从节点的HDFS会自动启动，进入从节点，使用jps查看，有两个进程即成功

 [hadoop@hadoop2 hadoop]$ jps
 29876 Jps
 29575 DataNode
 29692 NodeManager

webUI方式

使用webUI确认所有进程成功启动，查看HDFS节点

查看YARN

已经有三个从节点注册到YARN

分布式集群测试

分布式文件测试

上传测试文件再查看节点情况初始节点为空

 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ ls
 bin cloudera examples include libexec logs README.txt share
 bin-mapreduce1 etc examples-mapreduce1 lib LICENSE.txt NOTICE.txt sbin src
 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -ls /
 18/04/21 22:33:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

新建数据目录

 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -mkdir /data
 18/04/21 22:33:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -ls /
 18/04/21 22:33:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 Found 1 items
 drwxr-xr-x - hadoop supergroup 0 2018-04-21 22:33 /data

上传小文件

 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -put README.txt /data
 18/04/21 22:34:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -ls /data
 18/04/21 22:34:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 Found 1 items
 -rw-r--r-- 3 hadoop supergroup 1366 2018-04-21 22:34 /data/README.txt

上传大文件

 [hadoop@hadoop0 hadoop]$ ll
 总用量 304284
 drwxrwxr-x 3 hadoop hadoop 17 4月 21 22:16 app
 drwxrwxrwx 15 www www 253 4月 21 22:08 hadoop-2.6.0-cdh5.7.0
 -rwxrwxrwx 1 www www 311585484 3月 31 23:06 hadoop-2.6.0-cdh5.7.0.tar.gz
 drwxr-xr-x 2 www www 301 4月 11 06:31 testFile
 [hadoop@hadoop0 hadoop]$ hadoop fs -put hadoop-2.6.0-cdh5.7.0.tar.gz /data
 18/04/21 22:35:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 [hadoop@hadoop0 hadoop]$ hadoop fs -ls /data
 18/04/21 22:36:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 Found 2 items
 -rw-r--r-- 3 hadoop supergroup 1366 2018-04-21 22:34 /data/README.txt
 -rw-r--r-- 3 hadoop supergroup 311585484 2018-04-21 22:36 /data/hadoop-2.6.0-cdh5.7.0.tar.gz

通过webUI查看文件情况

分布式MapReduce测试

测试MapReduce，进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce目录，执行pi运算hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 2 3 【注意】由于此处是虚拟机，不建议将后面的精度设置过大，否则会报IO错误 进入YARN的web界面查看

运行成功

分布式日志离线处理测试

上传日志数据 [hadoop@hadoop0 testFile]$ hadoop fs -put access.log /data 18/04/21 23:42:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable 执行

 hadoop jar hadoopstudy-0.6-jar-with-dependencies.jar com.fjy.hadoop.hadoop.project.LogApp /data/access.log /logaccess/out

Web查看工作详情

查看结果

题外话

其实同时运行三台虚拟机是很吃配置的，特别是内存，对于hadoop来说内存太小很多都会报异常，因此本次的测试机器配置还可以的，三台虚拟机均运行在480G SATA3的固态上 CPU:7700 4core 8thread RAM:16G DDR4 Dual Channal SSD:128+480 nvme+sata3 HDD:9T sata3 Eth:1000M I219V 但在实际运行中依然顶着内存瓶颈，多次使用了系统盘的交换分区

MapReduce对网站日志进行分析

Fri, 20 Apr 2018 22:58:00 +0800

代码

GitHub链接开源工具类UserAgentParser地址

日志来源

根据之前的作业提交系统项目的后台获取，为了本地测试，故先从数据库中备份下来并做了适当的处理，仅保留UserAgent的相关信息

本地文件测试

使用开源工具类解析UserAgent中的信息

@Test
 public void testReadFile() throws Exception {
 String path = "F:\\JAVA Workspace\\hadoopstudy\\access.log";
 BufferedReader reader = new BufferedReader(new InputStreamReader(new FileInputStream(new File(path))));

 String line = "";
 int count = 0;

 //模拟MapReduce存储
 Map<String, Integer> browerMap = new HashMap<String, Integer>();

 UserAgentParser userAgentParser = new UserAgentParser();

 while (line != null) {
 line = reader.readLine();
 count++;
 if (StringUtils.isNotBlank(line)) {
 String source = line;
 UserAgent agent = userAgentParser.parse(source);
 //测试
 String browser = agent.getBrowser();
 String engine = agent.getEngine();
 String engineVersion = agent.getEngineVersion();
 String os = agent.getOs();
 String platform = agent.getPlatform();
 boolean mobile = agent.isMobile();
 Integer browserValue = browerMap.get(browser);
 
 if (browserValue != null) {
 browerMap.put(browser, browerMap.get(browser) + 1);
 } else {
 browerMap.put(browser, 1);
 }

 //输出解析的信息
 System.out.println(browser + "," + engine + "," + engineVersion + "," + os + "," + platform + "," + mobile);
 }

 }
 System.out.println("总记录数：" + count);
 System.out.println("====================================");
 for (Map.Entry<String,Integer>entry :browerMap.entrySet()){
 System.out.println(entry.getKey()+":"+entry.getValue());
 }
 }

执行单元测试

使用MapReduce统计

代码以WordCount为原型，结合本地测试的代码编写 … boolean mobile = agent.isMobile();

 if (mobile){
 mobileuser = "手机用户";
 }else {
 mobileuser = "非手机用户";
 }

 //通过上下文把map的处理结果输出
 context.write(new Text(browser), one);
...

打包运行

将工程用mvn assembly:assembly命令把插件一起打包，上传至虚拟机，同时将log上传至HDFS的根目录 [hadoop@localhost testFile]$ hadoop fs -put access.log / 18/04/11 06:12:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable [hadoop@localhost testFile]$ hadoop fs -ls / 18/04/11 06:12:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable Found 9 items -rw-r–r– 1 hadoop supergroup 73 2018-04-11 00:30 /PartitionerTest.txt -rw-r–r– 1 hadoop supergroup 26888 2018-04-11 06:12 /access.log drwxr-xr-x - hadoop supergroup 0 2018-04-08 10:56 /hdfsapi -rw-r–r– 1 hadoop supergroup 60 2018-04-10 23:03 /hello.txt drwxrwx— - hadoop supergroup 0 2018-04-11 01:28 /history drwxr-xr-x - hadoop supergroup 0 2018-04-11 00:35 /output drwxr-xr-x - hadoop supergroup 0 2018-04-08 10:35 /test drwx—— - hadoop supergroup 0 2018-04-11 01:36 /tmp drwxr-xr-x - hadoop supergroup 0 2018-04-09 20:11 /user 运行

结果与本地测试一致 [hadoop@localhost testFile]$ hadoop fs -ls /logaccess/browserout 18/04/11 06:15:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable Found 2 items -rw-r–r– 1 hadoop supergroup 0 2018-04-11 06:14 /logaccess/browserout/_SUCCESS -rw-r–r– 1 hadoop supergroup 32 2018-04-11 06:14 /logaccess/browserout/part-r-00000 [hadoop@localhost testFile]$ hadoop fs -text /logaccess/browserout/part-r-00000 18/04/11 06:15:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable Chrome 191 Firefox 10 Unknown 1 增加其他属性重新运行MapReduce，结果如下 [hadoop@localhost testFile]$ hadoop fs -ls /logaccess 18/04/11 06:33:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable Found 2 items -rw-r–r– 1 hadoop supergroup 0 2018-04-11 06:32 /logaccess/_SUCCESS -rw-r–r– 1 hadoop supergroup 192 2018-04-11 06:32 /logaccess/part-r-00000 [hadoop@localhost testFile]$ hadoop fs -text /logaccess/part-r-00000 18/04/11 06:33:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable 20100101 10 537.36 191 604.3.5 1 Android 48 Chrome 191 Firefox 10 Gecko 10 Linux 48 Unknown 1 Webkit 192 Windows 296 Windows 7 10 iPhone 1 iPhone OS 11.1 1 手机用户 49 非手机用户 153 日志离线统计完成！

单节点MapReduce开发

Tue, 17 Apr 2018 13:03:00 +0800

官方原理解析

MapReduce演进

MapReduce2.x的架构

MapReduce1.x的架构，一旦JobTracker挂了，整个分布式节点就崩溃了，因此被淘汰

开发WordCount

WordCount原理

数据输入到MapReduce，每个MapReduce任务都被初始化为一个Job，每个Job又可以分为两种阶段：map阶段和reduce阶段。map方法接收一个<key,value>形式的输入，key表示词组偏移量，value表示该偏移量对应的值，然后同样产生一个<key,value>形式的中间输出，Hadoop函数接收一个如<key,(list of values)>形式的输入，list of values即Java中的集合，用于表示相同偏移量下的字符统计数量，然后对这个value集合进行处理，每个Reduce方法产生0或1个输出，reduce的输出也是<key,value>形式的。

如果文本为

Hadoop Hello Hadoop

实际过程大概为：

Mapping：

<Hadoop ,1>，<Hello,1>，<Hadoop ,1>

Shuffling：

<Hadoop ,1>，<Hadoop ,1>
<Hello,1>

Reducing：

<Hadoop ,2>
<Hello,1>

多节点下的并行计算原理示意图

开发代码

查看源码

通过查看Mapper类的源码，发现官方已经定义的run方法，包含了所有基础的Mapper操作（模板方法设计模式），主要操作在map中，因此，只要重写map方法即可。

同理Reduce类中也有类似的定义

重写方法

这里只列举部分代码，其余所有代码可去本人的GitHub中查看：GitHub链接 /** * Map：读取输入文件 * Text:类似字符串 */ public static class MyMapper extends Mapper<LongWritable, Text, Text, LongWritable> {

 LongWritable one = new LongWritable(1);

 /**
 * @param key 偏移量
 * @param value 每行的字符串
 * @param context 上下文
 * @throws IOException
 * @throws InterruptedException
 */
 @Override
 protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
 /*super.map(key, value, context);*/

 //接收到每一行数据
 String line = value.toString();

 //按照指定分隔符进行拆分
 /*line.split("\t");//以Tab分隔*/
 String[] words = line.split(" ");//以空格分隔

 for (String word : words) {
 //通过上下文把map的处理结果输出
 context.write(new Text(word), one);
 }
 }
}

运行

通过宝塔面板上传到服务器，确保此时Hadoop所有服务均为启动状态

在HDFS中预先创建好要测试的文件，通过绝对路径的方法hadoop fs -ls hdfs://192.168.79.129:8020/确认文件是可以访问的

输入运行命令即可运行

 hadoop jar hadoopstudy-1.0-SNAPSHOT.jar com.fjy.hadoop.mapreduce.WordCountApp hdfs://192.168.79.129:8020/hello.txt hdfs://192.168.79.129:8020/output/wc

启动至FileInputFormat

 [hadoop@localhost testFile]$ hadoop jar hadoopstudy-1.0-SNAPSHOT.jar com.fjy.hadoop.mapreduce.WordCountA2.168.79.129:8020/hello.txt hdfs://192.168.79.129:8020/output/wc
 18/04/10 09:57:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 18/04/10 09:57:44 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
 18/04/10 09:57:45 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
 18/04/10 09:57:45 INFO input.**FileInputFormat**: Total input paths to process : 1

执行Job

 18/04/10 09:57:46 INFO mapreduce.JobSubmitter: number of splits:1
 18/04/10 09:57:46 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1523322221448_0001
 18/04/10 09:57:47 INFO impl.YarnClientImpl: Submitted application application_1523322221448_0001
 18/04/10 09:57:47 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1523322221448_0001/
 18/04/10 09:57:47 INFO mapreduce.Job: Running job: job_1523322221448_0001
 18/04/10 09:57:56 INFO mapreduce.Job: Job job_1523322221448_0001 running in uber mode : false
 18/04/10 09:57:56 INFO mapreduce.Job: map 0% reduce 0%
 18/04/10 09:58:04 INFO mapreduce.Job: map 100% reduce 0%
 18/04/10 09:58:10 INFO mapreduce.Job: map 100% reduce 100%
 18/04/10 09:58:12 INFO mapreduce.Job: Job job_1523322221448_0001 completed successfully
 18/04/10 09:58:12 INFO mapreduce.Job: Counters: 49
 File System Counters
 FILE: Number of bytes read=39
 FILE: Number of bytes written=222915
 FILE: Number of read operations=0
 FILE: Number of large read operations=0
 FILE: Number of write operations=0
 HDFS: Number of bytes read=113
 HDFS: Number of bytes written=17
 HDFS: Number of read operations=6
 HDFS: Number of large read operations=0
 HDFS: Number of write operations=2
 Job Counters

Map方法执行

 Launched map tasks=1
 Launched reduce tasks=1
 Data-local map tasks=1
 Total time spent by all maps in occupied slots (ms)=5920
 Total time spent by all reduces in occupied slots (ms)=3321
 Total time spent by all map tasks (ms)=5920
 Total time spent by all reduce tasks (ms)=3321
 Total vcore-seconds taken by all map tasks=5920
 Total vcore-seconds taken by all reduce tasks=3321
 Total megabyte-seconds taken by all map tasks=6062080
 Total megabyte-seconds taken by all reduce tasks=3400704
 Map-Reduce Framework
 Map input records=1
 Map output records=2
 Map output bytes=29
 Map output materialized bytes=39

Reduce 方法执行

 Input split bytes=101
 Combine input records=0
 Combine output records=0

 Reduce input groups=2
 Reduce shuffle bytes=39
 Reduce input records=2
 Reduce output records=2
 Spilled Records=4
 Shuffled Maps =1
 Failed Shuffles=0
 Merged Map outputs=1
 GC time elapsed (ms)=115
 CPU time spent (ms)=1080
 Physical memory (bytes) snapshot=300331008
 Virtual memory (bytes) snapshot=5493092352
 Total committed heap usage (bytes)=165810176

Shuffle 方法执行

 Shuffle Errors
 BAD_ID=0
 CONNECTION=0
 IO_ERROR=0
 WRONG_LENGTH=0
 WRONG_MAP=0
 WRONG_REDUCE=0
`File Input`和`File Output` 方法执行 
 File Input Format Counters
 Bytes Read=12
 File Output Format Counters
 Bytes Written=17

运行结果如图

进一步优化

使用Combiner

使用Combiner组件可以在Mapping时就先进行一次聚合，再发送到Shuffling继续运算，从而提高运算效率在主方法中进行如下配置即可

 //通过job设置combiner处理类，逻辑上与reduce一致，注意，如果要计算平均数等不能使用Combiner!
 job.setCombinerClass(MyReducer.class);

编写完后使用Maven重新打包

 mvn clean package -DskipTests

上传到服务器后再次执行计算命令

 hadoop jar hadoopstudy-0.3.jar com.fjy.hadoop.mapreduce.WordCountApp hdfs://192.168.79.129:8020/hello.txt hdfs://192.168.79.129:8020/output/wc

查看图形界面可观察是否完成

通过查看代码可发现Combine input records已经不为0(之前未使用Combiner 时该值为0)，说明使用成功

 Map-Reduce Framework
 Map input records=4
 ...
 Combine input records=9
 Combine output records=5
 ...
 Physical memory (bytes) snapshot=298844160
 Virtual memory (bytes) snapshot=5493088256
 Total committed heap usage (bytes)=165810176
 Shuffle Errors
 BAD_ID=0
 ...

查看结果

 [hadoop@localhost testFile]$ hadoop fs -cat /output/wc/part-r-00000
 18/04/10 23:06:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 ! 1
 Hadoop 3
 Hello 2
 MapReduce 2
 YARN 1

使用Partitioner组件

作用

Partitioner决定MapTask输出的数据交由哪个ReduceTask处理，其默认实现为分发的key的hash值对ReduceTask个数取模

文件准备

在宝塔测试目录/www/hadoop/testFile下准备好PartitionerTest.txt文件，文件内容如图，将其上传到HDFS根目录中

 [hadoop@localhost root]$ cd /www/hadoop/testFile
 [hadoop@localhost testFile]$ ls
 PartitionerTest.txt hello.txt
 [hadoop@localhost testFile]$ hadoop fs -put PartitionerTest.txt /
 18/04/11 00:30:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 [hadoop@localhost testFile]$ hadoop fs -ls /
 18/04/11 00:30:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 Found 7 items
 -rw-r--r-- 1 hadoop supergroup 73 2018-04-11 00:30 /PartitionerTest.txt
 drwxr-xr-x - hadoop supergroup 0 2018-04-08 10:56 /hdfsapi
 -rw-r--r-- 1 hadoop supergroup 60 2018-04-10 23:03 /hello.txt
 drwxr-xr-x - hadoop supergroup 0 2018-04-10 23:05 /output
 drwxr-xr-x - hadoop supergroup 0 2018-04-08 10:35 /test
 drwx------ - hadoop supergroup 0 2018-04-09 20:11 /tmp
 drwxr-xr-x - hadoop supergroup 0 2018-04-09 20:11 /user
 [hadoop@localhost testFile]$ hadoop fs -text /PartitionerTest.txt
 18/04/11 00:30:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 xiaomi 200
 huawei 300
 xiaomi 100
 huawei 200
 iphone 300
 iphone 200
 sony 50

代码编写

主要对分隔符和上下文内容进行修改，map类中修改为

 //每一个空格是一个手机品牌，另一个是销售数量
 context.write(new Text(words[0]), new LongWritable(Long.parseLong(words[1])));

新增Partitioner处理类

 /**
 * Partitioner处理类
 */
 public static class MyPartitioner extends Partitioner<Text,LongWritable>{
 @Override
 public int getPartition(Text key, LongWritable value, int i) {

 if ("xiaomi".equals(key.toString())) {
 return 0;//若为xiaomi则交由0 ReduceTask处理
 }
 if ("huawei".equals(key.toString())) {
 return 1;//若为huawei则交由1 ReduceTask处理
 }
 if ("iphone".equals(key.toString())) {
 return 2;//若为iphone则交由2 ReduceTask处理
 }

 return 3;//若为其他，则交由3 ReduceTask处理
 }
 }

最后在驱动中配置Partitioner

 //设置job的Partition
 job.setPartitionerClass(MyPartitioner.class);
 //设置四个reducer，每个分区一个，否则Partitioner配置不生效
 job.setNumReduceTasks(4);

通过复制类名全路径（以IDEA为例），修改运行命令

此时，运行命令为

 hadoop jar hadoopstudy-0.4.jar com.fjy.hadoop.mapreduce.WordCountPartitionerApp hdfs://192.168.79.129:8020/PartitionerTest.txt hdfs://192.168.79.129:8020/output/partitioner

执行

再次用Maven打包并上传到测试目录运行运行前：

运行中：

运行结束

查看结果内容：

 [hadoop@localhost testFile]$ hadoop fs -ls /output/partitioner
 18/04/11 00:36:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 Found 5 items
 -rw-r--r-- 1 hadoop supergroup 0 2018-04-11 00:35 /output/partitioner/_SUCCESS
 -rw-r--r-- 1 hadoop supergroup 11 2018-04-11 00:35 /output/partitioner/part-r-00000
 -rw-r--r-- 1 hadoop supergroup 11 2018-04-11 00:35 /output/partitioner/part-r-00001
 -rw-r--r-- 1 hadoop supergroup 11 2018-04-11 00:35 /output/partitioner/part-r-00002
 -rw-r--r-- 1 hadoop supergroup 8 2018-04-11 00:35 /output/partitioner/part-r-00003
 [hadoop@localhost testFile]$ hadoop fs -text /output/partitioner/part-r-00000
 18/04/11 00:36:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 xiaomi 300
 [hadoop@localhost testFile]$ hadoop fs -text /output/partitioner/part-r-00001
 18/04/11 00:36:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 huawei 500
 [hadoop@localhost testFile]$ hadoop fs -text /output/partitioner/part-r-00002
 18/04/11 00:37:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 iphone 500
 [hadoop@localhost testFile]$ hadoop fs -text /output/partitioner/part-r-00003
 18/04/11 00:37:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 sony 50

结果正确！

开启JobHistory

JobHistory作用

记录已经运行完的MapReduce信息到指定的HDFS目录下，但默认不开启因此此时不配置直接访问如下

![jobhistory][33]

配置JobHistory

在mapred-site.xml加入配置

 <!-- jobhistory地址-->
	<property>
		<name>mapreduce.jobhistory.address</name>
		<value>192.168.79.129:10020</value>
		<description>MapReduce JobHistory Server IPC host:port</description>
	</property>
 <!-- jobhistory web的地址-->
	<property>
		<name>mapreduce.jobhistory.webapp.address</name>
		<value>192.168.79.129:19888</value>
		<description>MapReduce JobHistory Server Web UI host:port</description>
	</property>
 <!-- 作业运行完存放地址-->
	<property>
		<name>mapreduce.jobhistory.done-dir</name>
		<value>/history/done</value>
	</property>
 <!-- 作业运行中存放地址-->
	<property>
		<name>mapreduce.jobhistory.intermediate-done-dir</name>
		<value>/history/done_intermediate</value>
	</property>

配置完后重启YARN

 [hadoop@localhost mapreduce]$ cd /www/hadoop/hadoop-2.6.0-cdh5.7.0/sbin
 [hadoop@localhost sbin]$ ./stop-yarn.sh
 stopping yarn daemons
 stopping resourcemanager
 localhost: stopping nodemanager
 no proxyserver to stop
 [hadoop@localhost sbin]$ jps
 2353 DataNode
 80224 Jps
 2228 NameNode
 2522 SecondaryNameNode
 [hadoop@localhost sbin]$ ./start-yarn.sh
 starting yarn daemons
 starting resourcemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-localhost.localdomain.out
 localhost: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-localhost.localdomain.out
 [hadoop@localhost mapreduce]$ jps
 2353 DataNode
 80336 ResourceManager
 2228 NameNode
 81510 Jps
 2522 SecondaryNameNode
 63582 JobHistoryServer
 80447 NodeManager

启动jobhistory服务

 [hadoop@localhost sbin]$ ./mr-jobhistory-daemon.sh start historyserver
 starting historyserver, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/mapred-hadoop-historyserver-localhost.localdomain.out

运行PI测试

 [hadoop@localhost sbin]$ cd /www/hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce
 [hadoop@localhost mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 1 2

访问报错

进入日志查看，发现仍然无法访问

报未开启聚合的错误，同时发现指向跳转链接为localhost

增加配置

因此在yarn-site.xml加入配置

 <!-- 开启YARN log的聚合功能-->
	<property>
		<name>yarn.log-aggregation-enable</name>
		<value>true</value>
	</property>

同时，为了让虚拟机的跳转链接不为localhost，此处再加入一段配置，制定IP

 <property> 
 <name>yarn.nodemanager.hostname</name> 
 <value>192.168.79.129</value> 
 </property>

重启服务

再次重启YARN，并同时重启jobhistory服务

 [hadoop@localhost sbin]$ ./mr-jobhistory-daemon.sh stop historyserver
 stopping historyserver
 [hadoop@localhost sbin]$ ./stop-yarn.sh
 stopping yarn daemons
 stopping resourcemanager
 localhost: stopping nodemanager
 no proxyserver to stop
 [hadoop@localhost sbin]$ ./start-yarn.sh
 starting yarn daemons
 starting resourcemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-localhost.localdomain.out
 localhost: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-localhost.localdomain.out
 [hadoop@localhost sbin]$ ./mr-jobhistory-daemon.sh start historyserver
 starting historyserver, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/mapred-hadoop-historyserver-localhost.localdomain.out

【注意】此时若只重启YARN就算配置正确，仍然可能出现Aggregation is not enabled的错误

验证

再次执行PI运算

 [hadoop@localhost sbin]$ cd /www/hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce
 [hadoop@localhost mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 1 2

查看Logs记录

已经有日志记录，配置成功！

YARN单节点伪分布式搭建

Mon, 09 Apr 2018 20:14:00 +0800

配置YARN

首先启动之前配置好的HDFS服务，然后进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/etc/hadoop目录，按照官方指导配置YARN单节点环境

使用cp mapred-site.xml.template mapred-site.xml命令把mapred-site.xml.template复制一份，加入官方的配置 mapreduce.framework.name yarn

同时对yarn-site.xml也进行配置 yarn.nodemanager.aux-services mapreduce_shuffle

启动YARN

进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/sbin目录，使用./start-yarn.sh命令启动YARN服务

[hadoop@localhost sbin]$ ./start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-localhost.localdomain.out
localhost: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-localhost.localdomain.out

用jps命令查看是否启动成功，看到如下显示即为启动成功，主要是NodeManager和ResourceManager两个服务

[hadoop@localhost sbin]$ jps
7056 SecondaryNameNode
7475 NodeManager
6775 NameNode
6872 DataNode
7374 ResourceManager
7679 Jps

开放端口

YARN的浏览器客户端占用8088端口，故使用宝塔面板开放8088端口

访问YARN图形化界面

使用浏览器打开http://ip:8088/可见如下界面，则YARN启动成功

测试

YARN官方提供了示例java程序来测试资源调度，进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce目录，可见

[hadoop@localhost mapreduce]$ ls
hadoop-mapreduce-client-app-2.6.0-cdh5.7.0.jar hadoop-mapreduce-client-nativetask-2.6.0-cdh5.7.0.jar
hadoop-mapreduce-client-common-2.6.0-cdh5.7.0.jar hadoop-mapreduce-client-shuffle-2.6.0-cdh5.7.0.jar
hadoop-mapreduce-client-core-2.6.0-cdh5.7.0.jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar
hadoop-mapreduce-client-hs-2.6.0-cdh5.7.0.jar lib
hadoop-mapreduce-client-hs-plugins-2.6.0-cdh5.7.0.jar lib-examples
hadoop-mapreduce-client-jobclient-2.6.0-cdh5.7.0.jar sources
hadoop-mapreduce-client-jobclient-2.6.0-cdh5.7.0-tests.jar

其中hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar就是MapReduce的测试程序，通过hadoop jar的方式来运行用命令查看可配置的参数hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar

[hadoop@localhost mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar
An example program must be given as the first argument.
Valid program names are:
 aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
 aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
 bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
 dbcount: An example job that count the pageview counts from a database.
 distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
 grep: A map/reduce program that counts the matches of a regex in the input.
 join: A job that effects a join over sorted, equally partitioned datasets
 multifilewc: A job that counts words from several files.
 pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
 pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
 randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
 randomwriter: A map/reduce program that writes 10GB of random data per node.
 secondarysort: An example defining a secondary sort to the reduce.
 sort: A map/reduce program that sorts the data written by the random writer.
 sudoku: A sudoku solver.
 teragen: Generate data for the terasort
 terasort: Run the terasort
 teravalidate: Checking results of terasort
 wordcount: A map/reduce program that counts the words in the input files.
 wordmean: A map/reduce program that counts the average length of the words in the input files.
 wordmedian: A map/reduce program that counts the median length of the words in the input files.
 wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.
[hadoop@localhost mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi
Usage: org.apache.hadoop.examples.QuasiMonteCarlo <nMaps> <nSamples>
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|resourcemanager:port> specify a ResourceManager
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

从上可知测试程序可以做诸如数学π、随机文件写入等操作，这里做数学π的运算使用命令hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 2 3做两个map计算精度为3的数学π运算,运算结束后会显示结果如下

Job Finished in 26.996 seconds
Estimated value of Pi is 4.00000000000000000000

同时浏览器也可看到作业情况

HDFS伪分布式单节点搭建

Sat, 07 Apr 2018 13:01:00 +0800

环境

系统：CentOS Linux 7.4.1708 (Core) 客户端：putty 服务器：虚拟机1core、2G RAM 同时搭建了宝塔面板（自动配置了java8）

Hadoop下载

CDH版本Hadoop下载

为了统一版本，全部使用hadoop-2.6.0-cdh5.7.0

点击可进入官方推荐配置页

解压并配置

将下载好的文件通过宝塔上传到服务器，这里选择上传到/www/hadoop目录

解压tar包

配置

新建用户

新建hadoop用户，忽略密码强度提示

 adduser hadoop
 passwd hadoop

为方便远程连接和学习，同时授予root权限，输入whereis sudoers查看权限控制文件位置

 [root@fjy ~]# whereis sudoers
 sudoers: /etc/sudoers /etc/sudoers.d /usr/libexec/sudoers.so /usr/share/man/man5/sudoers.5.gz

使用宝塔找到并修改/etc/sudoers文件

配置新用户免密码远程登录

切换至hadoop用户，使用 ssh-keygen -t rsa生成私钥信息，位置在/home/hadoop/，并且忽略所有提示输入的信息

查看生成的文件，该SSH文件夹为隐藏文件夹，故用ls -a查看，可见自动生成了.ssh目录和id_rsa,id_rsa.pub两个文件

复制公钥到.ssh目录

 cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys

查看JAVA目录

因为是宝塔自动配置的java8，所以要首先查看当前java环境变量位置，使用如下命令

 update-alternatives --config java

此时宝塔自动将java配置到/usr/java/jdk1.8.0_121/jre/bin/java目录即/usr/java/jdk1.8.0_121为java环境变量的目录

配置hadoop的java目录

使用宝塔进入hadoop目录下的etc/hadoop/hadoop-env.sh，对应此处的绝对路径应为/www/hadoop/hadoop-2.6.0-cdh5.7.0/etc/hadoop 使用宝塔编辑文件，加入java环境变量位置

配置分布式文件参数

打开core-site.xml，配置如下

 <configuration>
 <property>
 <name>fs.defaultFS</name>
 <!-- 配置主机IP-->
 <value>hdfs://192.168.79.129:8020</value>
 </property>
 
 <!-- 配置临时文件位置-->
 <property>
 <name>hadoop.tmp.dir</name>
 <value>/www/hadoop/tmp</value>
 </property>
 </configuration>

打开hdfs-site.xml，配置如下

 <configuration>
 <property>
 <name>dfs.replication</name>
 <!-- 配置副本数量-->
 <value>1</value>
 </property>
 </configuration>

启动

初始化

进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/bin目录，使用如下语句初始化Hadoop，该操作仅执行一次即可，以后使用无需重复执行，否则执行后会删除当前hadoop上的所有文件

 hadoop namenode -format

滚动均为info级别的日志

可看到已初始化成功

启动HDFS

切换到hadoop用户，进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/sbin目录

输入./start-dfs.sh启动HDFS服务

【注意】，要提前在root用户下对/www/hadoop授予hadoop用户读写权限，否则无法启动

使用jps查看当前的java进程，若出现如下四个进程表示HDFS服务启动成功

外部访问浏览器客户端

首先使用宝塔开放50070端口

使用外部浏览器访问ip:50070可查看如下页面，则表示HDFS已成功启动

hdfs命令

配置hdfs命令

使用vi ~/.bash_profile进入配置文件，按如下配置

 export HADOOP_HOME=/www/hadoop/hadoop-2.6.0-cdh5.7.0
 export PATH=$HADOOP_HOME/bin:$PATH

使配置文件生效source ~/.bash_profile 输入hdfs回车测试命令是否生效，若出现如下画面则表示命令已经生效

命令测试

先在本地创建hello.txt文件，使用 hadoop fs -put hello.txt /命令将文件推送到hdfs根目录下

查看服务器根目录hadoop fs -ls /，已经有该文件，hadoop伪分布式配置完成