Hadoop伪分布式安装 基础环境 Hadoop：版本2.7.3，下载地址 HBase：版本1.2.4，下载地址 JDK8 jdk 安装命令 $ yum -y install java-devel java 查看当前系统jre和jdk路径，输入update-alternatives --config java和update-alternatives --config javac $ update-alternatives --config java 共有 1 个提供“java”的程序。 选项 命令 ----------------------------------------------- *+ 1 java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/bin/java) $ update-alternatives --config javac 共有 1 个提供“javac”的程序。 选项 命令 ----------------------------------------------- *+ 1 java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/bin/javac) 查看环境 $ ll 总用量 311132 -rw-------. 1 root root 1241 8月 11 10:53 anaconda-ks.cfg -rw-r--r--. 1 root root 214092195 8月 26 2016 hadoop-2.7.3.tar.gz -rw-r--r--. 1 root root 104497899 10月 26 2016 hbase-1.2.4-bin.tar.gz $ java -version openjdk version "1.8.0_191" OpenJDK Runtime Environment (build 1.8.0_191-b12) OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode) $ javac -version javac 1.8.0_191 免密登录 新建hadoop用户，忽略密码强度提示 $ adduser hadoop $ passwd hadoop 为方便远程连接和学习，同时授予root权限，输入whereis sudoers查看权限控制文件位置 $ whereis sudoers sudoers: /etc/sudoers /etc/sudoers.d /usr/libexec/sudoers.so /usr/share/man/man5/sudoers.5.gz vi /etc/sudoers修改内容为 root ALL=(ALL) ALL hadoop ALL=(ALL) ALL 切换至hadoop用户，使用 ssh-keygen -t rsa生成私钥信息，位置在/home/hadoop/，并且忽略所有提示输入的信息 查看生成的文件，该SSH文件夹为隐藏文件夹，故用ls -a查看，可见自动生成了.ssh目录和id_rsa,id_rsa.pub两个文件 $ ls /home/hadoop/.ssh/ id_rsa id_rsa.pub 复制公钥到.ssh目录 $ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys 免密测试 $ ssh localhost The authenticity of host 'localhost (::1)' can't be established. ECDSA key fingerprint is SHA256:3UEr3lx2jznmN3FL3SflViR05IZe6AWweb3TeSsfX0M. ECDSA key fingerprint is MD5:0f:31:c1:cc:54:69:7f:d8:8d:7c:8c:22:95:2e:03:4e. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts. Last login: Sat Dec 15 11:14:27 2018 $ exit 登出 Connection to localhost closed. 安装Hadoop 解压文件 $ tar -zxvf hadoop-2.7.3.tar.gz 修改配置文件 $ vi /hadoop-2.7.3/etc/hadoop/hadoop-env.sh 配置Java路径 # The java implementation to use. export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64 修改HDFS配置 $ vi /hadoop-2.7.3/etc/hadoop/hdfs-site.xml 配置内容 dfs.replication 1 dfs.namenode.name.dir file:/home/hadoop/hadoop_data/dfs/name dfs.namenode.data.dir file:/home/hadoop/hadoop_data/dfs/data 手动创建相关目录 $ mkdir -p /home/hadoop/hadoop_data/dfs/name $ mkdir -p /home/hadoop/hadoop_data/dfs/data 配置core-site.xml，内容为 fs.defaultFS hdfs://0.0.0.0:8020 hadoop.tmp.dir file:/home/hadoop/hadoop_data 放行端口并重载 $ firewall-cmd --zone=public --add-port=8020/tcp --add-port=50070/tcp --permanent $ firewall-cmd --reload 启动Hadoop 初始化 进入/hadoop-2.7.3/bin目录，使用如下语句初始化Hadoop，该操作仅执行一次即可，以后使用无需重复执行，否则执行后会删除当前hadoop上的所有文件 $ hadoop namenode -format 滚动均为info级别的日志并且status 0表示已初始化成功 ... 18/12/15 17:07:32 INFO namenode.FSImage: Allocated new BlockPoolId: BP-99731851-127.0.0.1-1544864852133 18/12/15 17:07:32 INFO common.Storage: Storage directory /home/hadoop/hadoop_data/dfs/name has been successfully formatted. 18/12/15 17:07:32 INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/hadoop_data/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression 18/12/15 17:07:32 INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/hadoop_data/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 353 bytes saved in 0 seconds. 18/12/15 17:07:32 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 18/12/15 17:07:32 INFO util.ExitUtil: Exiting with status 0 18/12/15 17:07:32 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1 ************************************************************/ 启动HDFS 进入/hadoop-2.7.3/sbin目录输入./start-dfs.sh启动HDFS服务 $ ./start-dfs.sh Starting namenodes on [0.0.0.0] The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established. ECDSA key fingerprint is SHA256:3UEr3lx2jznmN3FL3SflViR05IZe6AWweb3TeSsfX0M. ECDSA key fingerprint is MD5:0f:31:c1:cc:54:69:7f:d8:8d:7c:8c:22:95:2e:03:4e. Are you sure you want to continue connecting (yes/no)? yes 0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts. 0.0.0.0: starting namenode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-namenode-localhost.localdomain.out localhost: starting datanode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-datanode-localhost.localdomain.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-secondarynamenode-localhost.localdomain.out 进程验证，若出现如下四个进程表示HDFS服务启动成功 $ jps 28930 Jps 28821 SecondaryNameNode 28663 DataNode 28541 NameNode 配置hdfs命令 使用vi ~/.bash_profile进入配置文件，按如下配置 export HADOOP_HOME=/home/hadoop/hadoop-2.7.3 export PATH=$HADOOP_HOME/bin:$PATH 使配置文件生效source ~/.bash_profile 输入hdfs回车测试命令是否生效，若出现如下画面则表示命令已经生效 $ hdfs Usage: hdfs [--config confdir] [--loglevel loglevel] COMMAND where COMMAND is one of: dfs run a filesystem command on the file systems supported in Hadoop. classpath prints the classpath namenode -format format the DFS filesystem secondarynamenode run the DFS secondary namenode namenode run the DFS namenode journalnode run the DFS journalnode zkfc run the ZK Failover Co ... 命令测试 $ hadoop fs -mkdir /test $ hadoop fs -ls / Found 1 items drwxr-xr-x - hadoop supergroup 0 2018-12-15 17:14 /test HBase伪分布式搭建 HBase安装 解压安装包，目录如下 $ ll 总用量 320 drwxr-xr-x. 4 hadoop hadoop 4096 1月 29 2016 bin -rw-r--r--. 1 hadoop hadoop 122439 10月 26 2016 CHANGES.txt drwxr-xr-x. 2 hadoop hadoop 178 1月 29 2016 conf drwxr-xr-x. 12 hadoop hadoop 4096 10月 26 2016 docs drwxr-xr-x. 7 hadoop hadoop 80 10月 26 2016 hbase-webapps -rw-rw-r--. 1 hadoop hadoop 261 10月 26 2016 LEGAL drwxrwxr-x. 3 hadoop hadoop 8192 12月 15 11:24 lib -rw-rw-r--. 1 hadoop hadoop 130696 10月 26 2016 LICENSE.txt -rw-rw-r--. 1 hadoop hadoop 42025 10月 26 2016 NOTICE.txt -rw-r--r--. 1 hadoop hadoop 1477 12月 27 2015 README.txt 拷贝Hadoop配置文件到/conf目录 $ cp /home/hadoop/hadoop-2.7.3/etc/hadoop/hdfs-site.xml . $ cp /home/hadoop/hadoop-2.7.3/etc/hadoop/core-site.xml . 修改hbase-env.sh文件，增加JDK路径 # The java implementation to use. Java 1.7+ required. export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64 由于使用的是jdk8，主要注释内存配置 # Configure PermSize. Only needed in JDK7. You can safely remove it for JDK8+ #export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m" #export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m" 另外打开文件最后zookeeper配置，设置为使用内置的zookeeper # Tell HBase whether it should manage it's own instance of Zookeeper or not. export HBASE_MANAGES_ZK=true 修改hbase-site.xml文件，内容如下 hbase.rootdir hdfs://localhost:8020/hbase hbase.zookeeper.property.dataDir /home/hadoop/hadoop_data/zookeeper hbase.cluster.distributed true 参考文档 启动HBase 在/bin目录下执行./start-hbase.sh启动HBase $ ./start-hbase.sh OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N localhost: starting zookeeper, logging to /home/hadoop/hbase-1.2.4/bin/../logs/hbase-hadoop-zookeeper-localhost.localdomain.out localhost: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N starting master, logging to /home/hadoop/hbase-1.2.4/bin/../logs/hbase-hadoop-master-localhost.localdomain.out OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N starting regionserver, logging to /home/hadoop/hbase-1.2.4/bin/../logs/hbase-hadoop-1-regionserver-localhost.localdomain.out 由于虚拟机只分配了单核单线程，GC警告暂时忽略 进程验证，确认启动成功 $ jps 28821 SecondaryNameNode 28663 DataNode 29911 HRegionServer 29785 HMaster 29724 HQuorumPeer 28541 NameNode 30303 Jps 进入HBase命令行测试 $ ./hbase shell OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/hadoop/hbase-1.2.4/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] HBase Shell; enter 'help' for list of supported commands. Type "exit" to leave the HBase Shell Version 1.2.4, r67592f3d062743907f8c5ae00dbbe1ae4f69e5af, Tue Oct 25 18:10:20 CDT 2016 hbase(main):001:0> status 1 active master, 0 backup masters, 1 servers, 0 dead, 2.0000 average load hbase(main):002:0> 查看其是否自动在HDFS上创建目录/hbase $ hadoop fs -ls / Found 2 items drwxr-xr-x - hadoop supergroup 0 2018-12-15 18:43 /hbase drwxr-xr-x - hadoop supergroup 0 2018-12-15 17:14 /test
概述 HBase 简介 Hbase是一个分布式的、面向列的开源数据库 列式数据库：将列值统一存储 行式数据库：关系型数据库，将每一行值统一存储 Hbase在Hadoop之上提供了类似于Bigtable的能力 Bigtable是压缩、高性能、高可扩展性的基于Google GFS文件系统的数据库，用于存储大规模的结构化数据，在扩展性和性能上有很大的优势 Hbase不同于一般的关系数据库，它适合非结构化数据存储 非结构化数据：不能用二维表进行存储的数据，例如图片、文档等 Hbase地位 Apache基金会顶级项目 基于Hadoop核心HDFS系统进行数据存储，类似于Hive 可以存储超大数据并适合用来进行大数据的的实时查询 Hbase与HDFS Hbase建立在Hadoop文件系统上，利用了Hadoop文件系统的容错能力 Hbase提供对数据随机实时读/写访问功能 Hbase内部使用哈希表存储索引，可将HDFS文件中的数据进行快速查找 适用场景 瞬间写入量很大，常用数据库不好支持或者需要很高的成本支撑 数据需要长久保存，且数据量会持久增长到比较大的场景 Hbase不适合用于有join、多级索引、表关系复杂的数据模型 存储模型与原理 CAP定理 分布式系统的CAP理论：理论首先把分布式系统中的三个特性进行了如下归纳： 一致性（C）：所有节点在同一时间具有相同的数据 可用性（A）：保证每个请求不管成功或者失败（包含节点宕机）都有响应，但不保证获取数据的正确性 分区容错性（P）：系统中任意信息的丢失或失败不会影响系统的继续运作。系统如果不能在时限内达成数据一致性，必须就当前操作在C和A之间做出选择。 Hbase为CP特性 ACID ACID的定义： Atomic原子性:所有的步骤要么全部完成要么一个也不会完成 Consistent一致性: 通过各种途径包括外键约束等任何写入数据库的数据都是有效的，不能发生表与表之间存在外键约束 Isolated隔离性: 一个未完成事务不会影响另外一个未完成事务 Durable持久性: 一旦一个事务被提交，它应该持久保存 HBase非强ACID HBase概念 字段 概念 备注 NameSpace 类似RDBMS的“数据库” Table 表名必须是能用在文件路径里的合法名字 原因：每个表都会映射为HDFS的文件 Row 在表里，每一行代表一个数据对象，每一行都以一个行键（Row Key）来唯一标识的，行键没有特定的数据类型，以二进制的字节进行存储 Row Key 唯一标识一行记录，不可改变，只能删除 Column 由Column family和Column qualifier组成，由（：）进行分隔 例如：family：qualifier Column family 在定义HBase表的时候需要提前设置好列族，表中所有的列都需要组织在列族里面 同一个列族的成员有相同的前缀，物理上，一个列族的成员都是储存在一起，用于存储优化 Column qualifier（列限定符） 列族中的数据通过列标识来进行映射，可以理解为一个键值对，Column qualifier就是Key 类似具体的列名 Cell 每一个行键，列族和列标识共同组成的一个单元 无特定的数据类型，以二进制字节存储 Timestamp 每个值都有一个Timestamp，作为该值特定版本的标识符 读未指定版本则返回最新版，写未指定时间则使用当前时间，HBase默认保留3个版本数据 HBase与传统关系数据库的区别 Hbase RDBMS 数据库大小 PB GB、TB 数据类型 Bytes 丰富的数据类型 事务支持 ACID只支持单个Row级别 全面的ACID支持 索引 只支持Row-Key 支持 吞吐量 百万查询/每秒 数千查询/每秒 HBase基础架构 HBase分布式系统简介 HMaster HBase主/从集群架构中的中央节点 负责将region（HBase中存储最小单元，表格的基本单位）分配给RegionServer，协调RegionServer的负载并维护集群状态 维护表和Region的元数据，不参与数据的输入/输出过程 RegionServer 维护HMaster分配给他的region，处理对这些region的IO请求 负责切分正在运行过程中变得过大的region Zookeeper 集群协调器，保证至少一个节点出于active状态 HMaster启动将系统表加载到ZK（用于服务发现） 提供HBase RegionServer状态信息 HBase原理 HBase写流程 Client先访问zookeeper，得到对应的RegionServer地址 Client对RegionServer发起写请求，RegionServer接受数据写入内存 当MemStore的大小达到一定值后，flush到StoreFile并存储到HDFS HBase特殊点在于其先写内存再写日志，通过类似MySQL中MVCC机制保证一致性 HBase读流程 Client先访问zookeeper，得到对应的RegionServer地址 Client对RegionServer发起读请求 当RegionServer收到client的读请求后，先扫描自己的Memstore，再扫描BlockCache（加速读内容缓存区）如果还没找到则从StoreFile中读取数据，然后将数据返回给Client 读特点 读过程与HMaster无关，只和ZK有关，有效的减小HMaster负载 HBase模块协作 有关HBase的三个问题 HBase启动时发生了什么 当RegionServer失效后会发生什么 当HMaster失效后会发生什么 HBase启动 HBase启动，注册到zookeeper，等待RegionServer汇报 RegionServer注册到zookeeper，并向HMaster汇报 对各个RegionServer（包括失效的）的数据进行整理，分配Region和meta信息（所有表的索引） RegionServer失效 HMaster将失效RegionServer上的Region分配到其他节点 HMaster更新HBase：meta表保证数据正常访问 HMaster失效 配置高可用后 出于Backup状态的其他HMaster节点选举出一个转为Active状态 未配置高可用 数据能正常读写，但不能创建删除表，也不能更改表结构，因为涉及meta表更新
DataNode没启动 这种情况多出现于该机器之前配置过hadoop，没清理干净的情况，即仍有文件目录，导致新hadoop的clusterID与旧的不一致，导致HDFS无法启动，报JavaIO异常 2018-04-22 00:21:15,868 WARN org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /www/hadoop/app/tmp/dfs/data: namenode clusterID = CID-6c7766c6-b260-42b6-8c03-07dda802c0cf; datanode clusterID = CID-a53df7ad-e593-4a29-ad15-a93486c54776 2018-04-22 00:21:15,868 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool (Datanode Uuid unassigned) service to hadoop0/192.168.79.129:8020. Exiting. java.io.IOException: All specified directories are failed to load. at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1394) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1355) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:228) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:829) at java.lang.Thread.run(Thread.java:745) 2018-04-22 00:21:15,869 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool (Datanode Uuid unassigned) service to hadoop0/192.168.79.129:8020 解决方案 清理所有节点的旧文件存储目录，重新再根节点格式化存储空间 rm -rf app/ hdfs namenode -format 重新启动HDFS [hadoop@hadoop0 bin]$ cd ../sbin [hadoop@hadoop0 sbin]$ ./start-all.sh 查看进程，DataNode已启动 [hadoop@hadoop0 sbin]$ jps 37618 DataNode 37783 SecondaryNameNode 37932 ResourceManager 37485 NameNode 38045 NodeManager 38365 Jps NodeManager没启动/启动后进程消失 该情况由于yarn-site.xml文件配置有误，导致从节点找不到主节点，报JavaNet异常 org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.BindException: Problem binding to [hadoop0:0] java.net.BindException: 无法指定被请求的地址; For more details see: http://wiki.apache.org/hadoop/BindException at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:139) at org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65) at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:406) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:266) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:475) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:521) 解决方案 重新配置yarn-site.xml，确保其有下面这一段，并且使用SSH 主机名能够访问该节点 yarn.resourcemanager.hostname hadoop0 YARN运算IO错误 该错误由于机器内存过小，却执行复杂的运算，导致内存不足，报JavaIO异常，例如在2G内存的机器上执行 hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 4 5运算 Job Finished in 34.889 seconds java.io.FileNotFoundException: File does not exist: hdfs://192.168.79.129:8020/user/hadoop/QuasiMonteCarlo_1524323663509_1454922894/out/reduce-out at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1219) at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1750) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1774) at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314) at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) YARN也显示工作失败 解决方案 调整虚拟内存，或者减小运算精度，例如设置精度为hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 2 3则一般不会有问题 机器时间不同步 该问题出现在MapReduce测试时，若主从节点机器时间不一致则会报同步错误 18/04/21 23:55:20 INFO mapreduce.Job: Task Id : attempt_1524320196806_0003_m_000000_2, Status : FAILED Container launch failed for container_1524320196806_0003_01_000004 : org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container. This token is expired. current time is 1524333912535 found 1524326717165 Note: System times on machines may be out of sync. Check system time and time zones. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:159) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 解决方案 首先检查各节点的时间 主节点 [hadoop@hadoop0 testFile]$ timedatectl status Local time: 六 2018-04-21 23:57:21 CST Universal time: 六 2018-04-21 15:57:21 UTC RTC time: 六 2018-04-21 15:44:44 Time zone: Asia/Shanghai (CST, +0800) NTP enabled: yes NTP synchronized: no RTC in local TZ: no DST active: n/a 从节点1 [hadoop@hadoop1 hadoop]$ timedatectl status Local time: 日 2018-04-22 02:06:49 CST Universal time: 六 2018-04-21 18:06:49 UTC RTC time: 六 2018-04-21 17:54:14 Time zone: Asia/Shanghai (CST, +0800) NTP enabled: yes NTP synchronized: no RTC in local TZ: no DST active: n/a 从节点2 [hadoop@hadoop2 hadoop]$ timedatectl status Local time: 日 2018-04-22 02:07:00 CST Universal time: 六 2018-04-21 18:07:00 UTC RTC time: 日 2018-04-22 04:00:48 Time zone: Asia/Shanghai (CST, +0800) NTP enabled: yes NTP synchronized: no RTC in local TZ: no DST active: n/a 三个节点显然时间不一致，使用ntpdate工具同步各机器时间 [root@hadoop2 hadoop]# yum -y install ntp ntpdate 已加载插件：fastestmirror base | 3.6 kB 00:00:00 epel/x86_64/metalink | 7.7 kB 00:00:00 epel | 4.7 kB 00:00:00 extras | 3.4 kB 00:00:00 updates | 3.4 kB 00:00:00 (1/4): epel/x86_64/updateinfo | 916 kB 00:00:01 (2/4): extras/7/x86_64/primary_db | 185 kB 00:00:01 (3/4): epel/x86_64/primary_db | 6.3 MB 00:00:01 (4/4): updates/7/x86_64/primary_db | 6.9 MB 00:00:03 Determining fastest mirrors * base: centos.ustc.edu.cn * epel: mirror01.idc.hinet.net * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com 软件包 ntp-4.2.6p5-25.el7.centos.2.x86_64 已安装并且是最新版本 软件包 ntpdate-4.2.6p5-25.el7.centos.2.x86_64 已安装并且是最新版本 无须任何处理 [root@hadoop2 hadoop]# ntpdate cn.pool.ntp.org 22 Apr 12:02:55 ntpdate[36396]: step time server 108.59.2.24 offset 35618.521668 sec 将时间偏差写入机器时间 [root@hadoop2 hadoop]# hwclock --systohc [root@hadoop2 hadoop]# date 2018年 04月 22日 星期日 12:03:33 CST 问题解决
搭建环境 Windows VM虚拟机 虚拟机配置 CPU:7700 单核 RAM:2G SSD:20G OS:CentOS 7.4 环境配置 修改主机名 因为三台虚拟机均为一台虚拟机克隆而成，且源主机名为localhost，先计划将三台主机分别设置为hadoop0，hadoop1，hadoop2 IP、主机名、职能分别为： 192.168.79.129 hadoop0 NameNode DataNode ResourceManager NodeManager 192.168.79.130 hadoop1 DataNode NodeManager 192.168.79.131 hadoop2 DataNode NodeManager CentOS 7使用hostnamectl set-hostname hadoop0设置主机名，其他几台同样操作，输入后可通过hostnamectl status查看是否修改成功 [root@hadoop0 ~]# hostnamectl status Static hostname: hadoop0 Icon name: computer-vm Chassis: vm Machine ID: 4eddbba909c84e159cfa474604412c3c Boot ID: df37e835fde94d039153028c9479b68e Virtualization: vmware Operating System: CentOS Linux 7 (Core) CPE OS Name: cpe:/o:centos:centos:7 Kernel: Linux 3.10.0-693.el7.x86_64 Architecture: x86-64 由于hadoop0是根节点，用于资源调度，故其需要具有所有DataNode的访问地址和权限，进入/etc/hosts文件，添加如下内容将主机名解析到对应的IP 192.168.79.129 hadoop0 192.168.79.130 hadoop1 192.168.79.131 hadoop2 SSH免密码登录 进入当前用户的根目录/home/hadoop/.ssh/，在该目录下执行ssh-keygen -t rsa生成主机公钥，过程中弹出的除了 Overwrite (y/n)? y选择y确认覆盖之外，其余全为空值，则可生成免密码登录的公钥 【注意】Ubuntu系统操作略有特殊，建议使用CentOS系统进行操作 [hadoop@hadoop0 root]$ cd /home/hadoop/.ssh/ [hadoop@hadoop0 .ssh]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): /home/hadoop/.ssh/id_rsa already exists. Overwrite (y/n)? y Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hadoop/.ssh/id_rsa. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub. The key fingerprint is: SHA256:RjyGCGedx9xzxXG97KGgjXMIZ/pg7Q+tLU1nJoxIhm8 hadoop@hadoop0 The key's randomart image is: +---[RSA 2048]----+ | . o. + . oo.o| | + .oo+ o . ...| | . o.= o . .| | . * + . + | | + S B . o .| | E B.* = . | | o +.=.= | | ++. | | .oo | +----[SHA256]-----+ 使用ssh-copy-id将本地主机的公钥复制到远程主机的authorized_keys文件上，此时先对本地主机hadoop0授权 [hadoop@hadoop0 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop0 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub" The authenticity of host 'hadoop0 (fe80::e99d:ad75:28e9:9276%ens33)' can't be established. ECDSA key fingerprint is SHA256:GEEH9kBVcDuaYpJZLNsaWZA3O47+HdqIbd91fGAWcew. ECDSA key fingerprint is MD5:cd:df:9e:8c:92:3a:9f:ae:b0:10:9b:b6:25:9c:4d:c6. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys hadoop@hadoop0's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'hadoop0'" and check to make sure that only the key(s) you wanted were added. 测试SSH免密码登录成功 [hadoop@hadoop0 /]$ ssh hadoop0 Last login: Sat Apr 21 17:18:37 2018 [hadoop@hadoop0 ~]$ exit 登出 Connection to hadoop0 closed. 对另外两台主机访问授权 hadoop1 [hadoop@hadoop0 root]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop1 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub" The authenticity of host 'hadoop1 (192.168.79.130)' can't be established. ECDSA key fingerprint is SHA256:GEEH9kBVcDuaYpJZLNsaWZA3O47+HdqIbd91fGAWcew. ECDSA key fingerprint is MD5:cd:df:9e:8c:92:3a:9f:ae:b0:10:9b:b6:25:9c:4d:c6. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys hadoop@hadoop1's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'hadoop1'" and check to make sure that only the key(s) you wanted were added. hadoop2 [hadoop@hadoop0 root]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop2 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub" The authenticity of host 'hadoop2 (192.168.79.131)' can't be established. ECDSA key fingerprint is SHA256:GEEH9kBVcDuaYpJZLNsaWZA3O47+HdqIbd91fGAWcew. ECDSA key fingerprint is MD5:cd:df:9e:8c:92:3a:9f:ae:b0:10:9b:b6:25:9c:4d:c6. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys hadoop@hadoop2's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'hadoop2'" and check to make sure that only the key(s) you wanted were added. 对另外两台主机做类似操作，即仅对本地主机授权，无需对hadoop0主机授权，操作后可查看authorized_keys文件内容，当前主机名即可 环境变量 三台主机均由之前的单节点主机克隆而来，故宝塔、java、hadoop环境是自带的 Java8 [hadoop@hadoop0 .ssh]$ java -version java version "1.8.0_121" Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode) [hadoop@hadoop0 .ssh]$ javac -version javac 1.8.0_121 hadoop [hadoop@hadoop0 .ssh]$ cd /www/hadoop [hadoop@hadoop0 hadoop]$ [hadoop@hadoop0 hadoop]$ ls hadoop-2.6.0-cdh5.7.0 hadoop-2.6.0-cdh5.7.0.tar.gz testFile tmp [hadoop@hadoop0 hadoop]$ source ~/.bash_profile $HADOOP_HOME [hadoop@hadoop0 hadoop]$ echo $HADOOP_HOME /www/hadoop/hadoop-2.6.0-cdh5.7.0 hadoop配置 若之前配置过hadoop，建议将原目录删除后重新解压，主节点和从节点的目录都要删除干净，否则会导致大量问题 配置主节点core-site.xml fs.defaultFS hdfs://192.168.79.129:8020 配置HDFS文件存放目录和节点数hdfs-site.xml dfs.namenode.name.dir /www/hadoop/app/tmp/dfs/name dfs.datanode.data.dir /www/hadoop/app/tmp/dfs/data 配置YARN的yarn.xml，删除单节点的配置，添加如下字段 yarn.nodemanager.aux-services mapreduce_shuffle yarn.log-aggregation-enable true yarn.resourcemanager.hostname hadoop0 配置从节点slave，默认slave为localhost [hadoop@hadoop0 hadoop]$ cat slaves localhost 加入host配置 [hadoop@hadoop0 hadoop]$ cat slaves hadoop0 hadoop1 hadoop2 额外配置，启用jobhistory mapreduce.framework.name yarn mapreduce.jobhistory.address 192.168.79.129:10020 MapReduce JobHistory Server IPC host:port mapreduce.jobhistory.webapp.address 192.168.79.129:19888 MapReduce JobHistory Server Web UI host:port mapreduce.jobhistory.done-dir /history/done mapreduce.jobhistory.intermediate-done-dir /history/done_intermediate 【注意】配置好后将单节点时的/www/hadoop/hadoop-2.6.0-cdh5.7.0/logs文件夹删除或者直接重新解压hadoop的压缩包防止出现文件夹权限导致的错误 资源分发 将主节点的所有配置分发到从节点，避免重复配置。在主节点执行如下命令即可 scp -r /www/hadoop hadoop@hadoop1:/www scp -r /www/hadoop hadoop@hadoop2:/www 分发完成 去根节点查看文件是否已经传到，分别执行source ~/.bash_profile以启用hadoop命令 注意分发前执行一次hadoop命令是报未找到命令，说明从节点的hadoop旧根目录是删除干净的，分发完成后再次执行命令就有效了 NameNode格式化 格式化只对主节点执行即可 hdfs namenode -format 格式化成功 hadoop分布式启动 主节点启动，在/sbin目录下，执行./start-all.sh [hadoop@hadoop0 bin]$ cd ../sbin [hadoop@hadoop0 sbin]$ ./start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh 18/04/21 22:16:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Starting namenodes on [hadoop0] 从节点的datanode和namenode逐一启动 hadoop0: starting namenode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop0.out hadoop1: starting datanode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop1.out hadoop0: starting datanode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop0.out hadoop2: starting datanode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop2.out Starting secondary namenodes [0.0.0.0] 主节点secondarynamenode启动 0.0.0.0: starting secondarynamenode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop0.out 18/04/21 22:16:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 启动yarn和resourcemanager、nodemanager starting yarn daemons starting resourcemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-hadoop0.out hadoop1: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop1.out hadoop2: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop2.out hadoop0: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop0.out hadoop分布式启动验证 命令行方式 使用jps查看进程，以下进程少一个都不行，若缺少，查看另一篇文章寻找解决办法hadoop分布式搭建的一些坑 hadoop分布式搭建的一些坑 启动主节点后从节点的HDFS会自动启动，进入从节点，使用jps查看，有两个进程即成功 [hadoop@hadoop2 hadoop]$ jps 29876 Jps 29575 DataNode 29692 NodeManager webUI方式 使用webUI确认所有进程成功启动，查看HDFS节点 查看YARN 已经有三个从节点注册到YARN 分布式集群测试 分布式文件测试 上传测试文件再查看节点情况 初始节点为空 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ ls bin cloudera examples include libexec logs README.txt share bin-mapreduce1 etc examples-mapreduce1 lib LICENSE.txt NOTICE.txt sbin src [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -ls / 18/04/21 22:33:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 新建数据目录 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -mkdir /data 18/04/21 22:33:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -ls / 18/04/21 22:33:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 1 items drwxr-xr-x - hadoop supergroup 0 2018-04-21 22:33 /data 上传小文件 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -put README.txt /data 18/04/21 22:34:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -ls /data 18/04/21 22:34:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 1 items -rw-r--r-- 3 hadoop supergroup 1366 2018-04-21 22:34 /data/README.txt 上传大文件 [hadoop@hadoop0 hadoop]$ ll 总用量 304284 drwxrwxr-x 3 hadoop hadoop 17 4月 21 22:16 app drwxrwxrwx 15 www www 253 4月 21 22:08 hadoop-2.6.0-cdh5.7.0 -rwxrwxrwx 1 www www 311585484 3月 31 23:06 hadoop-2.6.0-cdh5.7.0.tar.gz drwxr-xr-x 2 www www 301 4月 11 06:31 testFile [hadoop@hadoop0 hadoop]$ hadoop fs -put hadoop-2.6.0-cdh5.7.0.tar.gz /data 18/04/21 22:35:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [hadoop@hadoop0 hadoop]$ hadoop fs -ls /data 18/04/21 22:36:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 2 items -rw-r--r-- 3 hadoop supergroup 1366 2018-04-21 22:34 /data/README.txt -rw-r--r-- 3 hadoop supergroup 311585484 2018-04-21 22:36 /data/hadoop-2.6.0-cdh5.7.0.tar.gz 通过webUI查看文件情况 分布式MapReduce测试 测试MapReduce，进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce目录，执行pi运算hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 2 3 【注意】由于此处是虚拟机，不建议将后面的精度设置过大，否则会报IO错误 进入YARN的web界面查看 运行成功 分布式日志离线处理测试 上传日志数据 [hadoop@hadoop0 testFile]$ hadoop fs -put access.log /data 18/04/21 23:42:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 执行 hadoop jar hadoopstudy-0.6-jar-with-dependencies.jar com.fjy.hadoop.hadoop.project.LogApp /data/access.log /logaccess/out Web查看工作详情 查看结果 题外话 其实同时运行三台虚拟机是很吃配置的，特别是内存，对于hadoop来说内存太小很多都会报异常，因此本次的测试机器配置还可以的，三台虚拟机均运行在480G SATA3的固态上 CPU:7700 4core 8thread RAM:16G DDR4 Dual Channal SSD:128+480 nvme+sata3 HDD:9T sata3 Eth:1000M I219V 但在实际运行中依然顶着内存瓶颈，多次使用了系统盘的交换分区
代码 GitHub链接 开源工具类UserAgentParser地址 日志来源 根据之前的作业提交系统项目的后台获取，为了本地测试，故先从数据库中备份下来并做了适当的处理，仅保留UserAgent的相关信息 本地文件测试 使用开源工具类解析UserAgent中的信息 @Test public void testReadFile() throws Exception { String path = "F:\\JAVA Workspace\\hadoopstudy\\access.log"; BufferedReader reader = new BufferedReader(new InputStreamReader(new FileInputStream(new File(path)))); String line = ""; int count = 0; //模拟MapReduce存储 Map browerMap = new HashMap(); UserAgentParser userAgentParser = new UserAgentParser(); while (line != null) { line = reader.readLine(); count++; if (StringUtils.isNotBlank(line)) { String source = line; UserAgent agent = userAgentParser.parse(source); //测试 String browser = agent.getBrowser(); String engine = agent.getEngine(); String engineVersion = agent.getEngineVersion(); String os = agent.getOs(); String platform = agent.getPlatform(); boolean mobile = agent.isMobile(); Integer browserValue = browerMap.get(browser); if (browserValue != null) { browerMap.put(browser, browerMap.get(browser) + 1); } else { browerMap.put(browser, 1); } //输出解析的信息 System.out.println(browser + "," + engine + "," + engineVersion + "," + os + "," + platform + "," + mobile); } } System.out.println("总记录数：" + count); System.out.println("===================================="); for (Map.Entryentry :browerMap.entrySet()){ System.out.println(entry.getKey()+":"+entry.getValue()); } } 执行单元测试 使用MapReduce统计 代码以WordCount为原型，结合本地测试的代码编写 ... boolean mobile = agent.isMobile(); if (mobile){ mobileuser = "手机用户"; }else { mobileuser = "非手机用户"; } //通过上下文把map的处理结果输出 context.write(new Text(browser), one); ... 打包运行 将工程用mvn assembly:assembly命令把插件一起打包，上传至虚拟机，同时将log上传至HDFS的根目录 [hadoop@localhost testFile]$ hadoop fs -put access.log / 18/04/11 06:12:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [hadoop@localhost testFile]$ hadoop fs -ls / 18/04/11 06:12:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 9 items -rw-r--r-- 1 hadoop supergroup 73 2018-04-11 00:30 /PartitionerTest.txt -rw-r--r-- 1 hadoop supergroup 26888 2018-04-11 06:12 /access.log drwxr-xr-x - hadoop supergroup 0 2018-04-08 10:56 /hdfsapi -rw-r--r-- 1 hadoop supergroup 60 2018-04-10 23:03 /hello.txt drwxrwx--- - hadoop supergroup 0 2018-04-11 01:28 /history drwxr-xr-x - hadoop supergroup 0 2018-04-11 00:35 /output drwxr-xr-x - hadoop supergroup 0 2018-04-08 10:35 /test drwx------ - hadoop supergroup 0 2018-04-11 01:36 /tmp drwxr-xr-x - hadoop supergroup 0 2018-04-09 20:11 /user 运行 结果与本地测试一致 [hadoop@localhost testFile]$ hadoop fs -ls /logaccess/browserout 18/04/11 06:15:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 2 items -rw-r--r-- 1 hadoop supergroup 0 2018-04-11 06:14 /logaccess/browserout/_SUCCESS -rw-r--r-- 1 hadoop supergroup 32 2018-04-11 06:14 /logaccess/browserout/part-r-00000 [hadoop@localhost testFile]$ hadoop fs -text /logaccess/browserout/part-r-00000 18/04/11 06:15:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Chrome 191 Firefox 10 Unknown 1 增加其他属性重新运行MapReduce，结果如下 [hadoop@localhost testFile]$ hadoop fs -ls /logaccess 18/04/11 06:33:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 2 items -rw-r--r-- 1 hadoop supergroup 0 2018-04-11 06:32 /logaccess/_SUCCESS -rw-r--r-- 1 hadoop supergroup 192 2018-04-11 06:32 /logaccess/part-r-00000 [hadoop@localhost testFile]$ hadoop fs -text /logaccess/part-r-00000 18/04/11 06:33:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20100101 10 537.36 191 604.3.5 1 Android 48 Chrome 191 Firefox 10 Gecko 10 Linux 48 Unknown 1 Webkit 192 Windows 296 Windows 7 10 iPhone 1 iPhone OS 11.1 1 手机用户 49 非手机用户 153 日志离线统计完成！
官方原理解析 MapReduce演进 MapReduce2.x的架构 MapReduce1.x的架构，一旦JobTracker挂了，整个分布式节点就崩溃了，因此被淘汰 开发WordCount WordCount原理 数据输入到MapReduce，每个MapReduce任务都被初始化为一个Job，每个Job又可以分为两种阶段：map阶段和reduce阶段。map方法接收一个形式的输入，key表示词组偏移量，value表示该偏移量对应的值，然后同样产生一个形式的中间输出，Hadoop函数接收一个如形式的输入，list of values即Java中的集合，用于表示相同偏移量下的字符统计数量，然后对这个value集合进行处理，每个Reduce方法产生0或1个输出，reduce的输出也是形式的。 如果文本为 Hadoop Hello Hadoop 实际过程大概为： Mapping： ，， Shuffling： ， Reducing： 多节点下的并行计算原理示意图 开发代码 查看源码 通过查看Mapper类的源码，发现官方已经定义的run方法，包含了所有基础的Mapper操作（模板方法设计模式），主要操作在map中，因此，只要重写map方法即可。 同理Reduce类中也有类似的定义 重写方法 这里只列举部分代码，其余所有代码可去本人的GitHub中查看：GitHub链接 /** * Map：读取输入文件 * Text:类似字符串 */ public static class MyMapper extends Mapper { LongWritable one = new LongWritable(1); /** * @param key 偏移量 * @param value 每行的字符串 * @param context 上下文 * @throws IOException * @throws InterruptedException */ @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { /*super.map(key, value, context);*/ //接收到每一行数据 String line = value.toString(); //按照指定分隔符进行拆分 /*line.split("\t");//以Tab分隔*/ String[] words = line.split(" ");//以空格分隔 for (String word : words) { //通过上下文把map的处理结果输出 context.write(new Text(word), one); } } } 运行 通过宝塔面板上传到服务器，确保此时Hadoop所有服务均为启动状态 在HDFS中预先创建好要测试的文件，通过绝对路径的方法hadoop fs -ls hdfs://192.168.79.129:8020/确认文件是可以访问的 输入运行命令即可运行 hadoop jar hadoopstudy-1.0-SNAPSHOT.jar com.fjy.hadoop.mapreduce.WordCountApp hdfs://192.168.79.129:8020/hello.txt hdfs://192.168.79.129:8020/output/wc 启动至FileInputFormat [hadoop@localhost testFile]$ hadoop jar hadoopstudy-1.0-SNAPSHOT.jar com.fjy.hadoop.mapreduce.WordCountA2.168.79.129:8020/hello.txt hdfs://192.168.79.129:8020/output/wc 18/04/10 09:57:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 18/04/10 09:57:44 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 18/04/10 09:57:45 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 18/04/10 09:57:45 INFO input.**FileInputFormat**: Total input paths to process : 1 执行Job 18/04/10 09:57:46 INFO mapreduce.JobSubmitter: number of splits:1 18/04/10 09:57:46 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1523322221448_0001 18/04/10 09:57:47 INFO impl.YarnClientImpl: Submitted application application_1523322221448_0001 18/04/10 09:57:47 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1523322221448_0001/ 18/04/10 09:57:47 INFO mapreduce.Job: Running job: job_1523322221448_0001 18/04/10 09:57:56 INFO mapreduce.Job: Job job_1523322221448_0001 running in uber mode : false 18/04/10 09:57:56 INFO mapreduce.Job: map 0% reduce 0% 18/04/10 09:58:04 INFO mapreduce.Job: map 100% reduce 0% 18/04/10 09:58:10 INFO mapreduce.Job: map 100% reduce 100% 18/04/10 09:58:12 INFO mapreduce.Job: Job job_1523322221448_0001 completed successfully 18/04/10 09:58:12 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=39 FILE: Number of bytes written=222915 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=113 HDFS: Number of bytes written=17 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Map方法执行 Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=5920 Total time spent by all reduces in occupied slots (ms)=3321 Total time spent by all map tasks (ms)=5920 Total time spent by all reduce tasks (ms)=3321 Total vcore-seconds taken by all map tasks=5920 Total vcore-seconds taken by all reduce tasks=3321 Total megabyte-seconds taken by all map tasks=6062080 Total megabyte-seconds taken by all reduce tasks=3400704 Map-Reduce Framework Map input records=1 Map output records=2 Map output bytes=29 Map output materialized bytes=39 Reduce 方法执行 Input split bytes=101 Combine input records=0 Combine output records=0 Reduce input groups=2 Reduce shuffle bytes=39 Reduce input records=2 Reduce output records=2 Spilled Records=4 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=115 CPU time spent (ms)=1080 Physical memory (bytes) snapshot=300331008 Virtual memory (bytes) snapshot=5493092352 Total committed heap usage (bytes)=165810176 Shuffle 方法执行 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 `File Input`和`File Output` 方法执行 File Input Format Counters Bytes Read=12 File Output Format Counters Bytes Written=17 运行结果如图 进一步优化 使用Combiner 使用Combiner组件可以在Mapping时就先进行一次聚合，再发送到Shuffling继续运算，从而提高运算效率 在主方法中进行如下配置即可 //通过job设置combiner处理类，逻辑上与reduce一致，注意，如果要计算平均数等不能使用Combiner! job.setCombinerClass(MyReducer.class); 编写完后使用Maven重新打包 mvn clean package -DskipTests 上传到服务器后再次执行计算命令 hadoop jar hadoopstudy-0.3.jar com.fjy.hadoop.mapreduce.WordCountApp hdfs://192.168.79.129:8020/hello.txt hdfs://192.168.79.129:8020/output/wc 查看图形界面可观察是否完成 通过查看代码可发现Combine input records已经不为0(之前未使用Combiner 时该值为0)，说明使用成功 Map-Reduce Framework Map input records=4 ... Combine input records=9 Combine output records=5 ... Physical memory (bytes) snapshot=298844160 Virtual memory (bytes) snapshot=5493088256 Total committed heap usage (bytes)=165810176 Shuffle Errors BAD_ID=0 ... 查看结果 [hadoop@localhost testFile]$ hadoop fs -cat /output/wc/part-r-00000 18/04/10 23:06:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable ! 1 Hadoop 3 Hello 2 MapReduce 2 YARN 1 使用Partitioner组件 作用 Partitioner决定MapTask输出的数据交由哪个ReduceTask处理，其默认实现为分发的key的hash值对ReduceTask个数取模 文件准备 在宝塔测试目录/www/hadoop/testFile下准备好PartitionerTest.txt文件，文件内容如图，将其上传到HDFS根目录中 [hadoop@localhost root]$ cd /www/hadoop/testFile [hadoop@localhost testFile]$ ls PartitionerTest.txt hello.txt [hadoop@localhost testFile]$ hadoop fs -put PartitionerTest.txt / 18/04/11 00:30:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [hadoop@localhost testFile]$ hadoop fs -ls / 18/04/11 00:30:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 7 items -rw-r--r-- 1 hadoop supergroup 73 2018-04-11 00:30 /PartitionerTest.txt drwxr-xr-x - hadoop supergroup 0 2018-04-08 10:56 /hdfsapi -rw-r--r-- 1 hadoop supergroup 60 2018-04-10 23:03 /hello.txt drwxr-xr-x - hadoop supergroup 0 2018-04-10 23:05 /output drwxr-xr-x - hadoop supergroup 0 2018-04-08 10:35 /test drwx------ - hadoop supergroup 0 2018-04-09 20:11 /tmp drwxr-xr-x - hadoop supergroup 0 2018-04-09 20:11 /user [hadoop@localhost testFile]$ hadoop fs -text /PartitionerTest.txt 18/04/11 00:30:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable xiaomi 200 huawei 300 xiaomi 100 huawei 200 iphone 300 iphone 200 sony 50 代码编写 主要对分隔符和上下文内容进行修改，map类中修改为 //每一个空格是一个手机品牌，另一个是销售数量 context.write(new Text(words[0]), new LongWritable(Long.parseLong(words[1]))); 新增Partitioner处理类 /** * Partitioner处理类 */ public static class MyPartitioner extends Partitioner{ @Override public int getPartition(Text key, LongWritable value, int i) { if ("xiaomi".equals(key.toString())) { return 0;//若为xiaomi则交由0 ReduceTask处理 } if ("huawei".equals(key.toString())) { return 1;//若为huawei则交由1 ReduceTask处理 } if ("iphone".equals(key.toString())) { return 2;//若为iphone则交由2 ReduceTask处理 } return 3;//若为其他，则交由3 ReduceTask处理 } } 最后在驱动中配置Partitioner //设置job的Partition job.setPartitionerClass(MyPartitioner.class); //设置四个reducer，每个分区一个，否则Partitioner配置不生效 job.setNumReduceTasks(4); 通过复制类名全路径（以IDEA为例），修改运行命令 此时，运行命令为 hadoop jar hadoopstudy-0.4.jar com.fjy.hadoop.mapreduce.WordCountPartitionerApp hdfs://192.168.79.129:8020/PartitionerTest.txt hdfs://192.168.79.129:8020/output/partitioner 执行 再次用Maven打包并上传到测试目录运行 运行前： 运行中： 运行结束 查看结果内容： [hadoop@localhost testFile]$ hadoop fs -ls /output/partitioner 18/04/11 00:36:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 5 items -rw-r--r-- 1 hadoop supergroup 0 2018-04-11 00:35 /output/partitioner/_SUCCESS -rw-r--r-- 1 hadoop supergroup 11 2018-04-11 00:35 /output/partitioner/part-r-00000 -rw-r--r-- 1 hadoop supergroup 11 2018-04-11 00:35 /output/partitioner/part-r-00001 -rw-r--r-- 1 hadoop supergroup 11 2018-04-11 00:35 /output/partitioner/part-r-00002 -rw-r--r-- 1 hadoop supergroup 8 2018-04-11 00:35 /output/partitioner/part-r-00003 [hadoop@localhost testFile]$ hadoop fs -text /output/partitioner/part-r-00000 18/04/11 00:36:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable xiaomi 300 [hadoop@localhost testFile]$ hadoop fs -text /output/partitioner/part-r-00001 18/04/11 00:36:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable huawei 500 [hadoop@localhost testFile]$ hadoop fs -text /output/partitioner/part-r-00002 18/04/11 00:37:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable iphone 500 [hadoop@localhost testFile]$ hadoop fs -text /output/partitioner/part-r-00003 18/04/11 00:37:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable sony 50 结果正确！ 开启JobHistory JobHistory作用 记录已经运行完 的MapReduce信息到指定的HDFS目录下，但默认不开启 因此此时不配置直接访问如下 ![jobhistory][33] 配置JobHistory 在mapred-site.xml加入配置 mapreduce.jobhistory.address 192.168.79.129:10020 MapReduce JobHistory Server IPC host:port mapreduce.jobhistory.webapp.address 192.168.79.129:19888 MapReduce JobHistory Server Web UI host:port mapreduce.jobhistory.done-dir /history/done mapreduce.jobhistory.intermediate-done-dir /history/done_intermediate 配置完后重启YARN [hadoop@localhost mapreduce]$ cd /www/hadoop/hadoop-2.6.0-cdh5.7.0/sbin [hadoop@localhost sbin]$ ./stop-yarn.sh stopping yarn daemons stopping resourcemanager localhost: stopping nodemanager no proxyserver to stop [hadoop@localhost sbin]$ jps 2353 DataNode 80224 Jps 2228 NameNode 2522 SecondaryNameNode [hadoop@localhost sbin]$ ./start-yarn.sh starting yarn daemons starting resourcemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-localhost.localdomain.out localhost: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-localhost.localdomain.out [hadoop@localhost mapreduce]$ jps 2353 DataNode 80336 ResourceManager 2228 NameNode 81510 Jps 2522 SecondaryNameNode 63582 JobHistoryServer 80447 NodeManager 启动jobhistory服务 [hadoop@localhost sbin]$ ./mr-jobhistory-daemon.sh start historyserver starting historyserver, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/mapred-hadoop-historyserver-localhost.localdomain.out 运行PI测试 [hadoop@localhost sbin]$ cd /www/hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce [hadoop@localhost mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 1 2 访问报错 进入日志查看，发现仍然无法访问 报未开启聚合的错误，同时发现指向跳转链接为localhost 增加配置 因此在yarn-site.xml加入配置 yarn.log-aggregation-enable true 同时，为了让虚拟机的跳转链接不为localhost，此处再加入一段配置，制定IP yarn.nodemanager.hostname 192.168.79.129 重启服务 再次重启YARN，并同时重启jobhistory服务 [hadoop@localhost sbin]$ ./mr-jobhistory-daemon.sh stop historyserver stopping historyserver [hadoop@localhost sbin]$ ./stop-yarn.sh stopping yarn daemons stopping resourcemanager localhost: stopping nodemanager no proxyserver to stop [hadoop@localhost sbin]$ ./start-yarn.sh starting yarn daemons starting resourcemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-localhost.localdomain.out localhost: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-localhost.localdomain.out [hadoop@localhost sbin]$ ./mr-jobhistory-daemon.sh start historyserver starting historyserver, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/mapred-hadoop-historyserver-localhost.localdomain.out 【注意】此时若只重启YARN就算配置正确，仍然可能出现Aggregation is not enabled的错误 验证 再次执行PI运算 [hadoop@localhost sbin]$ cd /www/hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce [hadoop@localhost mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 1 2 查看Logs记录 已经有日志记录，配置成功！
配置YARN 首先启动之前配置好的HDFS服务，然后进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/etc/hadoop目录，按照官方指导配置YARN单节点环境 使用cp mapred-site.xml.template mapred-site.xml命令把mapred-site.xml.template复制一份，加入官方的配置 mapreduce.framework.name yarn 同时对yarn-site.xml也进行配置 yarn.nodemanager.aux-services mapreduce_shuffle 启动YARN 进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/sbin目录，使用./start-yarn.sh命令启动YARN服务 [hadoop@localhost sbin]$ ./start-yarn.sh starting yarn daemons starting resourcemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-localhost.localdomain.out localhost: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-localhost.localdomain.out 用jps命令查看是否启动成功，看到如下显示即为启动成功，主要是NodeManager和ResourceManager两个服务 [hadoop@localhost sbin]$ jps 7056 SecondaryNameNode 7475 NodeManager 6775 NameNode 6872 DataNode 7374 ResourceManager 7679 Jps 开放端口 YARN的浏览器客户端占用8088端口，故使用宝塔面板开放8088端口 访问YARN图形化界面 使用浏览器打开http://ip:8088/可见如下界面，则YARN启动成功 测试 YARN官方提供了示例java程序来测试资源调度，进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce目录，可见 [hadoop@localhost mapreduce]$ ls hadoop-mapreduce-client-app-2.6.0-cdh5.7.0.jar hadoop-mapreduce-client-nativetask-2.6.0-cdh5.7.0.jar hadoop-mapreduce-client-common-2.6.0-cdh5.7.0.jar hadoop-mapreduce-client-shuffle-2.6.0-cdh5.7.0.jar hadoop-mapreduce-client-core-2.6.0-cdh5.7.0.jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar hadoop-mapreduce-client-hs-2.6.0-cdh5.7.0.jar lib hadoop-mapreduce-client-hs-plugins-2.6.0-cdh5.7.0.jar lib-examples hadoop-mapreduce-client-jobclient-2.6.0-cdh5.7.0.jar sources hadoop-mapreduce-client-jobclient-2.6.0-cdh5.7.0-tests.jar 其中hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar就是MapReduce的测试程序，通过hadoop jar的方式来运行 用命令查看可配置的参数hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar [hadoop@localhost mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar An example program must be given as the first argument. Valid program names are: aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files. aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files. bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi. dbcount: An example job that count the pageview counts from a database. distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi. grep: A map/reduce program that counts the matches of a regex in the input. join: A job that effects a join over sorted, equally partitioned datasets multifilewc: A job that counts words from several files. pentomino: A map/reduce tile laying program to find solutions to pentomino problems. pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method. randomtextwriter: A map/reduce program that writes 10GB of random textual data per node. randomwriter: A map/reduce program that writes 10GB of random data per node. secondarysort: An example defining a secondary sort to the reduce. sort: A map/reduce program that sorts the data written by the random writer. sudoku: A sudoku solver. teragen: Generate data for the terasort terasort: Run the terasort teravalidate: Checking results of terasort wordcount: A map/reduce program that counts the words in the input files. wordmean: A map/reduce program that counts the average length of the words in the input files. wordmedian: A map/reduce program that counts the median length of the words in the input files. wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files. [hadoop@localhost mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi Usage: org.apache.hadoop.examples.QuasiMonteCarlo Generic options supported are -conf specify an application configuration file -D use value for given property -fs specify a namenode -jt specify a ResourceManager -files specify comma separated files to be copied to the map reduce cluster -libjars specify comma separated jar files to include in the classpath. -archives specify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions] 从上可知测试程序可以做诸如数学π、随机文件写入等操作，这里做数学π的运算 使用命令hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 2 3做两个map计算精度为3的数学π运算,运算结束后会显示结果如下 Job Finished in 26.996 seconds Estimated value of Pi is 4.00000000000000000000 同时浏览器也可看到作业情况
环境 系统：CentOS Linux 7.4.1708 (Core) 客户端：putty 服务器：虚拟机1core、2G RAM 同时搭建了宝塔面板（自动配置了java8） Hadoop下载 CDH版本Hadoop下载 为了统一版本，全部使用hadoop-2.6.0-cdh5.7.0 点击可进入官方推荐配置页 解压并配置 将下载好的文件通过宝塔上传到服务器，这里选择上传到/www/hadoop目录 解压tar包 配置 新建用户 新建hadoop用户，忽略密码强度提示 adduser hadoop passwd hadoop 为方便远程连接和学习，同时授予root权限，输入whereis sudoers查看权限控制文件位置 [root@fjy ~]# whereis sudoers sudoers: /etc/sudoers /etc/sudoers.d /usr/libexec/sudoers.so /usr/share/man/man5/sudoers.5.gz 使用宝塔找到并修改/etc/sudoers文件 配置新用户免密码远程登录 切换至hadoop用户，使用 ssh-keygen -t rsa生成私钥信息，位置在/home/hadoop/，并且忽略所有提示输入的信息 查看生成的文件，该SSH文件夹为隐藏文件夹，故用ls -a查看，可见自动生成了.ssh目录和id_rsa,id_rsa.pub两个文件 复制公钥到.ssh目录 cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys 查看JAVA目录 因为是宝塔自动配置的java8，所以要首先查看当前java环境变量位置，使用如下命令 update-alternatives --config java 此时宝塔自动将java配置到/usr/java/jdk1.8.0_121/jre/bin/java目录 即/usr/java/jdk1.8.0_121为java环境变量的目录 配置hadoop的java目录 使用宝塔进入hadoop目录下的etc/hadoop/hadoop-env.sh，对应此处的绝对路径应为/www/hadoop/hadoop-2.6.0-cdh5.7.0/etc/hadoop 使用宝塔编辑文件，加入java环境变量位置 配置分布式文件参数 打开core-site.xml，配置如下 fs.defaultFS hdfs://192.168.79.129:8020 hadoop.tmp.dir /www/hadoop/tmp 打开hdfs-site.xml，配置如下 dfs.replication 1 启动 初始化 进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/bin目录，使用如下语句初始化Hadoop，该操作仅执行一次即可，以后使用无需重复执行，否则执行后会删除当前hadoop上的所有文件 hadoop namenode -format 滚动均为info级别的日志 可看到已初始化成功 启动HDFS 切换到hadoop用户，进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/sbin目录 输入./start-dfs.sh启动HDFS服务 【注意】，要提前在root用户下对/www/hadoop授予hadoop用户读写权限，否则无法启动 使用jps查看当前的java进程，若出现如下四个进程表示HDFS服务启动成功 外部访问浏览器客户端 首先使用宝塔开放50070端口 使用外部浏览器访问ip:50070可查看如下页面，则表示HDFS已成功启动 hdfs命令 配置hdfs命令 使用vi ~/.bash_profile进入配置文件，按如下配置 export HADOOP_HOME=/www/hadoop/hadoop-2.6.0-cdh5.7.0 export PATH=$HADOOP_HOME/bin:$PATH 使配置文件生效source ~/.bash_profile 输入hdfs回车测试命令是否生效，若出现如下画面则表示命令已经生效 命令测试 先在本地创建hello.txt文件，使用 hadoop fs -put hello.txt /命令将文件推送到hdfs根目录下 查看服务器根目录hadoop fs -ls /，已经有该文件，hadoop伪分布式配置完成