生产环境实战spark （7）分布式集群 5台设备 Hadoop集群安装

生产环境实战spark （7）分布式集群 5台设备 Hadoop集群安装

1，Hadoop 下载。

下载地址：http://hadoop.apache.org/releases.html

下载版本：hadoop 2.6.5 版本    hadoop 2.6.x版本比较稳定

2，使用winscp工具上传到master节点。

[root@master hadoop-2.6.5]# vi /etc/profile

export JAVA_HOME=/usr/local/jdk1.8.0_121export SCALA_HOME=/usr/local/scala-2.11.8export HADOOP_HOME=/usr/local/hadoop-2.6.5export PATH=.:$PATH:$JAVA_HOME/bin:$SCALA_HOME/bin:$HADOOP_HOME/bin

在命令行中输入source /etc/profile，使刚才修改的HADOOP_HOME及PATH配置文件生效

[root@master hadoop-2.6.5]# source /etc/profile[root@master hadoop-2.6.5]# 

4，Hadoop core-site.xml配置文件修改。 5，Hadoop  hdfs-site.xml配置文件修改。

6，Hadoop  hadoop-env.sh配置文件修改。[root@master hadoop]# cat hadoop-env.sh # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements.  See the NOTICE file # distributed with this work for additional information # regarding copyright ownership.  The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License.  You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Set Hadoop-specific environment variables here. # The only required environment variable is JAVA_HOME.  All others are # optional.  When running a distributed configuration it is best to # set JAVA_HOME in this file, so that it is correctly defined on # remote nodes. # The java implementation to use. export JAVA_HOME=${JAVA_HOME} # The jsvc implementation to use. Jsvc is required to run secure datanodes # that bind to privileged ports to provide authentication of data transfer # protocol.  Jsvc is not required if SASL is configured for authentication of # data transfer protocol using non-privileged ports. #export JSVC_HOME=${JSVC_HOME} export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"} # Extra Java CLASSPATH elements.  Automatically insert capacity-scheduler. for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do   if [ "$HADOOP_CLASSPATH" ]; then     export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f   else     export HADOOP_CLASSPATH=$f   fi done # The maximum amount of heap to use, in MB. Default is 1000. #export HADOOP_HEAPSIZE= #export HADOOP_NAMENODE_INIT_HEAPSIZE="" # Extra Java runtime options.  Empty by default. export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true" # Command specific options appended to HADOOP_OPTS when specified export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS" export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS" export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS" export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS" export HADOOP_PORTMAP_OPTS="-Xmx512m $HADOOP_PORTMAP_OPTS" # The following applies to multiple commands (fs, dfs, fsck, distcp etc) export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" #HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData $HADOOP_JAVA_PLATFORM_OPTS" # On secure datanodes, user to run the datanode as after dropping privileges. # This **MUST** be uncommented to enable secure HDFS if using privileged ports # to provide authentication of data transfer protocol.  This **MUST NOT** be # defined if SASL is configured for authentication of data transfer protocol # using non-privileged ports. export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER} # Where log files are stored.  $HADOOP_HOME/logs by default. #export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER # Where log files are stored in the secure data environment. export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER} ### # HDFS Mover specific parameters ### # Specify the JVM options to be used when starting the HDFS Mover. # These options will be appended to the options specified as HADOOP_OPTS # and therefore may override any similar flags set in HADOOP_OPTS # # export HADOOP_MOVER_OPTS="" ### # Advanced Users Only! ### # The directory where pid files are stored. /tmp by default. # NOTE: this should be set to a directory that can only be written to by  #       the user that will run the hadoop daemons.  Otherwise there is the #       potential for a symlink attack. export HADOOP_PID_DIR=${HADOOP_PID_DIR} export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR} # A string representing this instance of hadoop. $USER by default. export HADOOP_IDENT_STRING=$USER export JAVA_HOME=/usr/local/jdk1.8.0_121 [root@master hadoop]# 

7，Hadoop  master slaves配置文件修改。master作为主节点和数据处理节点。

[root@master hadoop]# cat slavesmasterworker01worker02worker03worker04[root@master hadoop]#

Hadoop最简 最小化配置完成

8，编写脚本，将Hadoop 文件分发到worker节点。[root@master local]# cd rhzf_setup_scripts  [root@master rhzf_setup_scripts]# ls rhzf_hosts_scp.sh  rhzf_scala.sh  rhzf_ssh.sh [root@master rhzf_setup_scripts]# vi rhzf_hadoop.sh #!/bin/sh for i in  238 239 240 241 do scp   -rq /usr/local/hadoop-2.6.5  root@10.100.100.$i:/usr/local/hadoop-2.6.5 scp   -rq /etc/profile  root@10.100.100.$i:/etc/profile ssh   root@10.100.100.$i source /etc/profile done

执行脚本

[root@master rhzf_setup_scripts]# chmod u+x rhzf_hadoop.sh[root@master rhzf_setup_scripts]# lsrhzf_hadoop.sh  rhzf_hosts_scp.sh  rhzf_scala.sh  rhzf_ssh.sh[root@master rhzf_setup_scripts]# ./rhzf_hadoop.sh[root@master rhzf_setup_scripts]# 

9，worker节点检查。

10，Hadopp集群文件系统格式化。

之前在master节点将java升级更新了，但4个woker节点上 java没有更新升级

[root@worker04 local]# java -versionjava version "1.7.0_51"OpenJDK Runtime Environment (rhel-2.4.5.5.el7-x86_64 u51-b31)OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)[root@worker04 local]# jpsbash: jps: command not found...[root@worker04 local]# 

12，woker节点上安装java 8 

刷新 

 # source /etc/profile

检查4个woker节点上jdk 安装完成

[root@worker01 local]# java -versionjava version "1.8.0_121"Java(TM) SE Runtime Environment (build 1.8.0_121-b13)Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)[root@worker01 local]# 

[root@worker02 bin]# java -versionjava version "1.8.0_121"Java(TM) SE Runtime Environment (build 1.8.0_121-b13)Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)[root@worker02 bin]# 

[root@worker03 local]# source /etc/profile[root@worker03 local]# java -versionjava version "1.8.0_121"Java(TM) SE Runtime Environment (build 1.8.0_121-b13)Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)[root@worker03 local]# [root@worker04 local]# source /etc/profile[root@worker04 local]# java -versionjava version "1.8.0_121"Java(TM) SE Runtime Environment (build 1.8.0_121-b13)Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)[root@worker04 local]# 

13，停止Hadoop集群 [root@master sbin]#  stop-all.sh  This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh Stopping namenodes on [Master] Master: stopping namenode worker04: no datanode to stop master: stopping datanode worker02: no datanode to stop worker01: no datanode to stop worker03: no datanode to stop Stopping secondary namenodes [0.0.0.0] 0.0.0.0: stopping secondarynamenode stopping yarn daemons stopping resourcemanager worker04: no nodemanager to stop worker03: no nodemanager to stop master: stopping nodemanager worker01: no nodemanager to stop worker02: no nodemanager to stop no proxyserver to stop [root@master sbin]# 

14，再次重启Hadoop集群，一切正常 [root@master sbin]# start-all.sh     This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh Starting namenodes on [Master] Master: starting namenode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-namenode-master.out worker04: starting datanode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-datanode-worker04.out master: starting datanode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-datanode-master.out worker03: starting datanode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-datanode-worker03.out worker01: starting datanode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-datanode-worker01.out worker02: starting datanode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-datanode-worker02.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-secondarynamenode-master.out starting yarn daemons starting resourcemanager, logging to /usr/local/hadoop-2.6.5/logs/yarn-root-resourcemanager-master.out worker04: starting nodemanager, logging to /usr/local/hadoop-2.6.5/logs/yarn-root-nodemanager-worker04.out worker02: starting nodemanager, logging to /usr/local/hadoop-2.6.5/logs/yarn-root-nodemanager-worker02.out master: starting nodemanager, logging to /usr/local/hadoop-2.6.5/logs/yarn-root-nodemanager-master.out worker01: starting nodemanager, logging to /usr/local/hadoop-2.6.5/logs/yarn-root-nodemanager-worker01.out worker03: starting nodemanager, logging to /usr/local/hadoop-2.6.5/logs/yarn-root-nodemanager-worker03.out [root@master sbin]# jps 22869 ResourceManager 22330 NameNode 22490 DataNode 23323 Jps 22684 SecondaryNameNode 22990 NodeManager [root@master sbin]# 

4个worker节点检查

[root@worker01 local]# jps20752 DataNode20884 NodeManager21001 Jps[root@worker01 local]# 

[root@worker02 bin]# jps20771 DataNode21019 Jps20895 NodeManager[root@worker02 bin]# 

[root@worker03 local]# [root@worker03 local]# jps20528 DataNode20658 NodeManager20775 Jps[root@worker03 local]# [root@worker04 local]# jps20624 NodeManager20500 DataNode20748 Jps[root@worker04 local]# 

15，webui页面打不开，10.*.*237 ：50070

17，修改配置