阿里云-云小站(无限量代金券发放中)
【腾讯云】云服务器、云数据库、COS、CDN、短信等热卖云产品特惠抢购

Ubuntu 14.04 LTS下安装Hadoop 1.2.1(集群分布式模式)

218次阅读
没有评论

共计 20643 个字符,预计需要花费 52 分钟才能阅读完成。

Ubuntu 14.04 LTS 下安装 Hadoop 1.2.1(集群分布式模式)

安装步骤:

1) JDK — Hadoop 是用 Java 写的,不安装 Java 虚拟机怎么运行 Hadoop 的程序;

2)创建专门用于运行和执行 hadoop 任务(比如 map 和 reduce 任务)的 linux 用户,就像 windows 下的服务账户,并且授权给他可以访问运行 JDK 目录权限,让他可以执行 java 虚拟机。这个账户最后用来运行 bin\.start_all.sh 启动 hadoop 的所有服务,那运行账户肯定是他了,他肯定要有足够的权限。再者,需要配置这个账户的个人环境变量,把 Java 虚拟机的主目录地址环境变量之一,不然后面运行 hadoop 任务肯定报错,根本在当前的环境变量下找不到要运行的 java 命令。

2)修改 /etc/hosts 文件和 /etc/hostname — 集群模式下机器需要互相通信,靠的是 IP 地址。而通常我们利用的是主机名称,然后通过 IP 地址和主机名称的映射来和目标主机通信。那么我们自然就得改 /etc/hosts 文件和 /etc/hostname,前者是主机和 IP 地址的映射关系,后者是主机的本地主机名;

4)SSH — 集群机器间互相访问各自的资源需要建立连接通讯,使用的是 SSH 协议进行安全通道的数据交换,再者也需要利用 SSH 的授权证书办法来实现免密码登陆目标主机;

4)然后就是安装 hadoop 了。需要修改几个配置文件。hadoop 的配置文件有很多种。

     只读型配置文件:src/core/core-default.xml, src/hdfs/hdfs-default.xml, src/mapred/mapred-default.xml, conf/mapred-queues.xml。

     定位设置:conf/core-site.xml, conf/hdfs-site.xml, conf/mapred-site.xml, conf/mapred-queues.xml。这种文件一般用于配置 hadoop 一些核心功能,比如 hdfs 和 mapred 的目录信息。

     环境配置:conf/Hadoop-env.sh

说回正题,既然 hadoop 的核心是 hdfs 和 mapreduce,那你至少要配置 hdfs 的 namenode 目录位置,datanode 的目录位置,mapreduce 的 job tracker 和 task tracker 通信的端口号,系统目录和本地目录等等。

这些在 master 和 slave 机子上都是相同的。

5)配置完后就是格式化 hdfs 了。在 master 机上格式化 hdfs 系统。

6)格式化完 hdfs 系统就启动所有的 hdfs 进程。

搭建环境用的是 VMWare Workstation 12。这里用了三台 Linux 虚拟机:master,slave1,slave2。

具体配置:

master slave1 slave2
OS Ubuntu 14.04 LTE x64 Ubuntu 14.04 LTE x64 Ubuntu 14.04 LTE x64
memory 1GB 1GB 1GB
hard drive space 20GB 20GB 20GB
processors 2 2 2
IP Address 192.168.2.110 192.168.2.111 192.168.2.112
Roles NameNode
DataNode
JobTracker
TaskTracker
SecondaryNameNode
DataNode
TaskTracker
DataNode
TaskTracker
hadoop directory /opt/hadoop /opt/hadoop /opt/hadoop
jdk version JDK 1.8 JDK 1.8 JDK 1.8

先在 master 机子上安装 JDK 和 SSH

1. JDK 安装

linuxidc@ubuntu:/run/network$ scp linuxidc@192.168.2.100:/home/linuxidc/Download/jdk-8u65-linux-x64.tar.gz ~
The authenticity of host '192.168.2.100 (192.168.2.100)' can't be established.
ECDSA key fingerprint is da:b7:c3:2a:ea:a2:76:4c:c3:c1:68:ca:0e:c2:ea:92.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.2.100' (ECDSA) to the list of known hosts.
linuxidc@192.168.2.100's password: 
scp: /home/linuxidc/Download/jdk-8u65-linux-x64.tar.gz: No such file or directory
linuxidc@ubuntu:/run/network$ scp linuxidc@192.168.2.100:/home/linuxidc/Downloads/jdk-8u65-linux-x64.tar.gz ~
linuxidc@192.168.2.100's password: 
jdk-8u65-linux-x64.tar.gz                     100%  173MB  21.6MB/s   00:08    
linuxidc@ubuntu:/run/network$ cd ~
linuxidc@ubuntu:~$ ls
Desktop    Downloads         jdk-8u65-linux-x64.tar.gz  Pictures  Templates
Documents  examples.desktop  Music                      Public    Videos
linuxidc@ubuntu:~$ cd .
linuxidc@ubuntu:~$ cd /
linuxidc@ubuntu:/$ ls
bin    dev   initrd.img  lost+found  opt   run   sys  var
boot   etc   lib         media       proc  sbin  tmp  vmlinuz
cdrom  home  lib64       mnt         root  srv   usr
linuxidc@ubuntu:/$ sudo mkdir jvm
[sudo] password for linuxidc: 
jerry@ubuntu:/$ rm jvm/
rm: cannot remove‘jvm/’: Is a directory
linuxidc@ubuntu:/$ rm -d jvm/
rm: remove write-protected directory‘jvm/’? y
rm: cannot remove‘jvm/’: Permission denied
linuxidc@ubuntu:/$ ls
bin    dev   initrd.img  lib64       mnt   root  srv  usr
boot   etc   jvm         lost+found  opt   run   sys  var
cdrom  home  lib         media       proc  sbin  tmp  vmlinuz
linuxidc@ubuntu:/$ sudo mkdir /usr/lib/jvm
linuxidc@ubuntu:/$ cd ~
linuxidc@ubuntu:~$ sudo tar zxf ./jdk-8u65-linux-x64.tar.gz -C /usr/lib/jvm/
linuxidc@ubuntu:~$ cd /usr/lib/jvm/
linuxidc@ubuntu:/usr/lib/jvm$ sudo mv jdk1.8.0_65 java
linuxidc@ubuntu:/usr/lib/jvm$ cd java/
linuxidc@ubuntu:/usr/lib/jvm/java$ sudo vim ~/.bashrc
linuxidc@ubuntu:/usr/lib/jvm/java$ tail -n 4 ~/.bashrc 
export JAVA_HOME=/usr/lib/jvm/java 
export JRE_HOME=${JAVA_HOME}/jre  
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib  
export PATH=${JAVA_HOME}/bin:$PATH 
linuxidc@ubuntu:/usr/lib/jvm/java$ java -version
The program 'java' can be found in the following packages:
 * default-jre
 * gcj-4.8-jre-headless
 * openjdk-7-jre-headless
 * gcj-4.6-jre-headless
 * openjdk-6-jre-headless
Try: sudo apt-get install <selected package>
linuxidc@ubuntu:/usr/lib/jvm/java$ sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/java/bin/java 300
update-alternatives: using /usr/lib/jvm/java/bin/java to provide /usr/bin/java (java) in auto mode
linuxidc@ubuntu:/usr/lib/jvm/java$ sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/java/bin/javac 300
update-alternatives: using /usr/lib/jvm/java/bin/javac to provide /usr/bin/javac (javac) in auto mode
linuxidc@ubuntu:/usr/lib/jvm/java$ sudo update-alternatives --install /usr/bin/jar jar /usr/lib/jvm/java/bin/jar 300
update-alternatives: using /usr/lib/jvm/java/bin/jar to provide /usr/bin/jar (jar) in auto mode
linuxidc@ubuntu:/usr/lib/jvm/java$ sudo update-alternatives --install /usr/bin/javah javah /usr/lib/jvm/java/bin/javah 300
update-alternatives: using /usr/lib/jvm/java/bin/javah to provide /usr/bin/javah (javah) in auto mode
linuxidc@ubuntu:/usr/lib/jvm/java$ sudo update-alternatives --install /usr/bin/javap javap /usr/lib/jvm/java/bin/javap 300
update-alternatives: using /usr/lib/jvm/java/bin/javap to provide /usr/bin/javap (javap) in auto mode
linuxidc@ubuntu:/usr/lib/jvm/java$ sudo update-alternatives --config java
There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java/bin/java
Nothing to configure.
linuxidc@ubuntu:/usr/lib/jvm/java$ java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
linuxidc@ubuntu:/usr/lib/jvm/java$

 2. 添加 hadoop 用户和用户组

linuxidc@ubuntu:/usr/lib/jvm/java$ sudo groupadd hadoop
[sudo] password for linuxidc: 
jerry@ubuntu:/usr/lib/jvm/java$ useradd hadoop -g hadoop
useradd: Permission denied.
useradd: cannot lock /etc/passwd; try again later.
linuxidc@ubuntu:/usr/lib/jvm/java$ sudo useradd hadoop -g hadoop
linuxidc@ubuntu:/usr/lib/jvm/java$ 

3. 修改 /etc/hosts 文件,配置 master, slave1, slave2 的主机 IP 地址和主机名的映射关系。修改 /etc/hostname 文件,配置本机的电脑名。

linuxidc@ubuntu:~$ cat /etc/hosts
127.0.0.1    localhost
127.0.1.1    master

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
master 192.168.2.110
slave1 192.168.2.111
slave2 192.168.2.112    
linuxidc@ubuntu:~$ cat /etc/hostname
master
linuxidc@ubuntu:~$

4. 安装 SSH

linuxidc@ubuntu:/usr/lib/jvm/java$ sudo apt-get install ssh
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libck-connector0 ncurses-term openssh-client openssh-server
  openssh-sftp-server ssh-import-id
Suggested packages:
  libpam-ssh keychain monkeysphere rssh molly-guard
The following NEW packages will be installed:
  libck-connector0 ncurses-term openssh-server openssh-sftp-server ssh
  ssh-import-id
The following packages will be upgraded:
  openssh-client
1 upgraded, 6 newly installed, 0 to remove and 239 not upgraded.
Need to get 617 kB/1,181 kB of archives.
After this operation, 3,450 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://us.archive.ubuntu.com/ubuntu/ trusty/main libck-connector0 amd64 0.4.5-3.1ubuntu2 [10.5 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ trusty/main ncurses-term all 5.9+20140118-1ubuntu1 [243 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main openssh-sftp-server amd64 1:6.6p1-2ubuntu2.3 [34.1 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main openssh-server amd64 1:6.6p1-2ubuntu2.3 [319 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main ssh all 1:6.6p1-2ubuntu2.3 [1,114 B]
Get:6 http://us.archive.ubuntu.com/ubuntu/ trusty/main ssh-import-id all 3.21-0ubuntu1 [9,624 B]
Fetched 617 kB in 16s (37.3 kB/s)                                              
Preconfiguring packages ...
Selecting previously unselected package libck-connector0:amd64.
(Reading database ... 166519 files and directories currently installed.)
Preparing to unpack .../libck-connector0_0.4.5-3.1ubuntu2_amd64.deb ...
Unpacking libck-connector0:amd64 (0.4.5-3.1ubuntu2) ...
Preparing to unpack .../openssh-client_1%3a6.6p1-2ubuntu2.3_amd64.deb ...
Unpacking openssh-client (1:6.6p1-2ubuntu2.3) over (1:6.6p1-2ubuntu2) ...
Selecting previously unselected package ncurses-term.
Preparing to unpack .../ncurses-term_5.9+20140118-1ubuntu1_all.deb ...
Unpacking ncurses-term (5.9+20140118-1ubuntu1) ...
Selecting previously unselected package openssh-sftp-server.
Preparing to unpack .../openssh-sftp-server_1%3a6.6p1-2ubuntu2.3_amd64.deb ...
Unpacking openssh-sftp-server (1:6.6p1-2ubuntu2.3) ...
Selecting previously unselected package openssh-server.
Preparing to unpack .../openssh-server_1%3a6.6p1-2ubuntu2.3_amd64.deb ...
Unpacking openssh-server (1:6.6p1-2ubuntu2.3) ...
Selecting previously unselected package ssh.
Preparing to unpack .../ssh_1%3a6.6p1-2ubuntu2.3_all.deb ...
Unpacking ssh (1:6.6p1-2ubuntu2.3) ...
Selecting previously unselected package ssh-import-id.
Preparing to unpack .../ssh-import-id_3.21-0ubuntu1_all.deb ...
Unpacking ssh-import-id (3.21-0ubuntu1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
ureadahead will be reprofiled on next reboot
Processing triggers for ufw (0.34~rc-0ubuntu2) ...
Setting up libck-connector0:amd64 (0.4.5-3.1ubuntu2) ...
Setting up openssh-client (1:6.6p1-2ubuntu2.3) ...
Setting up ncurses-term (5.9+20140118-1ubuntu1) ...
Setting up openssh-sftp-server (1:6.6p1-2ubuntu2.3) ...
Setting up openssh-server (1:6.6p1-2ubuntu2.3) ...
Creating SSH2 RSA key; this may take some time ...
Creating SSH2 DSA key; this may take some time ...
Creating SSH2 ECDSA key; this may take some time ...
Creating SSH2 ED25519 key; this may take some time ...
ssh start/running, process 7611
Setting up ssh-import-id (3.21-0ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Processing triggers for ufw (0.34~rc-0ubuntu2) ...
Setting up ssh (1:6.6p1-2ubuntu2.3) ...
Processing triggers for libc-bin (2.19-0ubuntu6.6) ...

5. 解压 hadoop 压缩包到本地,修改配置文件

linuxidc@ubuntu:~$ sudo tar zxf hadoop-1.2.1.tar.gz -C /opt/
[sudo] password for linuxidc: 
jerry@ubuntu:~$ cd /opt/hadoop-1.2.1/
linuxidc@ubuntu:/opt/hadoop-1.2.1$ ls
bin          hadoop-ant-1.2.1.jar          ivy          sbin
build.xml    hadoop-client-1.2.1.jar       ivy.xml      share
c++          hadoop-core-1.2.1.jar         lib          src
CHANGES.txt  hadoop-examples-1.2.1.jar     libexec      webapps
conf         hadoop-minicluster-1.2.1.jar  LICENSE.txt
contrib      hadoop-test-1.2.1.jar         NOTICE.txt
docs         hadoop-tools-1.2.1.jar        README.txt
linuxidc@ubuntu:/opt/hadoop-1.2.1$ cd conf/
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ ls
capacity-scheduler.xml      hadoop-policy.xml      slaves
configuration.xsl           hdfs-site.xml          ssl-client.xml.example
core-site.xml               log4j.properties       ssl-server.xml.example
fair-scheduler.xml          mapred-queue-acls.xml  taskcontroller.cfg
hadoop-env.sh               mapred-site.xml        task-log4j.properties
hadoop-metrics2.properties  masters
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ vim hadoop-env.sh 
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ sudo vim hadoop-env.sh 
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ tail -n 1 hadoop-env.sh 
export JAVA_HOME=/usr/lib/jvm/java
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ sudo vim core-site.xml 
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ cat core-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://master:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/hadoop/hadooptmp</value>
    </property>
</configuration>
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ sudo vim hdfs-site.xml 
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ cat hdfs-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.name.dir</name>
        <value>/hadoop/hdfs/name</value>
    </property>
    <property>
        <name>dfs.data.dir</name>
        <value>/hadoop/hdfs/data</value>
    </property>
</configuration>
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ sudo vim mapred-
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ sudo vim mapred-site.xml 
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ cat mapred-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>master:9001</value>
</property>
</configuration>
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ sudo vim mapred-site.xml 
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ cat mapred-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
    <name>mapred.job.tracker</name>
    <value>master:9001</value>
</property>
<property>
     <name>mapred.system.dir</name>  
     <value>/hadoop/mapred/mapred_system</value>
</property>
<property>
     <name>mapred.local.dir</name>  
     <value>/hadoop/mapred/mapred_local</value>
</property>
</configuration>
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ ls
capacity-scheduler.xml      hadoop-policy.xml      slaves
configuration.xsl           hdfs-site.xml          ssl-client.xml.example
core-site.xml               log4j.properties       ssl-server.xml.example
fair-scheduler.xml          mapred-queue-acls.xml  taskcontroller.cfg
hadoop-env.sh               mapred-site.xml        task-log4j.properties
hadoop-metrics2.properties  masters
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ sudo vim masters 
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ sudo vim slaves 
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ cat masters 
master
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$ cat slaves 
master
slave1
slave2
linuxidc@ubuntu:/opt/hadoop-1.2.1/conf$

在 master、slave1 和 slave2 上的 hadoop 用户目录下创建好.ssh 文件夹

linuxidc@slave1:~$ sudo mkdir -p /hadoop/hdfs/name
[sudo] password for linuxidc: 
jerry@slave1:~$ sudo mkdir -p /hadoop/hdfs/data
linuxidc@slave1:~$ sudo mkdir -p /hadoop/mapred/mapred_system
linuxidc@slave1:~$ sudo mkdir -p /hadoop/mapred/mapred_local
linuxidc@slave1:~$ sudo mkdir -p /hadoop/hadooptmp
linuxidc@slave1:~$ chown -R hadoop:hadoop /hadoop/
chown: changing ownership of‘/hadoop/mapred/mapred_local’: Operation not permitted
chown: changing ownership of‘/hadoop/mapred/mapred_system’: Operation not permitted
chown: changing ownership of‘/hadoop/mapred’: Operation not permitted
chown: changing ownership of‘/hadoop/hdfs/data’: Operation not permitted
chown: changing ownership of‘/hadoop/hdfs/name’: Operation not permitted
chown: changing ownership of‘/hadoop/hdfs’: Operation not permitted
chown: changing ownership of‘/hadoop/hadooptmp’: Operation not permitted
chown: changing ownership of‘/hadoop/’: Operation not permitted
linuxidc@slave1:~$ sudo chown -R hadoop:hadoop /hadoop/
linuxidc@slave1:~$

在 master 上生存 ssh 秘钥然后拷贝到 slave1 和 slave2 上的 hadoop 用户主目录下的.ssh 文件目录下

hadoop@master:~$ sudo mkdir ~/.ssh
[sudo] password for hadoop: 
hadoop@master:~$ sudo chown -R hadoop:hadoop  ~/.ssh 
hadoop@master:~$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
36:99:e0:e9:aa:b8:c3:c6:cd:97:59:9f:49:87:0d:bf hadoop@master
The key's randomart image is:
+--[RSA 2048]----+
|                 |
|                 |
|      .          |
|     . o +       |
|      o S =      |
|     . o + +     |
|o o   = o + .    |
|.= o =   + E     |
|+o..o            |
+-----------------+
hadoop@master:~/.ssh$ scp authorized_keys slave1:/home/hadoop/.ssh
The authenticity of host 'slave1 (192.168.2.111)' can't be established.
ECDSA key fingerprint is 48:93:30:0d:bb:3a:85:da:46:3f:75:76:3e:b7:42:6a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave1,192.168.2.111' (ECDSA) to the list of known hosts.
hadoop@slave1's password: 
scp: /home/hadoop/.ssh/authorized_keys: Permission denied
hadoop@master:~/.ssh$ scp authorized_keys slave1:/home/hadoop/.ssh
hadoop@slave1's password: 
authorized_keys                               100%  395     0.4KB/s   00:00    
hadoop@master:~/.ssh$ scp authorized_keys slave2:/home/hadoop/.ssh
The authenticity of host 'slave2 (192.168.2.112)' can't be established.
ECDSA key fingerprint is 48:93:30:0d:bb:3a:85:da:46:3f:75:76:3e:b7:42:6a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave2,192.168.2.112' (ECDSA) to the list of known hosts.
hadoop@slave2's password: 
authorized_keys                               100%  395     0.4KB/s   00:00    
hadoop@master:~/.ssh$

再测试下是否再次 ssh 连接的时候需要键入密码。结果是不需要的。

hadoop@master:~/.ssh$ ssh slave1
Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.19.0-42-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

0 packages can be updated.
0 updates are security updates.

$ ls -al
total 239436
drwxr-xr-x 12 hadoop hadoop      4096 Jan  9 22:37 .
drwxr-xr-x  4 root   root        4096 Jan  9 22:25 ..
drwx------  2 hadoop hadoop      4096 Jan  9 22:37 .cache
drwxr-xr-x  2 hadoop hadoop      4096 Dec 27 04:19 Desktop
drwxr-xr-x  2 hadoop hadoop      4096 Dec 27 04:19 Documents
drwxr-xr-x  2 hadoop hadoop      4096 Dec 27 04:19 Downloads
-rw-r--r--  1 hadoop hadoop      8980 Dec 27 03:27 examples.desktop
-rw-rw-r--  1 hadoop hadoop  63851630 Dec 30 05:25 hadoop-1.2.1.tar.gz
-rw-rw-r--  1 hadoop hadoop 181260798 Dec 27 06:49 jdk-8u65-linux-x64.tar.gz
drwxr-xr-x  2 hadoop hadoop      4096 Dec 27 04:19 Music
drwxr-xr-x  2 hadoop hadoop      4096 Dec 27 04:19 Pictures
drwxr-xr-x  2 hadoop hadoop      4096 Dec 27 04:19 Public
drwxr-xr-x  2 hadoop hadoop      4096 Jan  9 22:39 .ssh
drwxr-xr-x  2 hadoop hadoop      4096 Dec 27 04:19 Templates
drwxr-xr-x  2 hadoop hadoop      4096 Dec 27 04:19 Videos
$ 
hadoop@master:~/.ssh$ ssh slave2
Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.19.0-42-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

27 packages can be updated.
23 updates are security updates.

$ ls -l
total 239424
drwxr-xr-x 2 hadoop hadoop      4096 Dec 27 04:19 Desktop
drwxr-xr-x 2 hadoop hadoop      4096 Dec 27 04:19 Documents
drwxr-xr-x 2 hadoop hadoop      4096 Dec 27 04:19 Downloads
-rw-r--r-- 1 hadoop hadoop      8980 Dec 27 03:27 examples.desktop
-rw-rw-r-- 1 hadoop hadoop  63851630 Dec 30 05:25 hadoop-1.2.1.tar.gz
-rw-rw-r-- 1 hadoop hadoop 181260798 Dec 27 06:49 jdk-8u65-linux-x64.tar.gz
drwxr-xr-x 2 hadoop hadoop      4096 Dec 27 04:19 Music
drwxr-xr-x 2 hadoop hadoop      4096 Dec 27 04:19 Pictures
drwxr-xr-x 2 hadoop hadoop      4096 Dec 27 04:19 Public
drwxr-xr-x 2 hadoop hadoop      4096 Dec 27 04:19 Templates
drwxr-xr-x 2 hadoop hadoop      4096 Dec 27 04:19 Videos
$ exit
Connection to slave2 closed.
hadoop@master:~/.ssh$

格式化 hdfs 的 namenode

hadoop@master:/opt/hadoop-1.2.1/bin$ ./hadoop namenode -format16/01/09 23:33:18 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = master/127.0.1.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.2.1
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG:   java = 1.8.0_65
************************************************************/
Re-format filesystem in /hadoop/hdfs/name ? (Y or N) Y
16/01/09 23:34:09 INFO util.GSet: Computing capacity for map BlocksMap
16/01/09 23:34:09 INFO util.GSet: VM type       = 64-bit
16/01/09 23:34:09 INFO util.GSet: 2.0% max memory = 1013645312
16/01/09 23:34:09 INFO util.GSet: capacity      = 2^21 = 2097152 entries
16/01/09 23:34:09 INFO util.GSet: recommended=2097152, actual=2097152
16/01/09 23:34:10 INFO namenode.FSNamesystem: fsOwner=hadoop
16/01/09 23:34:10 INFO namenode.FSNamesystem: supergroup=supergroup
16/01/09 23:34:10 INFO namenode.FSNamesystem: isPermissionEnabled=true
16/01/09 23:34:10 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
16/01/09 23:34:10 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
16/01/09 23:34:10 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
16/01/09 23:34:10 INFO namenode.NameNode: Caching file names occuring more than 10 times 
16/01/09 23:34:11 INFO common.Storage: Image file /hadoop/hdfs/name/current/fsimage of size 112 bytes saved in 0 seconds.
16/01/09 23:34:11 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/hadoop/hdfs/name/current/edits
16/01/09 23:34:11 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/hadoop/hdfs/name/current/edits
16/01/09 23:34:11 INFO common.Storage: Storage directory /hadoop/hdfs/name has been successfully formatted.
16/01/09 23:34:11 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/127.0.1.1
************************************************************/
hadoop@master:/opt/hadoop-1.2.1/bin$

在 master 上启动 hadoop 的所有服务

hadoop@master:/opt/hadoop-1.2.1/bin$ ./start-all.sh 
starting namenode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-namenode-master.out
slave2: starting datanode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-slave2.out
slave1: starting datanode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-slave1.out
master: starting datanode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-master.out
master: starting secondarynamenode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-secondarynamenode-master.out
starting jobtracker, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-jobtracker-master.out
slave2: starting tasktracker, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-slave2.out
slave1: starting tasktracker, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-slave1.out
master: starting tasktracker, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-master.out
hadoop@master:/opt/hadoop-1.2.1/bin$ jps
5088 JobTracker
5010 SecondaryNameNode
4871 DataNode
4732 NameNode
5277 Jps
5230 TaskTracker
hadoop@master:/opt/hadoop-1.2.1/bin$

观察 slave1

hadoop@slave1:~/.ssh$ jps
3669 TaskTracker
3718 Jps
3560 DataNode
hadoop@slave1:~/.ssh$

观察 slave2

hadoop@slave2:~$ jps
3216 Jps
2982 TaskTracker
3095 DataNode

过程中需要注意的问题:

你既然是用 hadoop 用户去启动 hadoop 服务的,那这个用户至少得对 java、hadoop 程序有访问执行权限吧,这点要注意。

其次,这个用户对 hadoop 的 hdfs 的文件和 mapreduce 文件都需要有写入权限吧

问题一:

hadoop@master:/usr/lib/jvm/java/bin$ jps
The program 'jps' can be found in the following packages:
 * openjdk-7-jdk
 * openjdk-6-jdk
Ask your administrator to install one of them

明明都安装了 jdk,而且环境变量也没有出现路径错误,为什么就是找不到 jps 命令呢?结果还是要用 update-alternative 命令来解决这个问题

hadoop@master:/usr/lib/jvm/java/bin$  sudo update-alternatives --install /usr/bin/jps jps /usr/lib/jvm/java/bin/jps 1
update-alternatives: using /usr/lib/jvm/java/bin/jps to provide /usr/bin/jps (jps) in auto mode

更多 Hadoop 相关信息见 Hadoop 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=13

本文永久更新链接地址 :http://www.linuxidc.com/Linux/2016-09/135402.htm

正文完
星哥玩云-微信公众号
post-qrcode
 0
星锅
版权声明:本站原创文章,由 星锅 于2022-01-21发表,共计20643字。
转载说明:除特殊说明外本站文章皆由CC-4.0协议发布,转载请注明出处。
【腾讯云】推广者专属福利,新客户无门槛领取总价值高达2860元代金券,每种代金券限量500张,先到先得。
阿里云-最新活动爆款每日限量供应
评论(没有评论)
验证码
【腾讯云】云服务器、云数据库、COS、CDN、短信等云产品特惠热卖中