Hadoop2.6.0的Eclipse插件编译和设置

181次阅读

共计 3543 个字符，预计需要花费 9 分钟才能阅读完成。

编译 Hadoop2.6.0 的 eclipse 插件

下载源码：

git clone https://github.com/winghc/hadoop2x-eclipse-plugin.git

编译源码：

src/contrib/eclipse-plugin
ant jar -Dversion=2.6.0 -Declipse.home=/usr/local/eclipse -Dhadoop.home=/usr/local/hadoop-2.6.0 // 需要手动安装的 eclipse，通过命令行一键安装的不行

eclipse.home 和 hadoop.home 设置成你自己的环境路径

生成位置：

[jar] Building jar: /home/hunter/hadoop2x-eclipse-plugin/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-2.6.0.jar

安装插件

登录桌面后面要打开 eclipse 的用户最好是 hadoop 的管理员，也就是 hadoop 安装时设置的那个用户，否则会出现拒绝读写权限问题。

复制编译好的 jar 到 eclipse 插件目录，重启 eclipse

配置 hadoop 安装目录

window ->preference -> hadoop Map/Reduce -> Hadoop installation directory

配置 Map/Reduce 视图

window ->Open Perspective -> other->Map/Reduce -> 点击“OK”

windows → show view → other->Map/Reduce Locations-> 点击“OK”

控制台会多出一个“Map/Reduce Locations”的 Tab 页

在“Map/Reduce Locations”Tab 页点击图标 < 大象 +> 或者在空白的地方右键，选择“New Hadoop location…”，弹出对话框“New hadoop location…”，配置如下内容：将 ha1 改为自己的 hadoop 用户

Hadoop2.6.0 的 Eclipse 插件编译和设置

注意：MR Master 和 DFS Master 配置必须和 mapred-site.xml 和 core-site.xml 等配置文件一致

打开 Project Explorer, 查看 HDFS 文件系统。

新建 Map/Reduce 任务

File->New->project->Map/Reduce Project->Next

编写 WordCount 类：记得先把服务都起来

import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
public class WordCount {
public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
output.collect(word, one);
}
}
}
public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
int sum = 0;
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
JobConf conf = new JobConf(WordCount.class);
conf.setJobName(“wordcount”);