编译Hadoop3.1.3 eclipse插件

Hadoop3.1.3源代码编译成功后，解压释放到/opt目录下

1
2
3
4
解压释放：
tar xvfz hadoop-3.1.3.tar.gz -C /opt
然后确定文件夹名为hadoop
mv hadoop-3.1.3 hadoop

安装eclipse

eclipse下载链接

1
2
3
4
5
6
7
8
9
10
11
12
#解压Eclipse软件
tar xvfz eclipse-java-2020-03-R-linux-gtk-x86_64.tar.gz -C /opt
#确保opt目录下解压的文件夹名为eclipse，不是的话mv修改
#添加环境变量
cat>> /etc/profile<<EOF
export ECLIPSE_HOME=/opt/eclipse
export PATH=\$ECLIPSE_HOME:\$PATH
EOF
#重启
reboot
#或
source /etc/profile

下载hadoop3x-eclipse-plugin源代码

1
2
git clone https://github.com/Woooosz/eclipse-hadoop3x
cd eclipse-hadoop3x
修改依赖包版本文件libraries.properties
1
cd eclipse-hadoop3x/ivy#文件路径

修改办法：

1
2
3
#列出相应的版本号
cd /opt/hadoop/share/hadoop/common/lib
ls

对着列出来的版本一个个修改libraries.properties里面的版本号（对的上的就改，对不上的就忽略，如果编译失败看一下输出的信息是那个jar版本对不上了，然后在bulid.xml里面看xxx.jar在哪里，说明xxx.jar前面的jar都找到了，这样去校对libraries.properties的时候可以直接从xxx.jar后面开始校对）。

额外注意：

1
2
3
4
5
#修改
commons-httpclient.version=3.0.1
#添加
stax2.version=3.1.4
woodstox.version=5.0.3

测试可编译通过libraries.properties：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

#This properties file lists the versions of the various artifacts used by hadoop and components.
#It drives ivy and the generation of a maven POM

# This is the version of hadoop we are generating
hadoop.version=2.7.7
hadoop-gpl-compression.version=0.1.0

#These are the versions of our dependencies (in alphabetical order)
apacheant.version=1.9.11
ant-task.version=2.0.10

asm.version=3.2
aspectj.version=1.6.5
aspectj.version=1.6.11

checkstyle.version=4.2

commons-cli.version=1.2
commons-codec.version=1.11
commons-collections.version=3.2.2
commons-configuration.version=2.1.1
commons-daemon.version=1.0.13
commons-httpclient.version=3.0.1
commons-lang.version=2.6
commons-logging.version=1.1.3
commons-logging-api.version=1.1.3
commons-math.version=3.1.1
commons-el.version=1.0
commons-fileupload.version=1.2
commons-io.version=2.4
commons-net.version=3.1
core.version=3.1.1
coreplugin.version=1.3.2

hsqldb.version=1.8.0.10

ivy.version=2.1.0

jasper.version=5.5.12
jackson.version=1.9.13
#not able to figureout the version of jsp & jsp-api version to get it resolved throught ivy
# but still declared here as we are going to have a local copy from the lib folder
jsp.version=2.1
jsp-api.version=5.5.12
jsp-api-2.1.version=6.1.14
jsp-2.1.version=6.1.14
jets3t.version=0.6.1
jetty.version=6.1.26
jetty-util.version=6.1.26
jersey-core.version=1.8
jersey-json.version=1.8
jersey-server.version=1.8
junit.version=4.5
jdeb.version=0.8
jdiff.version=1.0.9
json.version=1.0
protobuf.version=2.5.0

kfs.version=0.1
netty.version=3.10.5.Final
log4j.version=1.2.17
lucene-core.version=2.3.1
htrace.version=4.1.0-incubating
mockito-all.version=1.8.5
jsch.version=0.1.42

oro.version=2.0.8

rats-lib.version=0.5.1

servlet.version=4.0.6
servlet-api.version=2.5
slf4j-api.version=1.7.25
slf4j-log4j12.version=1.7.25
stax2.version=3.1.4
guava.version=27.0

wagon-http.version=1.0-beta-2
woodstox.version=5.0.3
xmlenc.version=0.52
xerces.version=1.4.4
修改build.xml文件
1
2
#路径
cd eclipse-hadoop3x/src/contrib/eclipse-plugin
  • 第1处修改：跳过查找不到的jar包

修改66行，将原来的target中的depends=“init, ivy-retrieve-common” 去掉其中的 ivy-retrieve-common

修改前：

修改后：

  • 第2处修改：让classpath包含Hadoop的jar

    添加

1
2
3
4
5
<!-- new add start-->
<fileset dir="${hadoop.root}">
<include name="**/*.jar" />
</fileset>
<!-- new add end-->
  • 第3处修改：包含ant runtime

修改前：

修改后：

  • 第4处修改：解决Add New Location没有反应的问题

在120行左右添加

1
2
<copy file="${hadoop.home}/libexec/share/hadoop/common/lib/woodstox-core-${woodstox.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/libexec/share/hadoop/common/lib/stax2-api-${stax2.version}.jar" todir="${build.dir}/lib" verbose="true"/>
  • 第5处修改：确保编译能找到hadoop的jar包
1
2
3
#建议不修改xml文件，直接创建链接：
cd /opt/hadoop/libexec
ln -s /opt/hadoop/share share
  • 第6处修改：guava的jar包文件命和hadoop3.1.3不一致

修改前：

修改后：

编译eclipse插件

1
2
3
4
5
cd eclipse-hadoop3x/src/contrib/eclipse-plugin

ant jar -Dversion=3.1.3  -Dhadoop.version=3.1.3 \
-Declipse.home=/opt/eclipse \
-Dhadoop.home=/opt/hadoop -debug

编译好的jar包:

1
cd eclipse-hadoop3x/build/contrib/eclipse-plugin

Linux Hadoop eclipse

本博客所有文章除特别声明外，均采用 CC BY-SA 4.0 协议 ，转载请注明出处！