1.安裝配置TEZ
1.1 環境要求
- CDH5.9.2(hadoop2.6.0)
- 編譯環境：gcc, gcc-c++, make, build
- Nodejs、npm (Tez-ui需要)
- Git
- pb2.5.0
- maven3
- Tez0.8.5
1.2編譯環境準備
安裝gcc, gcc-c++, make, build
yum install gcc gcc-c++ libstdc++-devel make build
安裝Nodejs，npm
wget http://nodejs.org/dist/v0.8.14/node-v0.8.14.tar.gz
./configure
make && make install
安裝GIt
https://git-scm.com/download ./configuremakemake install
安裝ProtocolBuffer.5.0
https://github.com/google/protobuf/releases/tag/v2.5.0
./configure
make && make install
1.3 編譯TEZ
1.3.1 官網下載tez
1.3.2 解壓
1.3.3修改源碼
/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/hadoop/mapreduce/JobContextImpl.java
diff --git a/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/hadoop/mapreduce/JobContextImpl.java b/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/hadoop/mapreduce/JobContextImpl.java
index 12491ed..b4ca24c 100644
--- a/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/hadoop/mapreduce/JobContextImpl.java
+++ b/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/hadoop/mapreduce/JobContextImpl.java
@@ -475,5 +475,16 @@ public class JobContextImpl implements JobContext {
public Progressable getProgressible() {
return progress;
}
+
+ /**
+ * Get the boolean value for the property that specifies which classpath
+ * takes precedence when tasks are launched. True - user's classes takes
+ * precedence. False - system's classes takes precedence.
+ * @return true if user's classes should take precedence
+ */
+ @Override
+ public boolean userClassesTakesPrecedence() {
+ return getJobConf().getBoolean(MRJobConfig.MAPREDUCE_JOB_USER_CLASSPATH_FIRST, false);
+ }
}
/tez-ext-service-tests/src/test/java/org/apache/tez/shufflehandler/ShuffleHandler.java
I figured out this answer from my coworker we have to use "headers().set, headers().get" instead of "setHeader(), getHeader()".
1.3.4 修改配置
vi pom.xml
<profile>
<id>cdh5.9.2</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<hadoop.version>2.5.0-cdh5.2.5</hadoop.version>
</properties>
<pluginRepositories>
<pluginRepository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</pluginRepository>
</pluginRepositories>
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
</profile>
vi tez-ui/pom.xml
<nodeVersion>v0.12.9</nodeVersion>
<npmVersion>2.14.9</npmVersion>
1.3.5 開始編譯
mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true -Dfrontend-maven-plugin.version=0.0.23
注意：出現node.gz.tar文件下載失敗 到tez-ui下手動編譯：
手動編譯tez-ui 和tez-ui2 使用taobao的源
npm --registry=https://registry.npm.taobao.org install --verbose
2.Hive On Tez
- 拷貝tez-0.8.2-minimal目錄至HDFS
hdfs dfs -put tez-dist/target/tez-0.8.5-minimal tez-dir/
- 把hadoop-mapreduce-client-common-2.6.0-cdh5.9.2.jar拷貝到hdfs的/tez-dir/tez-0.8.5-minimal目錄
- 把tez-0.8.5下jar和lib下的jar拷貝到hive客戶端部署的lib目錄，刪除hive/auxlib下的hive-exec-1.1.0-cdh5.9.3-core.jar和hive-exec-core.jar 否則會有kryo錯誤。
- 創建tez.size.xml 保存到/etc/hive/conf/
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/tez-dir/tez-0.8.5-minimal,${fs.defaultFS}/tez-dir/tez-0.8.5-minimal/lib</value>
</property>
<property>
<name>tez.use.cluster.hadoop-libs</name>
<value>true</value>
</property>
</configuration>
- 在hive中使用tez
set hive.execution.engine=tez;
截止2017-09-27日，已在公司線上環境使用3個多月，由于時間問題tez-ui沒有成功整合。有時間會解決下ui問題。有朋友成功整合tez-ui的也可分享下。
