Hive學習
標簽（空格分隔）： Hadoop Hive
官網地址：https://hive.apache.org/
下載地址：http://www-us.apache.org/dist/hive/hive-2.1.1/
官方文檔：https://cwiki.apache.org/confluence/display/Hive/GettingStarted
安裝與配置
# 下載解壓
wget http://www-eu.apache.org/dist/hive/hive-2.1.1/apache-hive-2.1.1-bin.tar.gz
tar zxvf apache-hive-2.1.1-bin.tar.gz
cd apache-hive-2.1.1-bin
# 配置mysql
cp conf/hive-default.xml.template conf/hive-site.xml
mv /path/to/mysql-connector-java-5.1.23-bin.jar lib/
vim conf/hive-site.xml
修改以下屬性：
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://127.0.0.1:3306/hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>password</value>
</property>
配置全局變量：
- 將hive的bin目錄加入PATH中，方便運行
運行：
# run the schematool command below as an initialization step
schematool --dbType mysql -initSchema
# To run HiveServer2 and Beeline from shell:
hiveserver2
beeline -u jdbc:hive2://localhost:10000
# Or to start Beeline and HiveServer2 in the same process for testing purpose
beeline -u jdbc:hive2://
異常：
Relative path in absolute URI: ${system:user.name%7D
Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
解決方法：
在hive-site.xml中加入：
<property>
<name>system:java.io.tmpdir</name>
<value>/home/lyndon/tmp/hive</value>
</property>
<property>
<name>system:user.name</name>
<value>lyndon</value>
</property>
安裝Hive的web管理界面
wget http://www-us.apache.org/dist/hive/hive-2.1.1/apache-hive-2.1.1-src.tar.gz
tar zxvf apache-hive-2.1.1-src.tar.gz
cd apache-hive-2.1.1-src/hwi
jar cvfM0 hive-hwi-2.1.1.war -C web/ .
cp hive-hwi-2.1.1.war /opt/hive/lib
https://cwiki.apache.org/confluence/display/Hive/HiveWebInterface
HWI is only available in Hive releases prior to 2.2.0. It was removed by HIVE-15622.
配置失敗
Hive數據導入
使用Load
- 語法：
LOAD DATA [LOCAL(是否為本地文件/HDFS上)] INPATH 'filepath' [OVERWRITE]
INTO TABLE tablename [PARTITION (partcol1=val1, partcol2=val2...)]
使用Sqoop
- 使用1.99.7版本
下載地址： http://www-eu.apache.org/dist/sqoop/1.99.7/
官方文檔： http://sqoop.apache.org/docs/1.99.7/
axel -n 10 http://www-eu.apache.org/dist/sqoop/1.99.7/sqoop-1.99.7-bin-hadoop200.tar.gz
tar zxvf sqoop-1.99.7-bin-hadoop200.tar.gz
sudo mv sqoop-1.99.7-bin-hadoop200 /usr/lib/sqoop
sudo mkdir -p /var/lib/lyndon
sudo cp ~/Downloads/mysql-connector-java-5.1.23-bin.jar /var/lib/lyndon/
export SQOOP_SERVER_EXTRA_LIB=/var/lib/lyndon/
...未繼續
- 使用1.4.6版本
查看分隔符：
desc formatted stations;默認:\1(SOH)
修改HIVE序列化存儲分隔符：alter table tablename set SERDEPROPERTIES('field.delim'=',');
- 安裝與配置
axel -n 10 http://www-us.apache.org/dist/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
tar zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
sudo tar zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
cd sqoop-1.4.6.bin__hadoop-2.0.4-alpha
cd bin
sudo cp ~/Downloads/mysql-connector-java-5.1.23-bin.jar ../lib
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
導入到HDFS：
./sqoop import --connect jdbc:mysql://127.0.0.1:3306/test --username root --password l --table stations --target-dir '/sqoop/stations'導入到HIVE：
./sqoop import --hive-import --connect jdbc:mysql://127.0.0.1:3306/test --username root --password l --table stations自定義表名，添加篩選條件：
./sqoop import --hive-import --connect jdbc:mysql://127.0.0.1:3306/test --username root --password l --table stations --hive-table stations_bj --where 'fchar like "%bj%"'使用query查詢語句:
使用--query，不能包含--table參數，必須包含--target-dir和--split-by
條件中必須有 $CONDITIONS
./sqoop import --hive-import --connect jdbc:mysql://127.0.0.1:3306/test --username root --password l --hive-table stations_q --query 'select * from stations where fchar="bj" and $CONDITIONS' --target-dir '/sqoop/stations_q' --split-by 'id'導出到MYSQL：
必須先在MYSQL中創建此表，且對應列關系
./sqoop export --connect jdbc:mysql://127.0.0.1:3306/test --username root --password l -m 1 --table stations_bj --export-dir '/user/hive/warehouse/stations_q'
JDBC接口
注意
- 需要導入Hive中lib下面的jar包
- 注冊JDBC驅動使用
org.apache.hive.jdbc.HiveDriver - 創建鏈接的URL為：
jdbc:hive2://localhost:10000需要寫入用戶名，否則報權限異常。如：Connection connection = DriverManager.getConnection("jdbc:hive2://localhost:10000", "lyndon", ""); - 使用
statement.execute()執行不返回ResultSet的sql語句，否則報錯。
示例代碼：
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
/**
* Created by lyndon on 17-4-2.
*/
public class HiveJdbcClient {
public static void main(String[] args) throws Exception {
// 注冊JDBC驅動
try {
Class.forName("org.apache.hive.jdbc.HiveDriver");
} catch (ClassNotFoundException e) {
e.printStackTrace();
System.exit(1);
}
// 創建鏈接
Connection connection = DriverManager.getConnection("jdbc:hive2://localhost:10000", "lyndon", "");
Statement statement = connection.createStatement();
String tableName = "u1_data";
statement.execute("DROP TABLE " + tableName);
statement.execute("CREATE TABLE " + tableName +
"(userid int," +
"movieid int," +
"rating int," +
"unixtime int)" +
"row format delimited fields terminated by ','" +
"stored as textfile");
String sql = "show tables";
System.out.println("Running: " + sql);
ResultSet resultSet = statement.executeQuery(sql);
while (resultSet.next()) {
System.out.println(resultSet.getString(1));
}
// load data
String filepath = "/home/lyndon/install/ratings.csv";
sql = "load data local inpath '" + filepath + "' overwrite into table " + tableName;
System.out.println("Running: " + sql);
statement.execute(sql);
// 查詢
sql = "SELECT * FROM " + tableName + " LIMIT 5";
System.out.println("Running: " + sql);
resultSet = statement.executeQuery(sql);
while (resultSet.next()) {
System.out.println(resultSet.getString(2)+"\t"+resultSet.getString(3));
}
}
}
