hadoop遇到問題總結 - 蝸牛123 - 博客園
http://www.cnblogs.com/itgg168/archive/2012/11/24/2786088.html
問題一
hadoop fs -ls時出現錯誤如下：
hadoop fs -ls
11/08/31 22:51:39 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 0 time(s).
Bad connection to FS. command aborted.
解決方案:
- 格式化namenode:
hadoop namenode -format
- 重新啟動hadoop
sh stop-all.sh
sh start-all.sh
- 查看后臺進程
jps
13508 NameNode
11008 SecondaryNameNode
14393 Jps
11096 JobTracker
此時namenode啟動
- 運行
hadoop fs -ls
12/01/31 14:04:39 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
12/01/31 14:04:39 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
Found 1 items
drwxr-xr-x - root supergroup 0 2012-01-31 13:57 /user/root/test
問題二# hadoop fs -put ../conf input 時出現錯誤如下：
12/01/31 16:01:25 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
12/01/31 16:01:25 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
12/01/31 16:01:26 WARN hdfs.DFSClient: DataStreamer Exception: java.io.IOException: File /user/root/input/ssl-server.xml.example could only be replicated to 0 nodes, instead of 1
put: File /user/root/input/ssl-server.xml.example could only be replicated to 0 nodes, instead of 1
12/01/31 16:01:26 ERROR hdfs.DFSClient: Exception closing file /user/root/input/ssl-server.xml.example : java.io.IOException: File /user/root/input/ssl-server.xml.example could only be replicated to 0 nodes, instead of 1
解決方案：
這個問題是由于沒有添加節點的原因，也就是說需要先啟動namenode，再啟動datanode，然后啟動jobtracker和tasktracker。這樣就不會存在這個問題了。 目前解決辦法是分別啟動節點#hadoop-daemon.sh start namenode #$hadoop-daemon.sh start datanode
- 重新啟動namenode
hadoop-daemon.sh stop namenode
stopping namenode
hadoop-daemon.sh start namenode
starting namenode, logging to /usr/hadoop-0.21.0/bin/../logs/hadoop-root-namenode-www.keli.com.out
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
- 重新啟動datanode
hadoop-daemon.sh stop datanode
stopping datanode
hadoop-daemon.sh start datanode
starting datanode, logging to /usr/hadoop-0.21.0/bin/../logs/hadoop-root-datanode-www.keli.com.out
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
- 切換到hadoop的bin目錄
cd /usr/hadoop-0.21.0/bin/
瀏覽hdfs目錄
[root@www bin]# hadoop fs -ls
12/01/31 16:09:45 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
12/01/31 16:09:45 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
Found 4 items
drwxr-xr-x - root supergroup 0 2012-01-31 16:01 /user/root/input
drwxr-xr-x - root supergroup 0 2012-01-31 15:24 /user/root/test
-rw-r--r-- 1 root supergroup 0 2012-01-31 14:37 /user/root/test-in
drwxr-xr-x - root supergroup 0 2012-01-31 14:32 /user/root/test1刪除hdfs中的input目錄
[root@www bin]# hadoop fs -rmr input
12/01/31 16:10:09 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
12/01/31 16:10:09 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
Deleted hdfs://m106:9000/user/root/input上傳數據到hdfs中的input目錄
[root@www bin]# hadoop fs -put ../conf input
12/01/31 16:10:14 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
12/01/31 16:10:14 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id瀏覽input目錄，檢查已上傳的數據
[root@www bin]# hadoop fs -ls input
12/01/31 16:10:21 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
12/01/31 16:10:21 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
Found 16 items
-rw-r--r-- 1 root supergroup 3426 2012-01-31 16:10 /user/root/input/capacity-scheduler.xml
-rw-r--r-- 1 root supergroup 1335 2012-01-31 16:10 /user/root/input/configuration.xsl
-rw-r--r-- 1 root supergroup 757 2012-01-31 16:10 /user/root/input/core-site.xml
-rw-r--r-- 1 root supergroup 321 2012-01-31 16:10 /user/root/input/fair-scheduler.xml
-rw-r--r-- 1 root supergroup 2237 2012-01-31 16:10 /user/root/input/hadoop-env.sh
-rw-r--r-- 1 root supergroup 1650 2012-01-31 16:10 /user/root/input/hadoop-metrics.properties
-rw-r--r-- 1 root supergroup 4644 2012-01-31 16:10 /user/root/input/hadoop-policy.xml
-rw-r--r-- 1 root supergroup 252 2012-01-31 16:10 /user/root/input/hdfs-site.xml
-rw-r--r-- 1 root supergroup 4141 2012-01-31 16:10 /user/root/input/log4j.properties
-rw-r--r-- 1 root supergroup 2997 2012-01-31 16:10 /user/root/input/mapred-queues.xml
-rw-r--r-- 1 root supergroup 430 2012-01-31 16:10 /user/root/input/mapred-site.xml
-rw-r--r-- 1 root supergroup 25 2012-01-31 16:10 /user/root/input/masters
-rw-r--r-- 1 root supergroup 26 2012-01-31 16:10 /user/root/input/slaves
-rw-r--r-- 1 root supergroup 1243 2012-01-31 16:10 /user/root/input/ssl-client.xml.example
-rw-r--r-- 1 root supergroup 1195 2012-01-31 16:10 /user/root/input/ssl-server.xml.example
-rw-r--r-- 1 root supergroup 250 2012-01-31 16:10 /user/root/input/taskcontroller.cfg
[root@www bin]#
問題三Hadoop啟動datanode時出現Unrecognized option: -jvm 和 Could not create the Java virtual machine.
[root@www bin]# hadoop-daemon.sh start datanode
starting datanode, logging to /usr/hadoop-0.20.203.0/bin/../logs/hadoop-root-datanode-www.keli.com.out
Unrecognized option: -jvm
Could not create the Java virtual machine.
解決辦法：
在hadoop安裝目錄/bin/hadoop中有如下一段shell:
CLASS='org.apache.hadoop.hdfs.server.datanode.DataNode'
if [[ $EUID -eq 0 ]]; then
HADOOP_OPTS="$HADOOP_OPTS -jvm server $HADOOP_DATANODE_OPTS"
else
HADOOP_OPTS="$HADOOP_OPTS -server $HADOOP_DATANODE_OPTS"
fi
其中的
if [[ $EUID -eq 0 ]]; then
HADOOP_OPTS="$HADOOP_OPTS -jvm server $HADOOP_DATANODE_OPTS"
如果 $EUID 為 0，什么意思呢?
有效用戶標識號（EUID）：該標識號負責標識以什么用戶身份來給新創建的進程賦所有權、檢查文件的存取權限和檢查通過系統調用kill向進程發送軟中斷信號的許可權限。
在root用戶下echo $EUID，echo結果為 0。
ok，在root下會有-jvm選項添加上去，上面說的Unrecognized option: -jvm難道就是這里產生的。
兩個想法。一個想法是自己改了這shell代碼，去掉里面的-jvm。另外一個想法是既然這里要求 $EUID -eq 0，那別用$EUID不為0的（root用戶）用戶運行即可。果斷試試，換上普通用戶根據文檔提示做。ok，成功。好奇的再試試第一個想法，其實暫時還是不太想動源碼。但是這shell動動也沒妨，果斷去掉上面的-jvm，直接把上面的if else 結構直接去掉改為
HADOOP_OPTS="$HADOOP_OPTS -server $HADOOP_DATANODE_OPTS"，
同樣運行成功。
問題四[root@www bin]# jps
3283 NameNode
2791 SecondaryNameNode
2856 JobTracker
3348 Jps
hadoop沒有啟動datanode
解決辦法：
format之后之前的datanode會有一個ID，這個ID沒有刪除，所以會拒絕當前Namenode鏈接和分配。所以需要刪除原來的datanode中的hdfs目錄。
[root@freepp ~]# rm -rf /hadoopdata/
重啟hadoop
[root@www bin]# jps
4132 Jps
3907 NameNode
4056 DataNode
2791 SecondaryNameNode
2856 JobTracker
