環境
- Hadoop 2.6.0
- Java 1.8.0_151
- Ubuntu 16.04
解決方案
開始在配置Hadoop時,由于是按照官網配置的非常簡潔版的偽分布式版本,缺少了某些選項,才會造成這個問題。
開始時,我的mapred-site.xml的配置如下:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
這個就是跟著官網上的tutorial一步步走到最后的結果。
由于我們想要查看歷史信息,那么肯定就需要有一個地方來存儲這些歷史信息啊,在之前的Hadoop版本中,使用的是JobHistoryServer,但是如果使用的mapreduce調度框架是YARN的話,就不會使用這一個了,而是ApplicationHistoryServer。
所以,我們還需要額外配置一下。
將上面的配置文件修改成如這樣:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>localhost:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>localhost:19888</value>
</property>
</configuration>
然后,還需要啟動它,通過下面的命令:
sbin/yarn-daemon.sh start historyserver
好啦,這樣就大功告成啦。