Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/09/29 10:03:49 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:364)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
..........................................................................................................................
經常在調試spark的時候出現上面的異常情況,其原因是windows系統下環境變量的未配置問題,解決這個問題的方案:
- 下載一個winutils.exe,http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe。
- 配置HADOOP_HOME的環境變量,復制winutils.exe 到${HADOOP_HOME}/bin目錄下
- 替代方案,在程序中設置System.setProperty("hadoop.home.dir", "hadoop的home目錄");