Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

17/09/29 10:03:49 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:364)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
..........................................................................................................................

經常在調試spark的時候出現上面的異常情況，其原因是windows系統下環境變量的未配置問題，解決這個問題的方案：

1. 下載一個winutils.exe，http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe。
2. 配置HADOOP_HOME的環境變量，復制winutils.exe 到${HADOOP_HOME}/bin目錄下
3. 替代方案，在程序中設置System.setProperty("hadoop.home.dir", "hadoop的home目錄");