1. 如何“非正則”地遍歷一個(gè)目錄下某些特定的文件？
在spark中newHadoopAPI()或者FSShell其實(shí)都是支持正則話輸入路徑的，不要怕輸入的路徑太長(zhǎng)，一定要正則化輸入路徑；
- https://stackoverflow.com/questions/31782763/how-to-use-regex-to-include-exclude-some-input-files-in-sc-textfile
- https://stackoverflow.com/questions/24224392/why-does-spark-throw-notserializableexception-org-apache-hadoop-io-nullwritable
2. 在讀取->repartition->重新再寫入的過程中，遇到序列化失敗該如何處理？
盡量使用kyro序列化，兼容性還是比較好；
同時(shí)注意null值的處理；
- http://apache-spark-developers-list.1001551.n3.nabble.com/NullWritable-not-serializable-td8412.html
- https://stackoverflow.com/questions/22592811/task-not-serializable-java-io-notserializableexception-when-calling-function-ou/22594142#22594142
- https://issues.apache.org/jira/browse/SPARK-2421
- https://spark.apache.org/docs/latest/tuning.html
