Spark RDD數據關聯過濾案例 - 簡書
http://www.lxweimin.com/p/f77cb1c93793

前言
最近在專注Spark開發，記錄下自己的工作和學習路程，希望能跟大家互相交流成長QQ:86608625 微信:guofei1990123
背景
detail.txt為用戶注冊信息全表，filter.txt為需要過濾掉的用戶的手機號碼
要求detail.txt關聯filter.txt并去除detail.txt中包含filter.txt中的用戶
原始數據
detail.txt數據張三 13007080388 man 山西 NULL 5 2015-11-05 21:22:41李四 13056677799 woman 四川 地推注冊邀請 5 2015-10-11 11:32:19王五 13084470421 man 四川 地推注冊邀請 6 2015-08-08 08:11:14filter.txt數據1300708038813056677799

實現思路
兩份數據通過手機號主鍵關聯并過濾掉不符合規則數據
具體如下：
1.分別加載兩份數據
2.解析數據并拼接成key為手機號的元組
3.用戶信息leftOuterJoin過濾表
4.過濾掉不符合規則數據并輸出
代碼實現
packagesparkimportorg.apache.spark.{SparkConf, SparkContext}/*** Created by郭飛on 2016/5/30.*/objectJoinFilterOne {defmain(args: Array[String]) {//創建SparkConf()并設置App名稱及Master地址valconf =newSparkConf().setAppName("JoinFilter").setMaster("local")//創建SparkContext，該對象是提交spark App的入口valsc =newSparkContext(conf)//加載源數據valsourceAll = sc.textFile("C://work//data//test//detail.txt")valsourceFilter = sc.textFile("C://work//data//test//filter.txt")//轉換RDD//(13007080388,(張三,13007080388,man,山西,NULL,5,2015-11-05 21:22:41)),// (13056677799,(李四,13056677799,woman,四川,地推注冊邀請,5,2015-10-11 11:32:19)),// (13084470421,(王五,13084470421,man,四川,地推注冊邀請,6,2015-08-08 08:11:14))valrddAll = sourceAll.map(x=>{valline = x.split("\t")valname = line(0)valphone = line(1)valsex = line(2)valaddr = line(3)valdataType = line(4)valdataCode = line(5)valdataDate = line(6)(phone,(name,phone,sex,addr,dataType,dataCode,dataDate))})//(13007080388,delete), (13056677799,delete)valrddFilter = sourceFilter.map(x=>{valline = x.split("\t")valphone = line(0)(phone,"delete")})//(13056677799,張三,13056677799,woman,四川,地推注冊邀請,5,2015-10-11 11:32:19,Some(delete)),// (13084470421,李四,13084470421,man,四川,地推注冊邀請,6,2015-08-08 08:11:14,None),// (13007080388,王五,13007080388,man,山西,NULL,5,2015-11-05 21:22:41,Some(delete))rddAll.leftOuterJoin(rddFilter).map(x=>{(x._1,x._2._1._1,x._2._1._2,x._2._1._3,x._2._1._4,x._2._1._5,x._2._1._6,x._2._1._7,x._2.2)}).filter(._9==None).saveAsTextFile("C://work//data//result.txt")//println(result.toBuffer)//過濾后//(13084470421,王宇,13084470421,man,四川,地推注冊邀請,6,2015-08-08 08:11:14,None)//停止sc，結束該任務sc.stop()}}

文／MichaelFly（簡書作者）原文鏈接：http://www.lxweimin.com/p/f77cb1c93793著作權歸作者所有，轉載請聯系作者獲得授權，并標注“簡書作者”。