項目簡介
- 源碼地址 https://github.com/XuefengHuang/RecommendationSystem
- 基于Spark, Python Flask, 和 Book-Crossing Dataset 的在線圖書推薦系統。
- 該圖書推薦系統參考https://github.com/jadianes/spark-movie-lens。
- 修改數據處理部分，使其支持Book-Crossing Dataset。
- 適合初學者學習如何搭建一個推薦系統，本文底下附有其他數據，可供參考學習。
基于模型的協同過濾應用---圖書推薦
本文實現對用戶推薦圖書的簡單應用。
- 推薦算法：
在我們的在線圖書推薦系統中，我們借用Spark的ALS算法的訓練和預測函數，每次收到新的數據后，將其更新到訓練數據集中，然后更新ALS訓練得到的模型。
假設我們有一組用戶，他們表現出了對一組圖書的喜好。用戶對一本圖書的喜好程度越高，就會給其更高的評分，范圍是從1到5。我們來通過一個矩陣來展示它，行代表用戶，列代表圖書。用戶對圖書的評分。所有的評分范圍從1到5，5代表喜歡程度最高。第一個用戶（行1）對第一個圖書（列1）的評分是4。空的單元格代表用戶未給圖書評價。
用戶圖書評分表.png
矩陣因子分解（如奇異值分解，奇異值分解+ +）將項和用戶都轉化成了相同的潛在空間，它所代表了用戶和項之間的潛相互作用。矩陣分解背后的原理是潛在特征代表了用戶如何給項進行評分。給定用戶和項的潛在描述，我們可以預測用戶將會給還未評價的項多少評分。
矩陣因子分解.png
- 數據描述：
評分數據文件:
- 數據描述：
"User-ID";"ISBN";"Book-Rating"
"276725";"034545104X";"0"
"276726";"0155061224";"5"
"276727";"0446520802";"0"
"276729";"052165615X";"3"
"276729";"0521795028";"6"
"276733";"2080674722";"0"
"276736";"3257224281";"8"
圖書數據文件:
"ISBN";"Book-Title";"Book-Author";"Year-Of-Publication";"Publisher";"Image-URL-S";"Image-URL-M";"Image-URL-L"
"0195153448";"Classical Mythology";"Mark P. O. Morford";"2002";"Oxford University Press";"http://images.amazon.com/images/P/0195153448.01.THUMBZZZ.jpg";"http://images.amazon.com/images/P/0195153448.01.MZZZZZZZ.jpg";"http://images.amazon.com/images/P/0195153448.01.LZZZZZZZ.jpg"
"0002005018";"Clara Callan";"Richard Bruce Wright";"2001";"HarperFlamingo Canada";"http://images.amazon.com/images/P/0002005018.01.THUMBZZZ.jpg";"http://images.amazon.com/images/P/0002005018.01.MZZZZZZZ.jpg";"http://images.amazon.com/images/P/0002005018.01.LZZZZZZZ.jpg"
"0060973129";"Decision in Normandy";"Carlo D'Este";"1991";"HarperPerennial";"http://images.amazon.com/images/P/0060973129.01.THUMBZZZ.jpg";"http://images.amazon.com/images/P/0060973129.01.MZZZZZZZ.jpg";"http://images.amazon.com/images/P/0060973129.01.LZZZZZZZ.jpg"
"0374157065";"Flu: The Story of the Great Influenza Pandemic of 1918 and the Search for the Virus That Caused It";"Gina Bari Kolata";"1999";"Farrar Straus Giroux";"http://images.amazon.com/images/P/0374157065.01.THUMBZZZ.jpg";"http://images.amazon.com/images/P/0374157065.01.MZZZZZZZ.jpg";"http://images.amazon.com/images/P/0374157065.01.LZZZZZZZ.jpg"
"0393045218";"The Mummies of Urumchi";"E. J. W. Barber";"1999";"W. W. Norton & Company";"http://images.amazon.com/images/P/0393045218.01.THUMBZZZ.jpg";"http://images.amazon.com/images/P/0393045218.01.MZZZZZZZ.jpg";"http://images.amazon.com/images/P/0393045218.01.LZZZZZZZ.jpg"
- 數據處理細節：
由于該數據中ISBN為string格式，spark的ALS默認product id為int格式，因此對該ISBN號進行計算hash處理并取前8位防止整數越界。詳細代碼如下：
dataset_path = os.path.join('datasets', 'BX-CSV-Dump')
sc = SparkContext("local[*]", "Test")
ratings_file_path = os.path.join(dataset_path, 'BX-Book-Ratings.csv')
ratings_raw_RDD = sc.textFile(ratings_file_path)
ratings_raw_data_header = ratings_raw_RDD.take(1)[0]
ratings_RDD = ratings_raw_RDD.filter(lambda line: line!=ratings_raw_data_header)\
.map(lambda line: line.split(";")).map(lambda tokens: (int(tokens[0][1:-1]), abs(hash(tokens[1][1:-1])) % (10 ** 8),float(tokens[2][1:-1]))).cache()
books_file_path = os.path.join(dataset_path, 'BX-Books.csv')
books_raw_RDD = sc.textFile(books_file_path)
books_raw_data_header = books_raw_RDD.take(1)[0]
books_RDD = books_raw_RDD.filter(lambda line: line!=books_raw_data_header)\
.map(lambda line: line.split(";"))\
.map(lambda tokens: (abs(hash(tokens[0][1:-1])) % (10 ** 8), tokens[1][1:-1], tokens[2][1:-1], tokens[3][1:-1], tokens[4][1:-1], tokens[5][1:-1])).cache()
books_titles_RDD = books_RDD.map(lambda x: (int(x[0]), x[1], x[2], x[3], x[4], x[5])).cache()
- 選擇模型參數：
from pyspark.mllib.recommendation import ALS
import math
seed = 5L
iterations = 10
regularization_parameter = 0.1
ranks = [4, 8, 12]
errors = [0, 0, 0]
err = 0
tolerance = 0.02
min_error = float('inf')
best_rank = -1
best_iteration = -1
for rank in ranks:
model = ALS.train(training_RDD, rank, seed=seed, iterations=iterations,
lambda_=regularization_parameter)
predictions = model.predictAll(validation_for_predict_RDD).map(lambda r: ((r[0], r[1]), r[2]))
rates_and_preds = validation_RDD.map(lambda r: ((int(r[0]), int(r[1])), float(r[2]))).join(predictions)
error = math.sqrt(rates_and_preds.map(lambda r: (r[1][0] - r[1][1])**2).mean())
errors[err] = error
err += 1
print 'For rank %s the RMSE is %s' % (rank, error)
if error < min_error:
min_error = error
best_rank = rank
print 'The best model was trained with rank %s' % best_rank
- 模型保存
from pyspark.mllib.recommendation import MatrixFactorizationModel
model_path = os.path.join('..', 'models', 'book_als')
# Save and load model
model.save(sc, model_path)
same_model = MatrixFactorizationModel.load(sc, model_path)
- 運行說明：
virtualenv book
pip install -r requirements.txt
python server.py
- API:
GET: /<int:user_id>/ratings/top/<int:count> 獲取用戶圖書推薦top N信息
GET: /<int:user_id>/ratings/<string:book_id> 獲取該用戶對某個圖書的評價信息
POST: /<int:user_id>/ratings 新增圖書評價信息
- 接口調用示例：
GET: /276729/ratings/top/3 獲取用戶ID為276729的圖書推薦top3信息
返回信息：
[
{
"Count": 30,
"Rating": 8.781754720405482,
"Author": "MARJANE SATRAPI",
"URL": "http://images.amazon.com/images/P/0375422307.01.THUMBZZZ.jpg",
"Publisher": "Pantheon",
"Title": "Persepolis : The Story of a Childhood (Alex Awards (Awards))",
"Year": "2003"
},
{
"Count": 31,
"Rating": 7.093566643463471,
"Author": "Stephen King",
"URL": "http://images.amazon.com/images/P/067081458X.01.THUMBZZZ.jpg",
"Publisher": "Viking Books",
"Title": "The Eyes of the Dragon",
"Year": "1987"
},
{
"Count": 25,
"Rating": 7.069147186199548,
"Author": "Jean Sasson",
"URL": "http://images.amazon.com/images/P/0967673747.01.THUMBZZZ.jpg",
"Publisher": "Windsor-Brooke Books",
"Title": "Princess: A True Story of Life Behind the Veil in Saudi Arabia",
"Year": "2001"
}
]
GET: /276729/ratings/0446520802 獲取用戶276729對圖書(ISBN:0446520802)的評價信息
返回信息：
[
{
"Count": 116,
"Rating": 1.4087434932956826,
"Author": "Nicholas Sparks",
"URL": "http://images.amazon.com/images/P/0446520802.01.THUMBZZZ.jpg",
"Publisher": "Warner Books",
"Title": "The Notebook",
"Year": "1996"
}
]
其他數據集推薦（參考https://gist.github.com/entaroadun/1653794）
以下數據可以提供給初學者學習如何訓練推薦算法模型
電影數據:
- MovieLens - Movie Recommendation Data Sets http://www.grouplens.org/node/73
- Yahoo! - Movie, Music, and Images Ratings Data Sets http://webscope.sandbox.yahoo.com/catalog.php?datatype=r
- Cornell University - Movie-review data for use in sentiment-analysis experiments http://www.cs.cornell.edu/people/pabo/movie-review-data/
音樂數據:
- Last.fm - Music Recommendation Data Sets http://www.dtic.upf.edu/~ocelma/MusicRecommendationDataset/index.html
- Yahoo! - Movie, Music, and Images Ratings Data Sets http://webscope.sandbox.yahoo.com/catalog.php?datatype=r
- Audioscrobbler - Music Recommendation Data Sets http://www-etud.iro.umontreal.ca/~bergstrj/audioscrobbler_data.html
- Amazon - Audio CD recommendations http://131.193.40.52/data/
圖書數據:
- Institut für Informatik, Universit?t Freiburg - Book Ratings Data Sets http://www.informatik.uni-freiburg.de/~cziegler/BX/
美食數據:
- Chicago Entree - Food Ratings Data Sets http://archive.ics.uci.edu/ml/datasets/Entree+Chicago+Recommendation+Data
商品數據:
- Amazon - Product Recommendation Data Sets http://131.193.40.52/data/
健康數據:
- Nursing Home - Provider Ratings Data Set http://data.medicare.gov/dataset/Nursing-Home-Compare-Provider-Ratings/mufm-vy8d
- Hospital Ratings - Survey of Patients Hospital Experiences http://data.medicare.gov/dataset/Survey-of-Patients-Hospital-Experiences-HCAHPS-/rj76-22dk
相親數據:
- www.libimseti.cz - Dating website recommendation (collaborative filtering) http://www.occamslab.com/petricek/data/
學術文章推薦:
- National University of Singapore - Scholarly Paper Recommendation http://www.comp.nus.edu.sg/~sugiyama/SchPaperRecData.html
