背景音樂：保留 - 郭頂
上一篇：Titanic生存預測1，主要講了如何做的特征工程。
這一篇講如何訓練模型來實現預測。
%matplotlib inline
from sklearn import svm, tree, linear_model, neighbors, naive_bayes, ensemble, discriminant_analysis, gaussian_process
from xgboost import XGBClassifier
from sklearn.preprocessing import OneHotEncoder, LabelEncoder
from sklearn import feature_selection
from sklearn import model_selection
from sklearn import metrics
import pandas as pd
import time
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
1. 讀取數據
path_data = '../../data/titanic/'
df = pd.read_csv(path_data + 'fe_data.csv')
df_data_y = df['Survived']
df_data_x = df.drop(['Survived', 'PassengerId'], 1)
df_train_x = df_data_x.iloc[:891, :] # 前891個數據是訓練集
df_train_y = df_data_y[:891]
2. 特征選擇
我選擇用GBDT來進行特征選擇，這是由決策樹本身的算法特性所決定的，每次通過計算信息增益（或其他準則）來選擇特征進行分割，在預測的同時也對特征的貢獻進行了“衡量”，因此比較容易可視化~
cv_split = model_selection.ShuffleSplit(n_splits = 10, test_size = .3, train_size = .6, random_state = 0)
gbdt_rfe = feature_selection.RFECV(ensemble.GradientBoostingClassifier(random_state=2018), step = 1, scoring = 'accuracy', cv = cv_split)
gbdt_rfe.fit(df_train_x, df_train_y)
columns_rfe = df_train_x.columns.values[gbdt_rfe.get_support()]
print('Picked columns: {}'.format(columns_rfe))
print("Optimal number of features : {}/{}".format(gbdt_rfe.n_features_, len(df_train_x.columns)))
plt.figure()
plt.xlabel("Number of features selected")
plt.ylabel("Cross validation score (nb of correct classifications)")
plt.plot(range(1, len(gbdt_rfe.grid_scores_) + 1), gbdt_rfe.grid_scores_)
plt.show()
結果顯示：
Picked columns: ['Age' 'Fare' 'Pclass' 'SibSp' 'FamilySize' 'Family_Survival' 'Sex_Code' 'Title_Master' 'Title_Mr' 'Cabin_C' 'Cabin_E' 'Cabin_X']
Optimal number of features : 12/24
大約在5個以上特征的時候，交叉驗證集的分數就已經趨于穩定了。說明在現有特征中，有貢獻的特征并不多……
最好的結果出現在12個特征的時候。但需要注意的是，比賽的比分不是由你的交叉驗證集決定，所以存在一定的偶然性，鑒于特征數量在比較長的跨度上表現接近，因此我覺得有機會的話，特征數量從5到24的每種選擇都值得一試。
我個人比較了24個特征和12個特征，表現最好的是24個全選……沒試其他的。
然后對特征進行標準化，用以訓練：
stsc = StandardScaler()
df_data_x = stsc.fit_transform(df_data_x)
print('mean:\n', stsc.mean_)
print('var:\n', stsc.var_)
df_train_x = df_data_x[:891]
df_train_y = df_data_y[:891]
df_test_x = df_data_x[891:]
df_test_output = df.iloc[891:, :][['PassengerId','Survived']]
3.模型融合
機器學習的套路是：
- 先選擇一個基礎模型，進行訓練和預測，最快建立起一個pipeline。
- 在此基礎上用交叉驗證和GridSearch對模型調參，查看模型的表現。
- 用模型融合進行多個模型的組合，用投票的方式（或其他）來預測結果。
一般來說，模型融合得到的結果會比單個模型的要好。
在這里，我跳過了步驟1和2，直接進行步驟3。
3.1 設置基本參數
vote_est = [
('ada', ensemble.AdaBoostClassifier()),
('bc', ensemble.BaggingClassifier()),
('etc', ensemble.ExtraTreesClassifier()),
('gbc', ensemble.GradientBoostingClassifier()),
('rfc', ensemble.RandomForestClassifier()),
('gpc', gaussian_process.GaussianProcessClassifier()),
('lr', linear_model.LogisticRegressionCV()),
('bnb', naive_bayes.BernoulliNB()),
('gnb', naive_bayes.GaussianNB()),
('knn', neighbors.KNeighborsClassifier()),
('svc', svm.SVC(probability=True)),
('xgb', XGBClassifier())
]
grid_n_estimator = [10, 50, 100, 300, 500]
grid_ratio = [.5, .8, 1.0]
grid_learn = [.001, .005, .01, .05, .1]
grid_max_depth = [2, 4, 6, 8, 10]
grid_criterion = ['gini', 'entropy']
grid_bool = [True, False]
grid_seed = [0]
grid_param = [
# AdaBoostClassifier
{
'n_estimators':grid_n_estimator,
'learning_rate':grid_learn,
'random_state':grid_seed
},
# BaggingClassifier
{
'n_estimators':grid_n_estimator,
'max_samples':grid_ratio,
'random_state':grid_seed
},
# ExtraTreesClassifier
{
'n_estimators':grid_n_estimator,
'criterion':grid_criterion,
'max_depth':grid_max_depth,
'random_state':grid_seed
},
# GradientBoostingClassifier
{
'learning_rate':grid_learn,
'n_estimators':grid_n_estimator,
'max_depth':grid_max_depth,
'random_state':grid_seed,
},
# RandomForestClassifier
{
'n_estimators':grid_n_estimator,
'criterion':grid_criterion,
'max_depth':grid_max_depth,
'oob_score':[True],
'random_state':grid_seed
},
# GaussianProcessClassifier
{
'max_iter_predict':grid_n_estimator,
'random_state':grid_seed
},
# LogisticRegressionCV
{
'fit_intercept':grid_bool, # default: True
'solver':['newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'],
'random_state':grid_seed
},
# BernoulliNB
{
'alpha':grid_ratio,
},
# GaussianNB
{},
# KNeighborsClassifier
{
'n_neighbors':range(6, 25),
'weights':['uniform', 'distance'],
'algorithm':['auto', 'ball_tree', 'kd_tree', 'brute']
},
# SVC
{
'C':[1, 2, 3, 4, 5],
'gamma':grid_ratio,
'decision_function_shape':['ovo', 'ovr'],
'probability':[True],
'random_state':grid_seed
},
# XGBClassifier
{
'learning_rate':grid_learn,
'max_depth':[1, 2, 4, 6, 8, 10],
'n_estimators':grid_n_estimator,
'seed':grid_seed
}
]
3.2 訓練
對于每個模型都進行調參再組合，不過有的迭代次數較多，為了節省時間我就用了RandomizedSearchCV來簡化（還沒來得及試驗全部GridSearchCV）。
start_total = time.perf_counter()
N = 0
for clf, param in zip (vote_est, grid_param):
start = time.perf_counter()
cv_split = model_selection.ShuffleSplit(n_splits = 10, test_size = .3, train_size = .6, random_state = 0)
if 'n_estimators' not in param.keys():
print(clf[1].__class__.__name__, 'GridSearchCV')
best_search = model_selection.GridSearchCV(estimator = clf[1], param_grid = param, cv = cv_split, scoring = 'accuracy')
best_search.fit(df_train_x, df_train_y)
best_param = best_search.best_params_
else:
print(clf[1].__class__.__name__, 'RandomizedSearchCV')
best_search2 = model_selection.RandomizedSearchCV(estimator = clf[1], param_distributions = param, cv = cv_split, scoring = 'accuracy')
best_search2.fit(df_train_x, df_train_y)
best_param = best_search2.best_params_
run = time.perf_counter() - start
print('The best parameter for {} is {} with a runtime of {:.2f} seconds.'.format(clf[1].__class__.__name__, best_param, run))
clf[1].set_params(**best_param)
run_total = time.perf_counter() - start_total
print('Total optimization time was {:.2f} minutes.'.format(run_total/60))
4. 預測
投票有兩種方式——軟投票和硬投票。
- 硬投票：少數服從多數。
- 軟投票：沒研究過，有文章表明，計算的是加權平均概率，預測結果是概率高的。
如果沒有先驗經驗，那么最好是兩種投票方式都算一遍，看看結果如何。
對于Titanic生存預測，我發現每次都是硬投票的結果要好。
grid_hard = ensemble.VotingClassifier(estimators = vote_est , voting = 'hard')
grid_hard_cv = model_selection.cross_validate(grid_hard, df_train_x, df_train_y, cv = cv_split, scoring = 'accuracy')
grid_hard.fit(df_train_x, df_train_y)
print("Hard Voting w/Tuned Hyperparameters Training w/bin score mean: {:.2f}". format(grid_hard_cv['train_score'].mean()*100))
print("Hard Voting w/Tuned Hyperparameters Test w/bin score mean: {:.2f}". format(grid_hard_cv['test_score'].mean()*100))
print("Hard Voting w/Tuned Hyperparameters Test w/bin score 3*std: +/- {:.2f}". format(grid_hard_cv['test_score'].std()*100*3))
print('-'*10)
grid_soft = ensemble.VotingClassifier(estimators = vote_est , voting = 'soft')
grid_soft_cv = model_selection.cross_validate(grid_soft, df_train_x, df_train_y, cv = cv_split, scoring = 'accuracy')
grid_soft.fit(df_train_x, df_train_y)
print("Soft Voting w/Tuned Hyperparameters Training w/bin score mean: {:.2f}". format(grid_soft_cv['train_score'].mean()*100))
print("Soft Voting w/Tuned Hyperparameters Test w/bin score mean: {:.2f}". format(grid_soft_cv['test_score'].mean()*100))
print("Soft Voting w/Tuned Hyperparameters Test w/bin score 3*std: +/- {:.2f}". format(grid_soft_cv['test_score'].std()*100*3))
結果為：
Hard Voting w/Tuned Hyperparameters Training w/bin score mean: 89.70
Hard Voting w/Tuned Hyperparameters Test w/bin score mean: 85.97
Hard Voting w/Tuned Hyperparameters Test w/bin score 3*std: +/- 5.95
----------
Soft Voting w/Tuned Hyperparameters Training w/bin score mean: 90.02
Soft Voting w/Tuned Hyperparameters Test w/bin score mean: 85.52
Soft Voting w/Tuned Hyperparameters Test w/bin score 3*std: +/- 6.07
硬投票得出的預測結果，在測試集上的分數較高，標準差較小，優選硬投票。
5. 提交結果：
用硬投票作為預測的方案，得到結果并提交。
df_test_output['Survived'] = grid_hard.predict(df_test_x)
df_test_output.to_csv('../../data/titanic/hardvote.csv', index = False)
在官網上提交結果，給出的分數是0.81339。
后記
Titanic這個項目很值得一試，在實踐的過程中，我參考了一些參賽者在kaggle上分享的kernel，收益良多。
但作為入門項目，重在參與，后面有空了再做一遍，看是否能有提高。
接下來，我會嘗試參加貓狗大戰。
也就是編寫一個算法來分類圖像是否包含狗或貓。
這對人類，狗和貓來說很容易，但用算法如何實現呢？拭目以待。
