網上下一份倚天屠龍記的小說txt

image.png

下述代碼分詞后的txt文件

image.png

代碼解釋

1. 用gensim做word2vec，用jieba做中文分詞

import gensim
import jieba
import warnings
warnings.filterwarnings('ignore')

2. 為jieba分詞提供一些名詞類的分詞參考

name=['峨眉派','倚天劍','屠龍刀','金毛獅王','青翼蝠王',\
      '謝遜','楊逍','冷謙','趙敏','乾坤大挪移','明教',\
      '郭襄','昆侖派','六大派','少林派','九陽真經','紫衫龍王','韋一笑']
for col in name:
    jieba.suggest_freq(col,True)

3. 刪除特殊字符或自己向指定的一些字符

#分詞
fin=open(filename_novel,'r')
fou=open(filename_segment,'w')
for line in fin.readlines():
    newline=jieba.cut(line,cut_all=False)
    str_out=' '.join(newline)\
        .replace('，','').replace('。','')\
        .replace(',','').replace('.','')\
        .replace('?','').replace('？','')\
        .replace('!','').replace('！','')\
        .replace('（','').replace('(','')\
        .replace('）','').replace(')','')\
        .replace('“','').replace('”','').replace('"','')\
        .replace('’','').replace('‘','').replace('\'','')\
        .replace('、','').replace('：','').replace(':','')\
        .replace('[','').replace(']','')\
        .replace('{','').replace('}','')\
        .replace('<','').replace('>','')\
        .replace('《','').replace('》','')\
        .replace(';','').replace('；','')\
        .replace('-','').replace('_','')\
        .replace('·','').replace('+','')\
        .replace('…','')\
        .replace('〗','').replace('〖','')
    fou.write(str_out)
fou.close()

4、訓練

sentences=gensim.models.word2vec.LineSentence(filename_segment)
model=gensim.models.word2vec.Word2Vec(sentences,size=20,window=5,min_count=10,workers=4)
model.save(filename_model)

5、查看相似度

def getSimilarity(s):
    print('similarity by word of :'+s)
    for k in model.similar_by_vector(s,5):
        print(k[0],k[1])
getSimilarity('張無忌')

image.png

image.png

image.png

image.png