已經有了性能超好的word2vec，為什么還要搞一個性能超差的word2vec？可以幫助了解原理啊，哈哈哈
裝好tensorflow之后，想學一下word2vec，結果，源碼下下來，一點運行，開始跑了，然而并不知道原理。
看下源碼吧，數據生成什么的都還好，輸入輸出的話，有點暈，loss函數的話，直接調了個現成的函數，我去，這根本學不到原理嘛！
如果你也是初學神經網絡的菜鳥，也有上述這種感覺，那就看下這篇文章吧，是我的一點小心得。
如果你是大神，愿意花時間對文章進行批評指正，那是很歡迎的。如果想從文章中獲取新的東西，估計就得失望了。
首先是對word2vec的原理（只針對：https://github.com/tensorflow/tensorflow/blob/r1.2/tensorflow/examples/tutorials/word2vec/word2vec_basic.py的實現）的一個直覺層面的理解：
word2vec第一步隨機生成每個單詞的embedding，存到一個變量池里。如果你的詞匯表長度是vocabulary_size（也就是你一共就研究這么幾個單詞，別的單詞都歸到unknown）， 而對每個單詞，你想用embedding_size維的向量來表示的話，那這個變量池自然是一個有vocabulary_size行，embedding_size列的矩陣啦。
然后就是給每個單詞打標簽，如果是Skip-gram Model的話，單詞w的標簽，就是出現在它前面的（緊挨著）單詞（的詞匯表id），或者是出現在他后面的單詞。對，一個單詞會有兩個標簽（參數不一樣的話，label還會更多）。也就是一張圖片既是貓，又是狗。似乎有點矛盾，但計算機會用數學的方法消化掉這種矛盾。
現在輸入，輸出，就明了了，輸入是一個單詞對應的embedding變量池中相應的一行，輸出是它對應的label（label不止一個的話，就多次輸入輸出）。
如果你的embedding_size是128的話，那你的模型按理說應該是128進1出的一個模型。
但是，跟識別數字一樣，對于輸出，你不能用0表示0，用1表示1，
你得用【1，0,0，...】這個onehot的向量表示0,1的話類似
所以，你的輸出不是1維的，而是你要分幾類就是幾維的。數字識別是分10類，所以y是10維的，word2vec要分的類和你的詞匯表數量一樣多，所以y是vocabulary_size維（以5000為例）的（onehot形式）。
恩，你就是要訓練一個128進5000出的分類器。在不斷訓練的過程中，通過梯度下降，不僅更新分類器的參數，還要更新embedding變量池中的每一個數。訓練結束后，我們反而并不太關心這個分類器，而是更希望得到性能更好的embedding變量池。
所以代碼的話：
'''
Created on 2017-6-19
@author: Administrator
'''
import collections
import math
import os
import numpy as np
import tensorflow as tf
import random
path="E:\\NLPdata\\"
filename = "text8"
with open(path+filename) as f:
data=tf.compat.as_str(f.read()).split()
words=data
print("data size:"+str(len(words)))
vocabulary_size=5000
count=[['UNK',-1]]
count.extend(collections.Counter(words).most_common(vocabulary_size - 1))
print(count[:5])
dictionary = dict()
for word,_ in count:
dictionary[word]=len(dictionary)
data=list()
unk_count=0
for word in words:
if word in dictionary:
index=dictionary[word]
else:
index=0
unk_count+=1
data.append(index)
count[0][1]=unk_count
reverse_dictionary = dict(zip(dictionary.values(), dictionary.keys()))
print('Most common words (+UNK)', count[:5])
print('Sample data', data[:10], [reverse_dictionary[i] for i in data[:10]])
data_index = 0
def one_hot(index,ds):
oh=np.zeros(ds)
oh[index]=1
return oh
def generate_batch(batch_size, num_skips, skip_window):
global data_index
assert batch_size % num_skips == 0
assert num_skips <= 2 * skip_window
batch = np.ndarray(shape=(batch_size), dtype=np.int32)
labels = np.ndarray(shape=(batch_size, vocabulary_size), dtype=np.int32)
span = 2 * skip_window + 1 # [ skip_window target skip_window ]
buffer = collections.deque(maxlen=span)
for _ in range(span):
buffer.append(data[data_index])
data_index = (data_index + 1) % len(data)
for i in range(batch_size // num_skips):
target = skip_window # target label at the center of the buffer
targets_to_avoid = [ skip_window ]
for j in range(num_skips):
while target in targets_to_avoid:
target = random.randint(0, span - 1)
targets_to_avoid.append(target)
batch[i * num_skips + j] = buffer[skip_window]
labels[i * num_skips + j] = one_hot(buffer[target],vocabulary_size)
buffer.append(data[data_index])
data_index = (data_index + 1) % len(data)
return batch, labels
print('data:', [reverse_dictionary[di] for di in data[:8]])
for num_skips, skip_window in [(2, 1), (4, 2)]:
data_index = 0
batch, labels = generate_batch(batch_size=8, num_skips=num_skips, skip_window=skip_window)
print('\nwith num_skips = %d and skip_window = %d:' % (num_skips, skip_window))
print(' batch:', [reverse_dictionary[bi] for bi in batch])
print(' labels:',labels)
# batch_size = 128
batch_size = 128
embedding_size = 128 # Dimension of the embedding vector.
skip_window = 1 # How many words to consider left and right.
num_skips = 2 # How many times to reuse an input to generate a label.
# We pick a random validation set to sample nearest neighbors. here we limit the
# validation samples to the words that have a low numeric ID, which by
# construction are also the most frequent.
valid_size = 16 # Random set of words to evaluate similarity on.
valid_window = 100 # Only pick dev samples in the head of the distribution.
valid_examples = np.array(random.sample(range(valid_window), valid_size))
# num_sampled = 64 # Number of negative examples to sample.
graph = tf.Graph()
with graph.as_default(), tf.device('/cpu:0'):
train_dataset = tf.placeholder(tf.int32, shape=[batch_size])
train_labels = tf.placeholder(tf.float32, shape=[batch_size, vocabulary_size])
valid_dataset = tf.constant(valid_examples, dtype=tf.int32)
embeddings = tf.Variable(tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0))
#softmax_weights = tf.Variable(tf.truncated_normal([vocabulary_size, embedding_size], stddev=1.0 / math.sqrt(embedding_size)))
#softmax_biases = tf.Variable(tf.zeros([vocabulary_size]))
W = tf.Variable(tf.zeros([embedding_size, vocabulary_size]))
b = tf.Variable(tf.zeros([vocabulary_size]))
embed = tf.nn.embedding_lookup(embeddings, train_dataset)
y=tf.nn.softmax(tf.matmul(embed,W)+b)
loss = tf.reduce_mean(-tf.reduce_sum(train_labels * tf.log(y), reduction_indices=[1]))
optimizer = tf.train.AdagradOptimizer(1.0).minimize(loss)
norm = tf.sqrt(tf.reduce_sum(tf.square(embeddings), 1, keep_dims=True))
normalized_embeddings = embeddings / norm
valid_embeddings = tf.nn.embedding_lookup(normalized_embeddings, valid_dataset)
similarity = tf.matmul(valid_embeddings, tf.transpose(normalized_embeddings))
num_steps = 100001
with tf.Session(graph=graph) as session:
tf.global_variables_initializer().run()
print('Initialized')
average_loss = 0
for step in range(num_steps):
batch_data, batch_labels = generate_batch(
batch_size, num_skips, skip_window)
feed_dict = {train_dataset : batch_data, train_labels : batch_labels}
_, l = session.run([optimizer, loss], feed_dict=feed_dict)
average_loss += l
if step % 2000 == 0:
if step > 0:
average_loss = average_loss / 2000
# The average loss is an estimate of the loss over the last 2000 batches.
print('Average loss at step %d: %f' % (step, average_loss))
average_loss = 0
# note that this is expensive (~20% slowdown if computed every 500 steps)
if step % 10000 == 0:
sim = similarity.eval()
for i in range(valid_size):
valid_word = reverse_dictionary[valid_examples[i]]
top_k = 8 # number of nearest neighbors
nearest = (-sim[i, :]).argsort()[1:top_k+1]
log = 'Nearest to %s:' % valid_word
for k in range(top_k):
close_word = reverse_dictionary[nearest[k]]
log = '%s %s,' % (log, close_word)
print(log)
final_embeddings = normalized_embeddings.eval()
其實就是基于原來的代碼改的，主要改動包括，
把詞匯表降到5000，降低y的維度
label直接變成one-hot
把訓練模型直接改成softmax分類器，更容易看懂，
loss函數就是之前的交叉熵，是不是很親切，
這段代碼訓練慢，效果一般，但還是有一定效果。不過它說明了word2vec就是個分類器的本質，對理解原理有幫助。
事實上，這里實現的就是官方文檔里，說的那個因為效果差被對比的方法（下圖）。哈哈，就是這樣。不過我們實現了這個的基礎上，理解了原理，再去親手實現NCE（而不是直接調函數），豈不是對NCE掌握的更好了？
希望對大家有用
