實(shí)現(xiàn)源自 neural networks and deep learning 第二章，詳情請(qǐng)參考本書。

實(shí)現(xiàn)一個(gè)基于SGD學(xué)習(xí)算法的神經(jīng)網(wǎng)絡(luò)，使用BP算法計(jì)算梯度。

Network類定義

class Network():

初始化方法

    def __init__(self, sizes):
        """The list ``sizes`` contains the number of neurons in the
        respective layers of the network.  For example, if the list
        was [2, 3, 1] then it would be a three-layer network, with the
        first layer containing 2 neurons, the second layer 3 neurons,
        and the third layer 1 neuron.  The biases and weights for the
        network are initialized randomly, using a Gaussian
        distribution with mean 0, and variance 1.  Note that the first
        layer is assumed to be an input layer, and by convention we
        won't set any biases for those neurons, since biases are only
        ever used in computing the outputs from later layers."""

        self.num_layers = len(sizes)
        self.sizes = sizes
        self.biases = [np.random.randn(y, 1) for y in sizes[1:]]
        self.weights = [np.random.randn(y, x) 
                        for x, y in zip(sizes[:-1], sizes[1:])]

我們可以看到相應(yīng)的參數(shù)，sizes列表包含了網(wǎng)絡(luò)中相應(yīng)層的神經(jīng)元個(gè)數(shù)。例如，如果列表是[2,3,1]，那么這個(gè)網(wǎng)絡(luò)就是三層的神經(jīng)網(wǎng)絡(luò)，第一層有2個(gè)節(jié)點(diǎn)，第二層3個(gè)，最后一層1個(gè)。biases 和 weights 使用高斯分布mean ＝ 0, variance ＝ 1 隨機(jī)初始化。注意首層一般是作為輸入層，該層不包含 biases。
這里使用了 numpy 庫(kù)的 random 模塊進(jìn)行高斯分布的采樣。

定義 feedforward 方法：

    def feedforward(self, a):
        """Return the output of the network if ``a`` is input."""
        for b, w in zip(self.biases, self.weights):
            a = sigmoid_vec(np.dot(w, a)+b)
        return a

向量a作為輸入時(shí)的網(wǎng)絡(luò)輸出，其結(jié)果是一個(gè)向量，sigmoid_vec 作用于向量中的每個(gè)元素。

定義 SGD 方法：

    def SGD(self, training_data, epochs, mini_batch_size, eta,
            test_data=None):
        """Train the neural network using mini-batch stochastic
        gradient descent.  The ``training_data`` is a list of tuples
        ``(x, y)`` representing the training inputs and the desired
        outputs.  The other non-optional parameters are
        self-explanatory.  If ``test_data`` is provided then the
        network will be evaluated against the test data after each
        epoch, and partial progress printed out.  This is useful for
        tracking progress, but slows things down substantially."""

        if test_data: n_test = len(test_data)
        n = len(training_data)
        for j in xrange(epochs):
            random.shuffle(training_data)
            mini_batches = [
                training_data[k:k+mini_batch_size]
                for k in xrange(0, n, mini_batch_size)]
            for mini_batch in mini_batches:
                self.update_mini_batch(mini_batch, eta)
            if test_data:
                print "Epoch {0}: {1} / {2}".format(
                    j, self.evaluate(test_data), n_test)
            else:
                print "Epoch {0} complete".format(j)

使用 mini-batch SGD 訓(xùn)練神經(jīng)網(wǎng)絡(luò)。training_data 訓(xùn)練樣本的列表，包含訓(xùn)練輸入和目標(biāo)輸出。test_data 如果指定，神經(jīng)網(wǎng)絡(luò)會(huì)在每個(gè) epoch 后在測(cè)試集上進(jìn)行評(píng)估，部分過(guò)程會(huì)打印出來(lái)。這對(duì)于追蹤進(jìn)度很有用，但在一定程度上會(huì)降低運(yùn)行速度。
在每個(gè) epoch，會(huì)對(duì)訓(xùn)練數(shù)據(jù)集進(jìn)行洗牌，然后在叢中選擇訓(xùn)練子集。
細(xì)節(jié)：以mini_batch_size為大小切割整個(gè)數(shù)據(jù)集。
遍歷該次劃分完備的數(shù)據(jù)集，使用 update_mini_batch 方法進(jìn)行參數(shù)調(diào)整。
如果是測(cè)試數(shù)據(jù)則打印相應(yīng)的 epoch，驗(yàn)證結(jié)果和測(cè)試用例個(gè)數(shù)。

定義 update_mini_batch

    def update_mini_batch(self, mini_batch, eta):
        """Update the network's weights and biases by applying
        gradient descent using backpropagation to a single mini batch.
        The ``mini_batch`` is a list of tuples ``(x, y)``, and ``eta``
        is the learning rate."""
        nabla_b = [np.zeros(b.shape) for b in self.biases]
        nabla_w = [np.zeros(w.shape) for w in self.weights]
        for x, y in mini_batch:
            delta_nabla_b, delta_nabla_w = self.backprop(x, y)
            nabla_b = [nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]
            nabla_w = [nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]
        self.weights = [w-(eta/len(mini_batch))*nw 
                        for w, nw in zip(self.weights, nabla_w)]
        self.biases = [b-(eta/len(mini_batch))*nb 
                       for b, nb in zip(self.biases, nabla_b)]

在一次 mini_batch 中使用基于BP的GD來(lái)更新權(quán)重和偏置。mini_batch是一個(gè)tuple的list，[(x, y)]。而eta 則是學(xué)習(xí)率。

Paste_Image.png

定義 backprop 方法：

    def backprop(self, x, y):
        """Return a tuple ``(nabla_b, nabla_w)`` representing the
        gradient for the cost function C_x.  ``nabla_b`` and
        ``nabla_w`` are layer-by-layer lists of numpy arrays, similar
        to ``self.biases`` and ``self.weights``."""
        nabla_b = [np.zeros(b.shape) for b in self.biases]
        nabla_w = [np.zeros(w.shape) for w in self.weights]
        # feedforward
        activation = x
        activations = [x] # list to store all the activations, layer by layer
        zs = [] # list to store all the z vectors, layer by layer
        for b, w in zip(self.biases, self.weights):
            z = np.dot(w, activation)+b
            zs.append(z)
            activation = sigmoid_vec(z)
            activations.append(activation)
        # backward pass
        delta = self.cost_derivative(activations[-1], y) * \
            sigmoid_prime_vec(zs[-1])
        nabla_b[-1] = delta
        nabla_w[-1] = np.dot(delta, activations[-2].transpose())
        # Note that the variable l in the loop below is used a little
        # differently to the notation in Chapter 2 of the book.  Here,
        # l = 1 means the last layer of neurons, l = 2 is the
        # second-last layer, and so on.  It's a renumbering of the
        # scheme in the book, used here to take advantage of the fact
        # that Python can use negative indices in lists.
        for l in xrange(2, self.num_layers):
            z = zs[-l]
            spv = sigmoid_prime_vec(z)
            delta = np.dot(self.weights[-l+1].transpose(), delta) * spv
            nabla_b[-l] = delta
            nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())
        return (nabla_b, nabla_w)

算法流程

BP 公式

BP 算法

返回tuple (nabla_b, nabla_w)，包含 對(duì)于代價(jià)函數(shù) C_x 的梯度。nabla_b 和 nabla_w 是層-層numpy 數(shù)組的列表，類似于 self.biases 和 self.weights。
首先進(jìn)行 feedforward過(guò)程，activations 保存所有層-層的 activations，zs 保存所有的層-層 z 向量。
遍歷所有的 layer，計(jì)算出 z 和 activation 最終保存到列表 zs 和 activations 中。
然后backprop，首先計(jì)算出最終輸出的 delta，以及sigmoid函數(shù)的導(dǎo)數(shù)。nabla_b[-1] 和 nabla_w[-1] 單獨(dú)算出。
然后對(duì)前面的層進(jìn)行遍歷，反向傳播。

測(cè)試評(píng)價(jià)函數(shù)

    def evaluate(self, test_data):
        """Return the number of test inputs for which the neural
        network outputs the correct result. Note that the neural
        network's output is assumed to be the index of whichever
        neuron in the final layer has the highest activation."""
        test_results = [(np.argmax(self.feedforward(x)), y) 
                        for (x, y) in test_data]
        return sum(int(x == y) for (x, y) in test_results)

代價(jià)函數(shù)的導(dǎo)數(shù)

    def cost_derivative(self, output_activations, y):
        """Return the vector of partial derivatives \partial C_x /
        \partial a for the output activations."""
        return (output_activations-y) 

Appendix

def sigmoid(z):
    """The sigmoid function."""
    return 1.0/(1.0+np.exp(-z))

sigmoid_vec = np.vectorize(sigmoid)

def sigmoid_prime(z):
    """Derivative of the sigmoid function."""
    return sigmoid(z)*(1-sigmoid(z))

sigmoid_prime_vec = np.vectorize(sigmoid_prime)

這里有 numpy 的函數(shù) vectorize 的使用

為何說(shuō) BP 是一個(gè)快速的算法

為了回答這個(gè)問(wèn)題，首先考慮另一個(gè)計(jì)算梯度的方法。就當(dāng)我們回到上世界50、60年代的神經(jīng)網(wǎng)絡(luò)研究。假設(shè)你是世界上首個(gè)考慮使用梯度下降方法學(xué)習(xí)的那位！為了讓自己的想法可行，就必須找出計(jì)算代價(jià)函數(shù)梯度的方法。想想自己學(xué)到的微積分，決定試試看鏈?zhǔn)椒▌t來(lái)計(jì)算梯度。但玩了一會(huì)后，就發(fā)現(xiàn)代數(shù)式看起來(lái)非常復(fù)雜，然后就退縮了。所以就試著找另外的方式。你決定僅僅把代價(jià)看做權(quán)重 C 的函數(shù)。你給這些權(quán)重 w_1, w_2, ... 進(jìn)行編號(hào)，期望計(jì)算關(guān)于某個(gè)權(quán)值 w_j 關(guān)于 C 的導(dǎo)數(shù)。而一種近似的方法就是下面這種：

Paste_Image.png

*This should be plausible, but it requires some analysis to make a careful statement. It's plausible because the dominant computational cost in the forward pass is multiplying by the weight matrices, while in the backward pass it's multiplying by the transposes of the weight matrices. These operations obviously have similar computational cost. 這個(gè)說(shuō)法是合理的，但需要額外的說(shuō)明來(lái)澄清這一事實(shí)。在前向傳播過(guò)程中主要的計(jì)算代價(jià)消耗在權(quán)重矩陣的乘法上，而反向傳播則是計(jì)算權(quán)重矩陣的轉(zhuǎn)置矩陣。這些操作顯然有著類似的計(jì)算代價(jià)。

所以最終的計(jì)算代價(jià)大概是兩倍的前向傳播計(jì)算大家。比起直接計(jì)算導(dǎo)數(shù)，顯然 BP 有著更大的優(yōu)勢(shì)。所以即使 BP 看起來(lái)要比 (46) 更加復(fù)雜，但實(shí)際上要更快。

這個(gè)加速在1986年首次被眾人接受，并直接導(dǎo)致神經(jīng)網(wǎng)絡(luò)可以處理的問(wèn)題的擴(kuò)展。這也導(dǎo)致了大量的研究者涌向了神經(jīng)網(wǎng)絡(luò)方向。當(dāng)然，BP 并不是萬(wàn)能鑰匙。在 1980 年代后期，人們嘗試挑戰(zhàn)極限，尤其是嘗試使用BP來(lái)訓(xùn)練深度神經(jīng)網(wǎng)絡(luò)。本書后面，我們將看到現(xiàn)代計(jì)算機(jī)和一些聰明的新想法已經(jīng)讓 BP 成功地訓(xùn)練這樣的深度神經(jīng)網(wǎng)絡(luò)。

BP 大框架

正如我所講解的，BP 提出了兩個(gè)神秘的問(wèn)題。首先，這個(gè)算法真正在干什么？我們已經(jīng)感受到從輸出處的錯(cuò)誤被反向傳回的圖景。但是我們能夠更深入一些，構(gòu)造出一種更加深刻的直覺(jué)來(lái)解釋所有這些矩陣和向量乘法么？第二神秘點(diǎn)就是，某人為什么能發(fā)現(xiàn)這個(gè) BP？跟著一個(gè)算法跑一遍甚至能夠理解證明算法 work 這是一回事。這并不真的意味著你理解了這個(gè)問(wèn)題到一定程度，能夠發(fā)現(xiàn)這個(gè)算法。是否有一個(gè)推理的思路可以指引我們發(fā)現(xiàn) BP 算法？本節(jié)，我們來(lái)探討一下這兩個(gè)謎題。
為了提升我們關(guān)于算法究竟做了什么的直覺(jué)，假設(shè)我們已經(jīng)做出一點(diǎn)小小的變動(dòng) \Delta w_{jk}^l

to be cont.