动手学深度学习笔记及部分推导

动手学深度学习

2.4 微积分

2.4.3 梯度

• 对于所有 $\mathbf A\in\mathbb R^{m\times n}$ ，都有 $\nabla_\mathbf x\mathbf{Ax} = \mathbf{A}^T$:

  证明：

  设

  $$\mathbf A = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n}\\ a_{21} & a_{22} & \cdots & a_{2n}\\ \vdots & \vdots & \ddots & \vdots\\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{bmatrix} \qquad,\qquad \mathbf x = \begin{bmatrix} x_1\\x_2\\\vdots\\x_n \end{bmatrix}$$

  ，则

  $$\begin{align} \nabla_\mathbf x\mathbf{Ax} &= \nabla_\mathbf x \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n}\\ a_{21} & a_{22} & \cdots & a_{2n}\\ \vdots & \vdots & \ddots & \vdots\\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{bmatrix} \begin{bmatrix} x_1\\x_2\\\vdots\\x_n \end{bmatrix} \\&= \nabla_\mathbf x \begin{bmatrix} a_{11}x_1 + a_{12}x_2 + \cdots + a_{1n}x_n\\ a_{21}x_1 + a_{22}x_2 + \cdots + a_{2n}x_n\\ \vdots\\ a_{m1}x_1 + a_{m2}x_2 + \cdots + a_{mn}x_n \end{bmatrix} \\&= \begin{bmatrix} a_{11} & a_{21} & \cdots & a_{m1}\\ a_{12} & a_{22} & \cdots & a_{m2}\\ \vdots & \vdots & \ddots & \vdots\\ a_{1n} & a_{2n} & \cdots & a_{mn} \end{bmatrix} \\&= \mathbf A^T \end{align}$$

  证毕。

• 对于所有 $\mathbf A\in \mathbb R^{n\times m}$ ，都有 $\nabla_\mathbf x\mathbf x^T\mathbf A = \mathbf A$

  证明：

  设

  $$\mathbf A = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1m}\\ a_{21} & a_{22} & \cdots & a_{2m}\\ \vdots & \vdots & \ddots & \vdots\\ a_{n1} & a_{n2} & \cdots & a_{nm} \end{bmatrix} \qquad,\qquad \mathbf x = \begin{bmatrix} x_1\\x_2\\\vdots\\x_n \end{bmatrix}$$

  ，则

  $$\begin{align} \nabla_\mathbf x\mathbf x^T\mathbf A &= \nabla_\mathbf x \begin{bmatrix} x_1 & x_2 & \cdots & x_n \end{bmatrix} \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1m} \\ a_{21} & a_{22} & \cdots & a_{2m} \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n2} & \cdots & a_{nm} \end{bmatrix} \\&= \nabla_\mathbf x \begin{bmatrix} x_1a_{11} + x_2a_{21} + \cdots + x_na_{n1} \\ x_1a_{12} + x_2a_{22} + \cdots + x_na_{n2} \\ \vdots \\ x_1a_{1m} + x_2a_{2m} + \cdots + x_na_{nm} \end{bmatrix} \\&= \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1m} \\ a_{21} & a_{22} & \cdots & a_{2m} \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n2} & \cdots & a_{nm} \end{bmatrix} \\&= \mathbf A \end{align}$$

  证毕。

• 对于所有 $\mathbf A\in\mathbb R^{n\times n}$ ，都有 $\nabla_\mathbf x\mathbf x^T\mathbf{Ax} = (\mathbf A + \mathbf A^T)\mathbf x$

  证明：

  设

  $$\mathbf A = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n2} & \cdots & a_{nn} \end{bmatrix} \qquad,\qquad \mathbf x = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}$$

  ，则

  $$\begin{align} \nabla_\mathbf x\mathbf x^T\mathbf{Ax} &= \nabla_\mathbf x \begin{bmatrix} x_1 & x_2 & \cdots & x_n \end{bmatrix} \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n2} & \cdots & a_{nn} \end{bmatrix} \mathbf x \\&= \nabla_\mathbf x \begin{bmatrix} x_1a_{11} + x_2a_{21} + \cdots + x_na_{n1} & x_1a_{12} + x_2a_{22} + \cdots + x_na_{n2} & \cdots & x_1a_{1n} + x_2a_{2n} + \cdots + x_na_{nn} \end{bmatrix} \mathbf x \\&= \nabla_\mathbf x \begin{bmatrix} (x_1a_{11} + x_2a_{21} + \cdots + x_na_{n1})x_1 + (x_1a_{12} + x_2a_{22} + \cdots + x_na_{n2})x_2 + \cdots + (x_1a_{1n} + x_2a_{2n} + \cdots + x_na_{nn})x_n \end{bmatrix} \\&= \nabla_\mathbf x \begin{bmatrix} a_{11}x_1 + (x_1a_{11} + x_2a_{21} + \cdots + x_na_{n1}) + a_{12}x_2 + \cdots + a_{1n}x_n \\ a_{21}x_1 + a_{22}x_2 + (x_1a_{12} + x_2a_{22} + \cdots + x_na_{n2}) + \cdots + a_{2n}x_n \\ \vdots \\ a_{n1}x_1 + a_{n2}x_2 + \cdots + a_{nn}x_n + (x_1a_{1n} + x_2a_{2n} + \cdots + x_na_{nn})\\ \end{bmatrix} \\&= \begin{bmatrix} 2a_{11} & a_{21} + a_{12} & \cdots & a_{n1} + a_{1n} \\ a_{21} + a_{12} & 2a_{22} & \cdots & a_{n2} + a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} + a_{1n} & a_{n2} + a_{2n} & \cdots & 2a_{nn} \end{bmatrix} \mathbf x \\&= (\mathbf A + \mathbf A^T)\mathbf x \end{align}$$

  证毕。

• $\nabla_\mathbf x||\mathbf x||^2 = \nabla_\mathbf x\mathbf x^T\mathbf x = 2\mathbf x$

  证明：

  设

  $$\mathbf x = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}$$

  则

  $$\begin{align} \nabla_\mathbf x||\mathbf x||^2 &= \nabla_\mathbf x(x_1^2 + x_2^2 + \cdots + x_n^2) \\&= \nabla_\mathbf x\mathbf x^T\mathbf x \\&= \begin{bmatrix} 2x_1 \\ 2x_2 \\ \vdots \\ 2x_n \end{bmatrix} \\&= 2\mathbf x \end{align}$$

  证毕。