gradient descent

to minimize loss

• compute gradient
• update weights $W\leftarrow W-\alpha \nabla L$ where $\alpha$ is the learning rate

walkthrough

given training data point (x,y), and linear classifier formula $\hat{y}=Wx$ assume correct label is k, so $y=k$

using cross-entropy loss function

$Loss=-{\hat{y}}_k+log{\Sigma }_j{e}^{{\hat{y}}_j}$ we want to update W by calculating the direction to change the weights to reduce the loss: $\frac{dL}{dW}=\frac{dL}{d\hat{y}}\frac{d\hat{y}}{dW}$

we know that $\frac{d\hat{y}}{dW}=x$ (basic derivative)

to compute $\frac{dL}{d\hat{y}}$

backpropagation for computing gradients