Perceptron

• A simplified mathematical model of a neuron

Neural Network

Regression

• Regression: predict a real value as a function of inputs

  $$ O=Wx+b $$

• o: N1, W:Nd, x:d1,b:N1

Classification

• Predicting a discrete value
• Represent class label: one-hot vector

Softmax

• Processing the output $\frac{exp(o_i)}{\sum exp(o_k)}$, the maximum function cannot differentiate

$$ Y=\text{softmax}(O) $$

Multi-Layer

• Activation function: 对线性层输出进行非线性变换的函数
• hidden layers: ReLU / Tanh / GELU 等激活函数
• output layer: Softmax（或 Sigmoid）

Gradient descent

Linear Regression