文章目录

• Logistic Regression
• • Three Steps of machine learning
  • • Step1：Function Set
    • Step 2: Goodness of a Function
    • Step 3: Find the best function
• Logistic Regression + Square error
• Generative v.s. Discriminative
• Multi-class Classification

Logistic Regression

Three Steps of machine learning

Step1：Function Set

$f_{w,b}(x)=P_{w,b}\left(C_1\mid x\right)$

$w_i:\text{ weight, }b:\text{ bias, }\sigma (z):\text{ sigmoid function, }{x}_i\text{ : input }$

Step 2: Goodness of a Function

由于$L(w,b)$是乘积项的形式，为了方便计算，将上式做个变换:
$$\begin{array}{l}{w}^{\ast },b^{\ast }=\arg {\max }_{w,b}L(w,b)=\arg {\min }_{w,b}(-\ln L(w,b))\\ -\ln L(w,b)=-\ln f_{w,b}\left(x^1\right)\\ -\ln f_{w,b}\left(x^2\right)\\ -\ln \left(1-f_{w,b}\left(x^3\right)\right)\\ -\dots \end{array}$$
由于class 1和class 2的概率表达式不统一，上面的式子无法写成统一的形式，为了统一格式，将Logistic Regression里的所有Training data都打上0和1的标签，即output$\hat{y}=1$代表class 1，output$\hat{y}=0$代表class 2，于是上式进一步改写成：
$$\begin{array}{rl}-\ln L(w,b)=& -\left[{\hat{y}}^1\ln f_{w,b}\left(x^1\right)+\left(1-{\hat{y}}^1\right)\ln \left(1-f_{w,b}\left(x^1\right)\right)\right]\\ & -\left[{\hat{y}}^2\ln f_{w,b}\left(x^2\right)+\left(1-{\hat{y}}^2\right)\ln \left(1-f_{w,b}\left(x^2\right)\right)\right]\\ & -\left[{\hat{y}}^3\ln f_{w,b}\left(x^3\right)+\left(1-{\hat{y}}^3\right)\ln \left(1-f_{w,b}\left(x^3\right)\right)\right]\\ & -\dots \end{array}$$
==>
$-\ln L(w,b)={\sum }_n-\left[{\hat{y}}^n\ln f_{w,b}\left(x^n\right)+\left(1-{\hat{y}}^n\right)\ln \left(1-f_{w,b}\left(x^n\right)\right)\right]$

Step 3: Find the best function

运用梯度下降来优化
以下为推导过程：

Sigmoid函数求导：参见激活函数Sigmoid求导

Logistic Regression + Square error

当$\hat{y}=1$时，会出现$\frac{\partial L}{\partial w_i}=0$，这会造成收敛过慢

可视化 交叉熵和平方误差

Generative v.s. Discriminative

Generative model和discriminative model的差别就在于，Generative的model它有做了某些假设，假设你的data来自于某个概率模型；而Discriminative的model是完全不作任何假设的

Multi-class Classification