参数拟合概念

拟合是指从训练数据学习目标函数，过拟合是指目标函数对训练数据的预测能力很强，但对测试数据预测能力很差，欠拟合是指目标函数对训练数据和测试数据的拟合程度都很差。

参数学习算法与非参数学习方法

参数学习方法是指目标函数在训练过程中参数是固定的，例如线性回归算法，而非参数学习方法是指目标函数在训练过程中参数是可变的，例如局部加权回归算法

局部加权回归算法

为了求出假设函数h<script id="MathJax-Element-14757" type="math/tex">h</script>在特定样本点x<script id="MathJax-Element-14758" type="math/tex">x</script>处得值
1. 线性回归算法：minθ∑i(yi−θTx)2<script id="MathJax-Element-14759" type="math/tex">\min_{\theta}\sum_{i}(y^i-\theta^Tx)^2</script> ,return θTx<script id="MathJax-Element-14760" type="math/tex">\theta^Tx</script>
2.局部加权回归算法：minθ∑iwi(yi−θTx)2<script id="MathJax-Element-14761" type="math/tex">\min_{\theta}\sum_{i}w^i(y^i-\theta^Tx)^2</script>
其中wi<script id="MathJax-Element-14762" type="math/tex">w^i</script>是权值，一般取wi=exp(−(xi−x)22τ2)<script id="MathJax-Element-14763" type="math/tex">w^i=exp(-\frac{(x^i-x)^2}{2\tau^2})</script>,τ<script id="MathJax-Element-14764" type="math/tex">\tau</script>称为波长函数，用来控制权值下降速率。
wi={10xi接近xxi远离x<script id="MathJax-Element-14765" type="math/tex">w^i=\begin{cases} 1 & x^i 接近 x \\ 0 & x^i 远离 x \end{cases}</script>
假设yi=θTx+ξi<script id="MathJax-Element-14766" type="math/tex">y^i=\theta^Tx+\xi^i</script>,其中ξi<script id="MathJax-Element-14767" type="math/tex">\xi^i</script>是误差项，一般取ξi=12πσ√2exp(−(ξi)22σ2)<script id="MathJax-Element-14768" type="math/tex">\xi^i=\frac{1}{\sqrt[2]{2\pi\sigma}}exp(-\frac{(\xi^i)^2}{2\sigma^2})</script>
那么有p(yi∣xi;θ)=12π√2σexp(−(yi−θTxi)22σ2)<script id="MathJax-Element-14769" type="math/tex">p(y^i\mid x^i;\theta)=\frac{1}{\sqrt[2]{2\pi}\sigma}exp(-\frac{(y^i-\theta^Tx^i)^2}{2\sigma^2})</script>,服从高斯分布N(θTxi,σ2)<script id="MathJax-Element-14770" type="math/tex">N(\theta^Tx^i,\sigma^2)</script>

极大似然法

极大似然法就是选择参数θ<script id="MathJax-Element-14771" type="math/tex">\theta</script>使得样本出现的概率最大
假设ξi<script id="MathJax-Element-14772" type="math/tex">\xi^i</script>独立同分布，定义似然函数L(θ)=p(y∣xi;θ)=∏ip(yi∣xi;θ)<script id="MathJax-Element-14773" type="math/tex">L(\theta)=p(y\mid x^i;\theta)=\prod_{i}p(y^i\mid x^i;\theta)</script>
定义对数似然函数ψ(θ)<script id="MathJax-Element-14774" type="math/tex">\psi(\theta)</script>如下：
ψ(θ)=logL(θ)=log∏ip(yi∣xi;θ)=∑i=1mlog12π−−√2σexp(−(yi−θTxi)22σ2)=mlog12π−−√2σ+∑i=1m−(yi−θTxi)22σ2<script id="MathJax-Element-14775" type="math/tex">\begin{align}\psi(\theta)=\log L(\theta)=\log\prod_{i}p(y^i\mid x^i;\theta) &=\sum_{i=1}^m\log\frac{1}{\sqrt[2]{2\pi}\sigma}exp(-\frac{(y^i-\theta^Tx^i)^2}{2\sigma^2}) \\ &=m\log\frac{1}{\sqrt[2]{2\pi}\sigma}+\sum_{i=1}^m-\frac{(y^i-\theta^Tx^i)^2}{2\sigma^2}\end{align}</script>
最大化似然函数等价于最小化损失函数J(θ)=∑mi=1(yi−θTxi)22<script id="MathJax-Element-14776" type="math/tex">J(\theta)=\sum_{i=1}^m\frac{(y^i-\theta^Tx^i)^2}{2}</script>，最小二乘法实质上就是假设误差项服从独立同分布的高斯分布使得似然函数最大化

极大似然法用于二分类问题

假设y∈{0,1}<script id="MathJax-Element-14777" type="math/tex">y\in\lbrace 0,1\rbrace</script>,那么hθ(x)∈[0,1]<script id="MathJax-Element-14778" type="math/tex">h_\theta(x)\in[0,1]</script>
选取hθ(x)=g(θTx)=g(z)=11+e−z<script id="MathJax-Element-14779" type="math/tex">h_\theta(x)=g(\theta^Tx)=g(z)=\frac{1}{1+e^{-z}}</script>，g(z)<script id="MathJax-Element-14780" type="math/tex">g(z)</script>称为逻辑函数（S型函数）
令p(y=1∣x;θ)=hθ(x)<script id="MathJax-Element-14781" type="math/tex">p(y=1\mid x;\theta)=h_\theta(x)</script>,则p(y=∣x;θ)=1−hθ(x)<script id="MathJax-Element-14782" type="math/tex">p(y=\mid x;\theta)=1-h_\theta(x)</script>
似然函数L(θ)=p(y∣x;θ)=∏ip(yi∣xi;θ)=hθ(x)yi(1−hθ(x))1−yi<script id="MathJax-Element-14783" type="math/tex">L(\theta)=p(y\mid x;\theta)=\prod_{i}p(y^i\mid x^i;\theta)=h_\theta(x)^{y^i}(1-h_\theta(x))^{1-y^i}</script>
对数似然函数ψ(θ)=logL(θ)=∑mi=1(yilog(hθ(x))+(1−yi)log(1−hθ(x)))<script id="MathJax-Element-14784" type="math/tex">\psi(\theta)=\log L(\theta)=\sum_{i=1}^m(y^i\log( h_\theta(x))+(1-y^i)\log(1-h_\theta(x)))</script>
那么可以将损失函数定义为：J(θ)=−1mψ(θ)<script id="MathJax-Element-14785" type="math/tex">J(\theta)=-\frac{1}{m}\psi(\theta)</script>,似然函数最大化就等价于损失函数最小化。
利用梯度上升法求最优解，即θ=θ+α▽θψ(θ)<script id="MathJax-Element-14786" type="math/tex">\theta=\theta+\alpha\triangledown_\theta\psi(\theta)</script>
∂ψ(θ)∂θj=−1m∑mi=1(yilog(hθ(xi))+(1−yi)log(1−hθ(xi))<script id="MathJax-Element-14787" type="math/tex">\frac{\partial{\psi(\theta)}}{\partial{\theta_j}}=-\frac{1}{m}\sum_{i=1}^m(y^i\log( h_\theta(x^i))+(1-y^i)\log(1-h_\theta(x^i))</script>
=−1m∑mi=1(yi1g(θTx)i−(1−yi)11−g(θTxi))∂g(θTxi)∂θj<script id="MathJax-Element-14788" type="math/tex">=-\frac{1}{m}\sum_{i=1}^m(y^i\frac{1}{g(\theta^Tx)^i}-(1-y^i)\frac{1}{1-g(\theta^Tx^i)})\frac{\partial{g(\theta^Tx^i)}}{\partial{\theta_j}}</script>
=−1m∑mi=1(yi1g(θTxi)−(1−yi)11−g(θTxi))g(θTxi)(1−g(θTxi))∂θTxi∂θj　　　　（１）<script id="MathJax-Element-14789" type="math/tex">=-\frac{1}{m}\sum_{i=1}^m(y^i\frac{1}{g(\theta^Tx^i)}-(1-y^i)\frac{1}{1-g(\theta^Tx^i)})g(\theta^Tx^i)(1-g(\theta^Tx^i))\frac{\partial \theta^Tx^i}{\partial \theta_j}　　　　（１）</script>
=−1m∑mi=1(yi(1−g(θTxI))−(1−yi)g(θTxi))xij　　　　　（２）<script id="MathJax-Element-14790" type="math/tex">=-\frac{1}{m}\sum_{i=1}^m(y^i(1-g(\theta^Tx^I))-(1-y^i)g(\theta^Tx^i))x_j^i　　　　　（２）</script>
=−1m∑mi=1(yi−g(θTx))xij=1m∑mi=1(hθ(xi)−yi)xij<script id="MathJax-Element-14791" type="math/tex">=-\frac{1}{m}\sum_{i=1}^m(y^i-g(\theta^Tx))x_j^i=\frac{1}{m}\sum_{i=1}^m(h_\theta(x^i)-y^i)x_j^i</script>
从（１）到（２）的过程用到以下公式：
f(x)=11+g(x)<script id="MathJax-Element-14792" type="math/tex">f(x)=\frac{1}{1+g(x)}</script>
∂f(x)∂x=1(1+eg(x))2eg(x)∂g(x)∂x<script id="MathJax-Element-14793" type="math/tex">\frac{\partial f(x)}{\partial x}=\frac{1}{(1+e^{g(x)})^2}e^{g(x)}\frac{\partial g(x)}{\partial x}</script>
　　　=11+eg(x)eg(x)1+eg(x)∂g(x)∂x<script id="MathJax-Element-14794" type="math/tex">=\frac{1}{1+e^{g(x)}}\frac{e^{g(x)}}{1+e^{g(x)}}\frac{\partial g(x)}{\partial x}</script>
　　　=f(x)(1−f(x))∂g(x)∂x<script id="MathJax-Element-14795" type="math/tex">=f(x)(1-f(x))\frac{\partial g(x)}{\partial x}</script>
所以有：θj=θj+α1m∑mi=1(yi−hθ(xi))xij<script id="MathJax-Element-14796" type="math/tex">\theta_j=\theta_j+\alpha \frac{1}{m} \sum_{i=1}^m(y^i-h_\theta(x^i))x_j^i</script>