refer to : https://www.microsoft.com/en-us/research/wp-content/uploads/2016/05/prml-slides-3.pdf http://norman3.github.io/prml/docs/chapter03/0

3.1 Linear Basis Function Model

Linear Regression Model

• Linear: $wx + b$

• Regression : 연속형 응답 변수를 하나 이상의 예측 변수의 함수로 설명하는데 사용되는 통계 모델링 기법

• Example: Polynomial Curve Fitting

  

  • 일반적으로, $y(\textbf x, \textbf w) = \sum\limits_{j=0}^{M-1}w_j\phi_i(\textbf x)= \textbf w^\textbf T \phi(\textbf x)$ where $\phi_j(\textbf x)$ are known as basis functions.
  • 보통, $\phi_0(\textbf x) = 1$, so that $w_0$ acts as a bias.
  • In the simplest case, we use linear basis functions : $\phi_d(\textbf x) = x_d$.

• Polynomial basis functions: $\phi_j(x) = x^j$

  

  • These are global; $x$의 약간의 변화가 함수 전체 수식에서 다르게 적용되어 최종 출력이 크게 변한다. → approximation에 대한 한계

• Gaussian basis functions: $\phi_j(x) = \text{exp}\{-\dfrac{(x-\mu_j)^2}{2s^2}\}$

  

  • These are local; a small change in $x$ only affect nearby basis functions. $\mu_j$ and $s$ control location and scale (width).

• Sigmoidal basis functions: $\phi_j(x) = \sigma(\dfrac{x-\mu_j}{s})$

  where $\sigma(a) = \dfrac{1}{1+\textit{exp}(-a)}$.

  

  • Also these are local; a small change in $x$ only affect nearby basis functions. $\mu_j$ and $s$ control location and scale(slope).
  • $\phi (x) = x$

3.1.1 Maximum Likelihood and Least Squares

• Assume observations from a deterministic function with added Gaussian noise: $t = y(\textbf{x}, \textbf{w}) + \epsilon$ where $p(\epsilon|\beta) = \textit N(\epsilon|0, \beta^{-1})$ ($\beta ^{-1}$을 분산 값으로 사용하는 것은 계산의 편리성 때문)

  which is the same as saying, $p(t|\textbf x, \textbf w, \beta) = \mathcal N(t|y(\textbf x, \textbf w), \beta^{-1})$

  주어진 $x$에 대해 얻어질 결과 $t$에 대한 확률 분포 == 가우시안 분포

• Given observed inputs, $\textbf X = \{\textbf x_1, ..., \textbf x_N\}$ , and targets, $\textbf t = [t_1,...,t_N]^\text T$, we obtain the likelihood function : 샘플 데이터를 얻는 확률

• Taking the logarithm, we get

  

  is the sum-of-squares error.

• Computing the gradient and setting it to zero yields

$\textbf w_\text{ML}$을 Least Square Method(최소 제곱 문제)의 normal equation(정규 방정식)이라고 한다. $\Phi$는 $N \times M$ matrix로, design matrix(설계 행렬)라 불린다.

3.1.2 Geometry of Least Squares

• Consider $\textbf y = \Phi\textbf w_\text{ML} = [\varphi_1...,\varphi_M]\textbf w_\text{ML}$.

  

  $S$ is spanned by $\varphi_1,...,\varphi_M$.

  $\textbf w_\text{ML}$ minimizes the distance between $\textbf t$ and its orthogonal projection on $S$, i.e. $\textbf y$

  • $\textbf y$가 놓일 수 있는 공간 $S$로부터 실제 결과 값인 $\textbf t$ 벡터 사이의 가장 가까운 거리를 가지는 한 점을 찾는 것

3.1.3 Sequential learning