by statian
Last Updated May 16, 2019 04:19 AM

I'm trying to understand how a Gaussian Process with a squared exponential covariance function can be obtained from Bayesian Linear Regression with a Gaussian prior $N(0,\sigma_p^2 I)$ on the parameters and an infinite number of basis functions. I'm following the proof in chapter four of *Gaussian Processes for Machine Learning*.

Let $\phi_c(x)=\exp(-\frac{(x-c)^2}{2\ell^2})$, where $c$ is the center of the basis function. Then Bayesian Linear Regression is a Gaussian Process with mean function $\mu(x)= 0$ and covariance function $k(x_p,x_q)=\sigma_p^2\sum_{i=1}^N\phi_c(x_p)\phi_c(x_q)$. Here's the part of the proof where I'm confused because it feels a little hand-wavy.

Now, allowing an infinite number of basis functions centered everywhere on an interval (and scaling down the variance of the prior on the weights with the number of basis functions) we obtain the limit $$ \lim_{N\to\infty}\frac{\sigma^2_p}{N}\sum_{i=1}^N\phi_c(x_p)\phi_c(x_q)=\sigma^2_p\int_{c_{\text{min}}}^{c_{\text{max}}}\phi_c(x_p)\phi_c(x_q)dc. $$ Plugging in the Gaussian-shaped basis functions and letting the integration limits go to infinity we obtain $$ \begin{align*} k(x_p, x_q) &= \sigma^2_p\int_{-\infty}^\infty\exp\Big(-\frac{(x_p - c)^2}{2\ell^2}\Big)\exp\Big(-\frac{(x_q - c)^2}{2\ell^2}\Big)dc\\ &=\sqrt{\pi}\ell\sigma^2_p\exp\Big(-\frac{(x_p - x_q)^2}{2(\sqrt{2}\ell)^2}\Big) \end{align*} $$ which we recognize as a squared exponential covariance function with a $\sqrt{2}$ times longer length-scale.

I'm hoping someone could add some further explanation to this proof. Why must the covariance function be scaled by a factor of $1/N?$ Also, it's unclear to me why we can simply change the limits of integration from $c_{\text{min}}$ and $c_{\text{max}}$ to $\pm\infty$. Maybe this is because I'm not seeing how the limit of the covariance function corresponds to having an infinite number of basis functions centered densely in $x$-space.

Updated January 28, 2019 10:19 AM

Updated May 06, 2019 19:19 PM

Updated July 02, 2018 09:19 AM

- Serverfault Query
- Superuser Query
- Ubuntu Query
- Webapps Query
- Webmasters Query
- Programmers Query
- Dba Query
- Drupal Query
- Wordpress Query
- Magento Query
- Joomla Query
- Android Query
- Apple Query
- Game Query
- Gaming Query
- Blender Query
- Ux Query
- Cooking Query
- Photo Query
- Stats Query
- Math Query
- Diy Query
- Gis Query
- Tex Query
- Meta Query
- Electronics Query
- Stackoverflow Query
- Bitcoin Query
- Ethereum Query