Bayesian Linear Regression to Gaussian Process

by statian   Last Updated May 16, 2019 04:19 AM

I'm trying to understand how a Gaussian Process with a squared exponential covariance function can be obtained from Bayesian Linear Regression with a Gaussian prior $N(0,\sigma_p^2 I)$ on the parameters and an infinite number of basis functions. I'm following the proof in chapter four of Gaussian Processes for Machine Learning.

Let $\phi_c(x)=\exp(-\frac{(x-c)^2}{2\ell^2})$, where $c$ is the center of the basis function. Then Bayesian Linear Regression is a Gaussian Process with mean function $\mu(x)= 0$ and covariance function $k(x_p,x_q)=\sigma_p^2\sum_{i=1}^N\phi_c(x_p)\phi_c(x_q)$. Here's the part of the proof where I'm confused because it feels a little hand-wavy.

Now, allowing an infinite number of basis functions centered everywhere on an interval (and scaling down the variance of the prior on the weights with the number of basis functions) we obtain the limit $$ \lim_{N\to\infty}\frac{\sigma^2_p}{N}\sum_{i=1}^N\phi_c(x_p)\phi_c(x_q)=\sigma^2_p\int_{c_{\text{min}}}^{c_{\text{max}}}\phi_c(x_p)\phi_c(x_q)dc. $$ Plugging in the Gaussian-shaped basis functions and letting the integration limits go to infinity we obtain $$ \begin{align*} k(x_p, x_q) &= \sigma^2_p\int_{-\infty}^\infty\exp\Big(-\frac{(x_p - c)^2}{2\ell^2}\Big)\exp\Big(-\frac{(x_q - c)^2}{2\ell^2}\Big)dc\\ &=\sqrt{\pi}\ell\sigma^2_p\exp\Big(-\frac{(x_p - x_q)^2}{2(\sqrt{2}\ell)^2}\Big) \end{align*} $$ which we recognize as a squared exponential covariance function with a $\sqrt{2}$ times longer length-scale.

I'm hoping someone could add some further explanation to this proof. Why must the covariance function be scaled by a factor of $1/N?$ Also, it's unclear to me why we can simply change the limits of integration from $c_{\text{min}}$ and $c_{\text{max}}$ to $\pm\infty$. Maybe this is because I'm not seeing how the limit of the covariance function corresponds to having an infinite number of basis functions centered densely in $x$-space.

Related Questions

Updated January 28, 2019 10:19 AM

Updated October 25, 2016 09:08 AM

Updated May 06, 2019 03:19 AM

Updated July 02, 2018 09:19 AM