* Martens [2020] James Martens.  New insights and perspectives on the natural gradient method.  _Journal of Machine Learning Research_ , 21(146):1–76, 2020.  URL http://jmlr.org/papers/v21/17-678.html.
  * Meng et al. [2020] X. Meng, R. Bachmann, and M. E. Khan.  Training binary neural networks using the bayesian learning rule.  _arXiv preprint arXiv:2002.10778_ , 2020.
  * Mishkin et al. [2018] A. Mishkin, F. Kunstner, D. Nielsen M. Schmidt, and M. E. Khan.  Slang: Fast structured covariance approximations for bayesian deep learning with natural gradient.  In _Advances in Neural Information Processing Systems 31_. 2018.
  * Mobahi and Fisher III [2015] H. Mobahi and J. W. Fisher III.  A theoretical analysis of optimization by Gaussian continuation.  In _AAAI Conference on Artificial Intelligence_ , pages 1205–1211, 2015.
  * Murphy [2012] K. P. Murphy.  _Machine Learning: A Probabilistic Perspective_.  The MIT Press, 2012.  ISBN 0262018020, 9780262018029.
  * Neal and Hinton [1998] R. M. Neal and G. E. Hinton.  A view of the em algorithm that justifies incremental, sparse, and other variants.  In _Learning in graphical models_ , pages 355–368. Springer, 1998\.
  * Nemirovski and Yudin [1978] A. Nemirovski and D. Yudin.  On cesaro’s convergence of the gradient descent method for finding saddle points of convex-concave functions.  _Doklady Akademii Nauk SSSR_ , 239(4), 1978.
  * Nielsen and Garcia [2009] F. Nielsen and V. Garcia.  Statistical exponential families: A digest with flash cards.  _arXiv preprint arXiv:0911.4863_ , 2009.
  * Ollivier [2018] Y. Ollivier.  Online natural gradient as a kalman filter.  _Electronic Journal of Statistics_ , 12(2):2930–2961, 2018.
  * Ollivier et al. [2017] Y. Ollivier, L. Arnold, A. Auger, and N. Hansen.  Information-geometric optimization algorithms: A unifying picture via invariance principles.  _Journal of Machine Learning Research_ , 18(18):1–65, 2017.
  * Opper and Archambeau [2009] M. Opper and C. Archambeau.  The variational Gaussian approximation revisited.  _Neural Computation_ , 21(3):786–792, 2009.
  * Osawa et al. [2019] K. Osawa, S. Swaroop, M. E. Khan, A. Jain, R. Eschenhagen, R. E. Turner, and R. Yokota.  Practical deep learning with bayesian principles.  In _Advances in neural information processing systems_ , pages 4287–4299, 2019.
  * Pascanu and Bengio [2013] R. Pascanu and Y. Bengio.  Revisiting natural gradient for deep networks.  _arXiv preprint arXiv:1301.3584_ , 2013.
  * Polyak [1964] B. T. Polyak.  Some methods of speeding up the convergence of iteration methods.  _USSR Computational Mathematics and Mathematical Physics_ , 4(5):1–17, 1964.
  * Price [1958] R. Price.  A useful theorem for nonlinear devices having gaussian inputs.  _IRE Transactions on Information Theory_ , 4(2):69–72, 1958.
  * Raskutti and Mukherjee [2015] G. Raskutti and S. Mukherjee.  The information geometry of mirror descent.  _IEEE Transactions on Information Theory_ , 61(3):1451–1457, 2015.
  * Rasmussen and Williams [2006] C. E. Rasmussen and C. K. I. Williams.  _Gaussian Processes for Machine Learning_.  MIT Press, 2006.
  * Rezende et al. [2014] D. J. Rezende, S. Mohamed, and D. Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  In _International Conference on Machine Learning_ , pages 1278–1286, 2014.
  * Ritter et al. [2018] H. Ritter, A. Botev, and D. Barber.  A scalable laplace approximation for neural networks.  In _International Conference on Learning Representations_ , 2018.
  * Robbins and Monro [1951] H. Robbins and S. Monro.  A stochastic approximation method.  _Ann. Math. Statist._ , 22(3):400–407, 9 1951\.  doi: 10.1214/aoms/1177729586.  URL http://dx.doi.org/10.1214/aoms/1177729586.
  * Roweis and Ghahramani [1999] S. Roweis and Z. Ghahramani.  A unifying review of linear gaussian models.  _Neural computation_ , 11(2):305–345, 1999.
  * Rue and Martino [2007] H. Rue and S. Martino.  Approximate Bayesian inference for hierarchical Gaussian Markov random fields models.  _Journal of Statistical Planning and Inference_ , 137(10):3177–3192, 2007.  Special Issue: Bayesian Inference for Stochastic Processes.
  * Rue et al. [2009] H. Rue, S. Martino, and N. Chopin.  Approximate Bayesian inference for latent Gaussian models using integrated nested Laplace approximations (with discussion).  _Journal of the Royal Statistical Society, Series B_ , 71(2):319–392, 2009.
  * Rue et al. [2017] H. Rue, A. Riebler, S. H. Sørbye, J. B. Illian, D. P. Simpson, and F. K. Lindgren.  Bayesian computing with INLA: A review.  _Annual Reviews of Statistics and Its Applications_ , 4(March):395–421, 2017.  doi: 10.1146/annurev-statistics-060116-054045.
  * Salimans and Knowles [2013] T. Salimans and D. A. Knowles.  Fixed-form variational posterior approximation through stochastic linear regression.  _Bayesian Analysis_ , 8(4):837–882, 2013.
  * Salimbeni et al. [2018] H. Salimbeni, S. Eleftheriadis, and J. Hensman.  Natural gradients in practice: Non-conjugate variational inference in Gaussian process models.  In _International Conference on Artificial Intelligence and Statistics_ , pages 689–697. PMLR, 2018.
  * Sato [1999] M-A. Sato.  Fast learning of on-line em algorithm.  Technical report, ATR Human Information Processing Research Laboratories, 1999.
  * Sato [2001] M-A. Sato.  Online model selection based on the variational Bayes.  _Neural Computation_ , 13(7):1649–1681, 2001\.
  * Schaul et al. [2013] T. Schaul, S. Zhang, and Y. LeCun.  No more pesky learning rates.  In _International Conference on Machine Learning_ , pages 343–351, 2013.
  * Sheth [2019] R. Sheth.  _Algorithms and Theory for Variational Inference in Two-Level Non-conjugate Models_.  PhD thesis, Tufts University, 2019.
  * Sheth and Khardon [2016] R. Sheth and R. Khardon.  Monte carlo structured svi for two-level non-conjugate models.  _arXiv preprint arXiv:1612.03957_ , 2016.
  * Srivastava et al. [2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(56):1929–1958, 2014.  URL http://jmlr.org/papers/v15/srivastava14a.html.
  * Stratonovich [1965] R. L. Stratonovich.  Conditional markov processes.  In _Non-linear transformations of stochastic processes_ , pages 427–453. Elsevier, 1965.
  * Sutskever et al. [2013] I. Sutskever, J. Martens, G. Dahl, and G. Hinton.  On the importance of initialization and momentum in deep learning.  In _International conference on machine learning_ , pages 1139–1147, 2013.
  * Tieleman and Hinton [2012] T. Tieleman and G. Hinton.  Lecture 6.5-RMSprop: Divide the gradient by a running average of its recent magnitude.  _COURSERA: Neural Networks for Machine Learning 4_ , 2012.
  * Tierney and Kadane [1986] L. Tierney and J. B. Kadane.  Accurate approximations for posterior moments and marginal densities.  _Journal of the American Statistical Association_ , 81(393):82–86, 1986.
  * Titterington [1984] D. M. Titterington.  Recursive parameter estimation using incomplete data.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 46(2):257–267, 1984.
  * van de Meent et al. [2018] J-W. van de Meent, B. Paige, H. Yang, and F. Wood.  An introduction to probabilistic programming.  _arXiv preprint arXiv:1809.10756_ , 2018.
  * Ver Hoef [2012] J. M. Ver Hoef.  Who invented the delta method?  _The American Statistician_ , 66(2):124–127, 2012\.
  * Vovk [1990] V. G. Vovk.  Aggregating strategies.  In _Proceedings of the Third Annual Workshop on Computational Learning Theory_ , COLT ’90, pages 371–386, San Francisco, CA, USA, 1990. Morgan Kaufmann Publishers Inc.  ISBN 1-55860-146-5.
  * Wang and Blei [2013] C. Wang and D. M. Blei.  Variational inference in nonconjugate models.  _Journal of Machine Learning Research_ , 14(1):1005–1031, 2013.
  * Welling et al. [2008] M. Welling, C. Chemudugunta, and N. Sutter.  Deterministic latent variable models and their pitfalls.  In _Proceedings of the 2008 SIAM International Conference on Data Mining_ , pages 196–207. SIAM, 2008.
  * Wierstra et al. [2014] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, and J. Schmidhuber.  Natural evolution strategies.  _Journal of Machine Learning Research_ , 15(1):949–980, 2014.
  * Wilson et al. [2017] A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht.  The marginal value of adaptive gradient methods in machine learning.  In _Advances in Neural Information Processing Systems_ , pages 4151–4161, 2017.
  * Winn and Bishop [2005] J. Winn and C. M. Bishop.  Variational message passing.  _Journal of Machine Learning Research_ , 6(Apr):661–694, 2005.
  * Wong [2015] K-C. Wong.  Evolutionary multimodal optimization: A short survey.  _arXiv preprint arXiv:1508.00457_ , 2015.
  * Yin et al. [2019] P. Yin, J. Lyu, S. Zhang, S. Osher, Y. Qi, and J. Xin.  Understanding straight-through estimator in training activation quantized neural nets.  _ICLR_ , 2019.
  * Yu and Gen [2010] X. Yu and M. Gen.  _Introduction to evolutionary algorithms_.  Springer Science & Business Media, 2010.
  * Zeiler [2012] M. D. Zeiler.  ADADELTA: An adaptive learning rate method.  _arXiv preprint arXiv:1212.5701_ , 2012.
  * Zellner [1988] A. Zellner.  Optimal information processing and Bayes’s theorem.  _The American Statistician_ , 42(4):278–280, 1988\.  doi: 10.1080/00031305.1988.10475585.  URL https://amstat.tandfonline.com/doi/abs/10.1080/00031305.1988.10475585.
  * Zhang et al. [2018a] C. Zhang, J. Bütepage, H. Kjellström, and S. Mandt.  Advances in variational inference.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(8):2008–2026, 2018a.
  * Zhang et al. [2018b] G. Zhang, S. Sun, D. K. Duvenaud, and R. B. Grosse.  Noisy natural gradient as variational inference.  _arXiv preprint arXiv:1712.02390_ , 2018b.
  * Zhang [1999] Tong Zhang.  Theoretical analysis of a class of randomized regularization methods.  In _Proceedings of the Twelfth Annual Conference on Computational Learning Theory_ , COLT ’99, page 156–163, New York, NY, USA, 1999\. Association for Computing Machinery.  ISBN 1581131674.  doi: 10.1145/307400.307433.  URL https://doi.org/10.1145/307400.307433.

##  Appendix A Bayesian inference as optimization

Bayesian inference is a special case of the Bayesian learning problem.
Although this follows directly from the variational formulation by Zellner
[1988], it might be useful to provide some more motivation. Bayesian inference
corresponds to a log-likelihood loss
$\ell(y,f_{\mathbf{\theta}}(\text{$\boldsymbol{x}$}))=-\log
p(y|f_{\mathbf{\theta}}(\text{$\boldsymbol{x}$}))$. With conditional
independent data and prior $p(\text{$\boldsymbol{\theta}$})$, the posterior
distribution is $p(\text{$\boldsymbol{\theta}$}|\text{${\cal
D}$})=p(\text{$\boldsymbol{\theta}$})/\mathcal{Z}(\text{${\cal
D}$})\prod_{i=1}^{N}p(\text{$\boldsymbol{y}$}_{i}|f_{\theta}(\text{$\boldsymbol{x}$}_{i}))$.
The minimizer of the Bayesian learning problem recovers this posterior
distribution. This follows directly from reorganizing the Bayesian objective
Eq. 2:

$\displaystyle\mathcal{L}(q)$
$\displaystyle=-\mathbb{E}_{q(\mathbf{\theta})}\left[\sum_{i=1}^{N}\log
p(y_{i}|f_{\mathbf{\theta}}(\text{$\boldsymbol{x}$}_{i}))\right]+\mathbb{D}_{KL}[q(\text{$\boldsymbol{\theta}$})\,\|\,p(\text{$\boldsymbol{\theta}$})]$
(61)
$\displaystyle=\mathbb{E}_{q(\mathbf{\theta})}\left[\log\frac{q(\text{$\boldsymbol{\theta}$})}{\frac{p(\text{$\boldsymbol{\theta}$})}{\mathcal{Z}(\mathcal{D})}\prod_{i=1}^{N}p(y_{i}|f_{\mathbf{\theta}}(\text{$\boldsymbol{x}$}_{i}))}\right]+\log\mathcal{Z}(\text{${\cal
D}$})$ (62)
$\displaystyle=\mathbb{D}_{KL}[q(\text{$\boldsymbol{\theta}$})\,\|\,p(\text{$\boldsymbol{\theta}$}|\text{${\cal
D}$})]+\log\mathcal{Z}(\text{${\cal D}$})$ (63)

Choosing $q(\text{$\boldsymbol{\theta}$})$ as
$p(\text{$\boldsymbol{\theta}$}|\text{${\cal D}$})$ minimizes the above
equation since $\mathcal{Z}(\text{${\cal D}$})$ is a constant, and the KL
divergence is zero.

##  Appendix B Natural gradient of the entropy

Below, we show that
$\widetilde{\nabla}_{\text{$\boldsymbol{\lambda}$}}\mathcal{H}(q)=-\text{$\boldsymbol{\lambda}$}$
when $h(\text{$\boldsymbol{\theta}$})$ is a constant, We rewrite the gradient
with respect to $\boldsymbol{\mu}$,

$-\nabla_{\text{$\boldsymbol{\lambda}$}}\mathcal{H}(q)=\text{$\boldsymbol{F}$}(\text{$\boldsymbol{\lambda}$})\,\nabla_{\text{$\boldsymbol{\mu}$}}\mathbb{E}_{q}[\log
q(\text{$\boldsymbol{\theta}$})],$

since that gradient has a simpler expression as shown below,

$\begin{split}\nabla_{\text{$\boldsymbol{\mu}$}}\mathbb{E}_{q}[\log
q(\text{$\boldsymbol{\theta}$})]&=\nabla_{\text{$\boldsymbol{\mu}$}}\left[\langle\text{$\boldsymbol{\lambda}$},\text{$\boldsymbol{\mu}$}\rangle-A(\text{$\boldsymbol{\lambda}$})\right]+\nabla_{\text{$\boldsymbol{\mu}$}}\mathbb{E}_{q}[\log
h(\text{$\boldsymbol{\theta}$})]\\\
&=\text{$\boldsymbol{\lambda}$}+\left(\nabla_{\text{$\boldsymbol{\lambda}$}}^{2}A(\text{$\boldsymbol{\lambda}$})\right)^{-1}\left[\text{$\boldsymbol{\mu}$}-\nabla_{\text{$\boldsymbol{\lambda}$}}A(\text{$\boldsymbol{\lambda}$})\right]+\nabla_{\text{$\boldsymbol{\mu}$}}\mathbb{E}_{q}[\log
h(\text{$\boldsymbol{\theta}$})]\\\
&=\text{$\boldsymbol{\lambda}$}+\nabla_{\text{$\boldsymbol{\mu}$}}\mathbb{E}_{q}[\log
h(\text{$\boldsymbol{\theta}$})].\end{split}$ (64)

By scaling with the inverse of the FIM, we get
$-\widetilde{\nabla}_{\text{$\boldsymbol{\lambda}$}}\mathcal{H}(q)=\widetilde{\nabla}_{\text{$\boldsymbol{\lambda}$}}\mathbb{E}_{q}[\log
q(\text{$\boldsymbol{\theta}$})]=\text{$\boldsymbol{\lambda}$}+\widetilde{\nabla}_{\text{$\boldsymbol{\lambda}$}}\mathbb{E}_{q}[\log
h(\text{$\boldsymbol{\theta}$})]$, which for constant
$h(\text{$\boldsymbol{\theta}$})$ reduces to $\boldsymbol{\lambda}$.

##  Appendix C The delta method

In the delta method [Dorfman, 1938, Ver Hoef, 2012], we approximate the
expectation of a function of a random variable by the expectation of the
function’s Taylor expansion. For example, given a distribution
$q(\text{$\boldsymbol{\theta}$})$ with mean $\boldsymbol{m}$, we can use the
first-order Taylor approximation to get,

$\mathbb{E}_{q}[f(\text{$\boldsymbol{\theta}$})]\approx\mathbb{E}_{q}\left[f(\text{$\boldsymbol{m}$})+(\text{$\boldsymbol{\theta}$}-\text{$\boldsymbol{m}$})^{\top}\left.\nabla_{\text{$\boldsymbol{\theta}$}}f(\text{$\boldsymbol{\theta}$})\right|_{\text{$\boldsymbol{\theta}$}=\text{$\boldsymbol{m}$}}\right]\,\,\approx
f(\text{$\boldsymbol{m}$})$

We will often use this approximation to approximate the expectation of
gradient/Hessian at their values at a single point (usually the mean). For
example, by using the first-order Taylor expansion, we approximate below the
expectation of the gradient by the gradient at the mean,

$\displaystyle\mathbb{E}_{q}[\nabla_{\text{$\boldsymbol{\theta}$}}f(\text{$\boldsymbol{\theta}$})]$
$\displaystyle=\nabla_{\text{$\boldsymbol{m}$}}\mathbb{E}_{q}[f(\text{$\boldsymbol{\theta}$})]$
$\displaystyle\approx\nabla_{\text{$\boldsymbol{m}$}}\mathbb{E}_{q}\left[f(\text{$\boldsymbol{m}$})+(\text{$\boldsymbol{\theta}$}-\text{$\boldsymbol{m}$})^{\top}\left.\nabla_{\text{$\boldsymbol{\theta}$}}f(\text{$\boldsymbol{\theta}$})\right|_{\text{$\boldsymbol{\theta}$}=\text{$\boldsymbol{m}$}}\right]$
$\displaystyle=\left.\nabla_{\text{$\boldsymbol{\theta}$}}f(\text{$\boldsymbol{\theta}$})\right|_{\text{$\boldsymbol{\theta}$}=\text{$\boldsymbol{m}$}},$
(65)

where the first line is due to Bonnet’s theorem (App. D) and the second line
is simply the first-order Taylor expansion. We refer to this as the first-
order delta method. Similar applications has been used in variational
inference [Blei and Lafferty, 2007, Braun and McAuliffe, 2010, Wang and Blei,
2013].

Similarly, by using the second-order Taylor expansion, we approximate below
the expectation of the Hessian by the Hessian at the mean,

$\displaystyle\mathbb{E}_{q}[\nabla_{\text{$\boldsymbol{\theta}$}}^{2}f(\text{$\boldsymbol{\theta}$})]$
$\displaystyle=2\nabla_{\text{$\boldsymbol{S}$}^{-1}}\mathbb{E}_{q}[f(\text{$\boldsymbol{\theta}$})]$
$\displaystyle\approx
2\nabla_{\text{$\boldsymbol{S}$}^{-1}}\mathbb{E}_{q}\left[f(\text{$\boldsymbol{m}$})+(\text{$\boldsymbol{m}$}-\text{$\boldsymbol{\theta}$})^{\top}\left.\nabla_{\text{$\boldsymbol{\theta}$}}f(\text{$\boldsymbol{\theta}$})\right|_{\text{$\boldsymbol{\theta}$}=\text{$\boldsymbol{m}$}}+\text{$\frac{1}{2}$}(\text{$\boldsymbol{m}$}-\text{$\boldsymbol{\theta}$})^{\top}\left.\nabla_{\text{$\boldsymbol{\theta}$}}^{2}f(\text{$\boldsymbol{\theta}$})\right|_{\text{$\boldsymbol{\theta}$}=\text{$\boldsymbol{m}$}}(\text{$\boldsymbol{m}$}-\text{$\boldsymbol{\theta}$})\right]$
$\displaystyle=2\nabla_{\text{$\boldsymbol{S}$}^{-1}}\text{Tr}\left[\text{$\frac{1}{2}$}\text{$\boldsymbol{S}$}^{-1}\left.\nabla_{\text{$\boldsymbol{\theta}$}}^{2}f(\text{$\boldsymbol{\theta}$})\right|_{\text{$\boldsymbol{\theta}$}=\text{$\boldsymbol{m}$}}\right]$
$\displaystyle=\left.\nabla_{\text{$\boldsymbol{\theta}$}}^{2}f(\text{$\boldsymbol{\theta}$})\right|_{\text{$\boldsymbol{\theta}$}=\text{$\boldsymbol{m}$}},$
(66)

where the first line is due to Price’s theorem (App. D) and the second line is
simply the second-order Taylor expansion. We refer to this as the second-order
delta method.

We also note that both the first and second-order approximation can be derived
in much simpler way by using a zeroth-order expansion where we do not expand
the function inside at all,

$\mathbb{E}_{q}[\nabla_{\text{$\boldsymbol{\theta}$}}f(\text{$\boldsymbol{\theta}$})]\approx\left.\nabla_{\text{$\boldsymbol{\theta}$}}f(\text{$\boldsymbol{\theta}$})\right|_{\text{$\boldsymbol{\theta}$}=\text{$\boldsymbol{m}$}}\qquad\mathbb{E}_{q}[\nabla^{2}_{\text{$\boldsymbol{\theta}$}}f(\text{$\boldsymbol{\theta}$})]\approx\left.\nabla_{\text{$\boldsymbol{\theta}$}}^{2}f(\text{$\boldsymbol{\theta}$})\right|_{\text{$\boldsymbol{\theta}$}=\text{$\boldsymbol{m}$}}.$

In any case, these can be seen as simply approximating the expectations of a
function with the function value at a single point (the mean).

##  Appendix D Newton’s method from the BLR

We start with a summary of the Theorem’s of Bonnet’s and Price’s [Bonnet,
1964, Price, 1958], see also Opper and Archambeau [2009], Rezende et al.
[2014]. Let $\boldsymbol{x}$ be multivariate normal with mean
$\boldsymbol{\mu}$ and covariance matrix $\boldsymbol{C}$,
$f(\text{$\boldsymbol{x}$})$ a twice differentiable function where
$\mathbb{E}{|f(\text{$\boldsymbol{x}$})|}<\infty$, then the Bonnet and Price’s
results are

$\frac{\partial}{\partial\mu_{i}}\mathbb{E}\left[f(\text{$\boldsymbol{x}$})\right]=\mathbb{E}\left[\frac{\partial}{\partial
x_{i}}f(\text{$\boldsymbol{x}$})\right]\quad\text{and}\quad\frac{\partial}{\partial
C_{ij}}\mathbb{E}\left[f(\text{$\boldsymbol{x}$})\right]=c_{ij}\mathbb{E}\left[\frac{\partial^{2}}{\partial
x_{i}\partial x_{j}}f(\text{$\boldsymbol{x}$})\right]$

where $c_{ij}=1/2$ if $i=j$ and $c_{ij}=1$ if $i\not=j$ (since
$\boldsymbol{C}$ is symmetric), respectively. Using these, we obtain Eq. 10
and Eq. 11.

The following derivation is also detailed in Khan et al. [2018]. We now use
the definitions of natural and expectation parameters from Eq. 9 in the BLR
Eq. 6. The base-measure is constant for the multivariate Gaussian, hence

$\displaystyle\text{$\boldsymbol{S}$}_{t+1}\text{$\boldsymbol{m}$}_{t+1}$
$\displaystyle\leftarrow(1-\rho_{t})\text{$\boldsymbol{S}$}_{t}\text{$\boldsymbol{m}$}_{t}-\rho_{t}\nabla_{\boldsymbol{\mu}^{(1)}}\mathbb{E}_{q_{t}}[\bar{\ell}(\text{$\boldsymbol{\theta}$})]$
$\displaystyle\text{$\boldsymbol{S}$}_{t+1}$
$\displaystyle\leftarrow(1-\rho_{t})\text{$\boldsymbol{S}$}_{t}+2\rho_{t}\nabla_{\boldsymbol{\mu}^{(2)}}\mathbb{E}_{q_{t}}[\bar{\ell}(\text{$\boldsymbol{\theta}$})].$

The update for $\text{$\boldsymbol{S}$}_{t+1}$ can be obtained using Eq. 11,

$\text{$\boldsymbol{S}$}_{t+1}\leftarrow(1-\rho_{t})\text{$\boldsymbol{S}$}_{t}+\rho_{t}\mathbb{E}_{q_{t}}\left[\nabla_{\text{\text{$\boldsymbol{\theta}$}}}^{2}\bar{\ell}(\text{$\boldsymbol{\theta}$})\right].$

We can simplify the update for the mean using Eq. 10,

$\displaystyle\text{$\boldsymbol{S}$}_{t+1}\text{$\boldsymbol{m}$}_{t+1}$
$\displaystyle=(1-\rho_{t})\text{$\boldsymbol{S}$}_{t}\text{$\boldsymbol{m}$}_{t}-\rho_{t}\left\\{\mathbb{E}_{q_{t})}[\nabla_{\boldsymbol{\theta}}\bar{\ell}(\text{$\boldsymbol{\theta}$})]-\mathbb{E}_{q_{t}}[\nabla_{\boldsymbol{\theta}}^{2}\bar{\ell}(\text{$\boldsymbol{\theta}$})]\text{$\boldsymbol{m}$}_{t}\right\\},$
$\displaystyle=\text{$\boldsymbol{S}$}_{t+1}\text{$\boldsymbol{m}$}_{t}-\rho_{t}\mathbb{E}_{q_{t}}[\nabla_{\boldsymbol{\theta}}\bar{\ell}(\text{$\boldsymbol{\theta}$})],$

and the update for $\text{$\boldsymbol{m}$}_{t+1}$ follows.

##  Appendix E Natural-gradient updates for mixture of Gaussians

The joint distribution $q(\text{$\boldsymbol{\theta}$},z)$ is a minimal-
conditional exponential-family distribution [Lin et al., 2019b], i.e., the
distribution then takes a minimal form conditioned over $z=k$ which ensures
that the FIM is non-singular. Details of the update are in Lin et al. [2019b,
Theorem 3]. The natural and expectation parameters are given in Lin et al.
[2019b, Table 1]).

The gradients with respect to $\text{$\boldsymbol{\mu}$}_{k}$ can be written
in terms of $\text{$\boldsymbol{m}$}_{k}$ and
$\text{$\boldsymbol{S}$}_{k}^{-1}$ as

$\displaystyle\nabla_{\boldsymbol{\mu}_{k}^{(1)}}\mathbb{E}_{q}[\bar{\ell}(\text{$\boldsymbol{\theta}$})]$
$\displaystyle=\left[\nabla_{\boldsymbol{\mu}_{k}^{(1)}}\text{$\boldsymbol{m}$}_{k}^{\top}\right]\nabla_{\boldsymbol{\mathbf{m}}_{k}}\mathbb{E}_{q}[\bar{\ell}(\text{$\boldsymbol{\theta}$})]=\frac{1}{\pi_{k}}\nabla_{\boldsymbol{\mathbf{m}}_{k}}\mathbb{E}_{q}[\bar{\ell}(\text{$\boldsymbol{\theta}$})]$

Hence we can utilize from the derivation of Newton’s method, Eq. 10 and Eq.
11, and do similarly as in App. D, to achieve a Newton-like update for a
mixture of Gaussians

$\displaystyle\text{$\boldsymbol{m}$}_{k,t+1}$
$\displaystyle\leftarrow\text{$\boldsymbol{m}$}_{k,t}-\frac{\rho}{\pi_{k}}\text{$\boldsymbol{S}$}_{k,t+1}^{-1}\nabla_{\mathbf{m}_{k}}\mathbb{E}_{q_{t}}\left[\bar{\ell}(\text{$\boldsymbol{\theta}$})+\log
q(\text{$\boldsymbol{\theta}$})\right],$ (67)
$\displaystyle\text{$\boldsymbol{S}$}_{k,t+1}$
$\displaystyle\leftarrow\text{$\boldsymbol{S}$}_{k,t}+\frac{2\rho}{\pi_{k}}\nabla_{\mathbf{S}_{k,t}^{-1}}\mathbb{E}_{q_{t}}\left[\bar{\ell}(\text{$\boldsymbol{\theta}$})+\log
q(\text{$\boldsymbol{\theta}$})\right].$ (68)

The gradients can be expressed in terms of the gradient and Hessian of the
loss, similar to Bonnet’s and Price’s Theorem in App. D, see [Lin et al.,
2019a, Thm 6 and 7]. This gives

$\displaystyle\nabla_{\mathbf{m}_{k}}\mathbb{E}_{q}[f(\text{$\boldsymbol{\theta}$})]$
$\displaystyle=\pi_{k}\mathbb{E}_{\mathcal{N}(\text{\text{$\boldsymbol{\theta}$}}|\mathbf{m}_{k},\mathbf{S}_{k}^{-1})}[\nabla_{\boldsymbol{\theta}}f(\text{$\boldsymbol{\theta}$})],$
$\displaystyle\nabla_{\mathbf{S}_{k}^{-1}}\mathbb{E}_{q}[f(\text{$\boldsymbol{\theta}$})]$
$\displaystyle=\frac{\pi_{k}}{2}\mathbb{E}_{\mathcal{N}(\text{\text{$\boldsymbol{\theta}$}}|\mathbf{m}_{k},\mathbf{S}_{k}^{-1})}[\nabla_{\boldsymbol{\theta}}^{2}f(\text{$\boldsymbol{\theta}$})],$

and hence the updates in Eqs. 29 and 30.

The gradient of $\log
q(\text{$\boldsymbol{\theta}$})=\log\sum_{k=1}^{K}\pi_{k}\text{${\cal
N}$}(\text{$\boldsymbol{\theta}$}|\text{$\boldsymbol{m}$}_{k},\text{$\boldsymbol{S}$}_{k}^{-1})$
were not spelled out explicit in Eqs. 67 and 68, but are as follows,

$\displaystyle\nabla_{\text{\text{$\boldsymbol{\theta}$}}}\log
q(\text{$\boldsymbol{\theta}$})$
$\displaystyle=\sum_{j=1}^{K}r_{j}(\text{$\boldsymbol{\theta}$})\text{$\boldsymbol{S}$}_{j}(\text{$\boldsymbol{m}$}_{j}-\text{$\boldsymbol{\theta}$}),$
(69) $\displaystyle\nabla^{2}_{\text{\text{$\boldsymbol{\theta}$}}}\log
q(\text{$\boldsymbol{\theta}$})$
$\displaystyle=\sum_{j=1}^{K}r_{j}(\text{$\boldsymbol{\theta}$})\left[\text{$\boldsymbol{A}$}_{jj}(\text{$\boldsymbol{\theta}$})-\text{$\boldsymbol{S}$}_{j}-\sum_{i=1}^{K}r_{i}(\text{$\boldsymbol{\theta}$})\text{$\boldsymbol{A}$}_{ij}(\text{$\boldsymbol{\theta}$})\right].$
(70)

Here, $r_{k}(\text{$\boldsymbol{\theta}$})$ is the responsibility of the
$k$’th mixture component at $\boldsymbol{\theta}$,
$q(z=k|\text{$\boldsymbol{\theta}$})$

$\displaystyle q(z=k|\text{$\boldsymbol{\theta}$})=\frac{\pi_{k}\text{${\cal
N}$}(\text{$\boldsymbol{\theta}$}|\text{$\boldsymbol{m}$}_{k},\text{$\boldsymbol{S}$}_{k}^{-1})}{\sum_{j=1}^{K}\pi_{j}\text{${\cal
N}$}(\text{$\boldsymbol{\theta}$}|\text{$\boldsymbol{m}$}_{j},\text{$\boldsymbol{S}$}_{j}^{-1})}.$
(71)

and
$\text{$\boldsymbol{A}$}_{ij}(\text{$\boldsymbol{\theta}$})=\text{$\boldsymbol{S}$}_{i}(\text{$\boldsymbol{m}$}_{i}-\text{$\boldsymbol{\theta}$})(\text{$\boldsymbol{m}$}_{j}-\text{$\boldsymbol{\theta}$})^{\top}\text{$\boldsymbol{S}$}_{j}$.

There are two natural situations that lead to simplifications, which can form
the basis for various approximations. First, is when
$\text{$\boldsymbol{\theta}$}_{k}=\text{$\boldsymbol{m}$}_{k}$, then
$\text{$\boldsymbol{A}$}_{ik}(\text{$\boldsymbol{\theta}$}_{k})=0$ for all
$i$, leading to

$\displaystyle\nabla_{\text{\text{$\boldsymbol{\theta}$}}}\log
q(\text{$\boldsymbol{\theta}$}_{k})$ $\displaystyle=\sum_{j\neq
k}r_{j}(\text{$\boldsymbol{\theta}$}_{k})\text{$\boldsymbol{S}$}_{j}(\text{$\boldsymbol{\theta}$}_{j}-\text{$\boldsymbol{\theta}$}_{k}),$
(72) $\displaystyle\nabla^{2}_{\text{\text{$\boldsymbol{\theta}$}}}\log
q(\text{$\boldsymbol{\theta}$}_{k})$
$\displaystyle=-r_{k}(\text{$\boldsymbol{\theta}$}_{k})\text{$\boldsymbol{S}$}_{k}+\sum_{j\neq
k}r_{j}(\text{$\boldsymbol{\theta}$}_{k})\left[\text{$\boldsymbol{A}$}_{ii}(\text{$\boldsymbol{\theta}$}_{k})-\text{$\boldsymbol{S}$}_{j}-\sum_{i=1,i\neq
k}^{K}r_{i}(\text{$\boldsymbol{\theta}$}_{k})\text{$\boldsymbol{A}$}_{ij}(\text{$\boldsymbol{\theta}$}_{k})\right],$
(73)

Second, when mixture components are far apart from each other, then the
responsibility is approximately zero for all $j\neq k$, leading to

$\displaystyle\nabla_{\text{\text{$\boldsymbol{\theta}$}}}\log
q(\text{$\boldsymbol{\theta}$}_{k})$ $\displaystyle\approx
0,\quad\quad\nabla^{2}_{\text{\text{$\boldsymbol{\theta}$}}\mathbf{\theta}}\log
q(\text{$\boldsymbol{\theta}$}_{k})\approx-\text{$\boldsymbol{S}$}_{k}.$ (74)

##  Appendix F Deep learning algorithms with momentum

Momentum is a useful technique to improve the performance of SGD [Sutskever et
al., 2013]. Modern adaptive-learning algorithms, such as RMSprop and Adam,
often employ variants of the classical momentum to increase the learning rate
[Sutskever et al., 2013].

The classical momentum method is based on the Polyak’s heavy-ball method
Polyak [1964], where the current
$\text{$\boldsymbol{\theta}$}_{t+1}-\text{$\boldsymbol{\theta}$}_{t}$ update
is pushed along the direction of the previous update
$\text{$\boldsymbol{\theta}$}_{t}-\text{$\boldsymbol{\theta}$}_{t-1}$:

$\displaystyle\text{$\boldsymbol{\theta}$}_{t+1}\leftarrow\text{$\boldsymbol{\theta}$}_{t}-\alpha\widehat{\nabla}_{\mathbf{\theta}}\bar{\ell}(\text{$\boldsymbol{\theta}$}_{t})+\gamma(\text{$\boldsymbol{\theta}$}_{t}-\text{$\boldsymbol{\theta}$}_{t-1})$
(75)

with a fixed momentum coefficient $\gamma>0$. Why this technique can be
beneficial when using noisy gradient is more transparent when we rewrite Eq.
75 as

$\text{$\boldsymbol{\theta}$}_{t+1}\leftarrow\text{$\boldsymbol{\theta}$}_{t}-\alpha\sum_{k=0}^{t}\gamma^{k}\widehat{\nabla}_{\mathbf{\theta}}\bar{\ell}(\text{$\boldsymbol{\theta}$}_{t-k})+\text{initial
conditions}.$ (76)

Hence, the effective gradient is an average of previous gradients with
decaying weights. This will smooth out and reduce the variance of the
stochastic gradient estimates.

Adaptive-learning rate algorithms, such as RMSprop and Adam, employ a variant
of the classical momentum method where exponential-smoothing is applied to the
gradient
$\text{$\boldsymbol{u}$}_{t+1}\leftarrow\gamma\text{$\boldsymbol{u}$}_{t}+(1-\gamma)\widehat{\nabla}_{\mathbf{\theta}}\bar{\ell}(\text{$\boldsymbol{\theta}$}_{t})$
which is used in the update while keeping the scaling unchanged [Graves, 2013,
Kingma and Ba, 2015]. This gives the update

$\displaystyle\text{$\boldsymbol{\theta}$}_{t+1}\leftarrow\text{$\boldsymbol{\theta}$}_{t}-\alpha\frac{1}{\sqrt{\text{$\boldsymbol{s}$}_{t+1}}+c\text{$\boldsymbol{1}$}}\circ\text{$\boldsymbol{u}$}_{t+1}.$
(77)

We can express this similar to Eq. 75, as [Wilson et al., 2017]

$\displaystyle\text{$\boldsymbol{\theta}$}_{t+1}$
$\displaystyle\leftarrow\text{$\boldsymbol{\theta}$}_{t}-\alpha(1-\gamma)\frac{1}{\sqrt{\text{$\boldsymbol{s}$}_{t+1}}+c\text{$\boldsymbol{1}$}}\circ\left[\widehat{\nabla}_{\mathbf{\theta}}\bar{\ell}(\text{$\boldsymbol{\theta}$}_{t})\right]+\gamma\frac{\sqrt{\text{$\boldsymbol{s}$}_{t}}+c\text{$\boldsymbol{1}$}}{\sqrt{\text{$\boldsymbol{s}$}_{t+1}}+c\text{$\boldsymbol{1}$}}\circ(\text{$\boldsymbol{\theta}$}_{t}-\text{$\boldsymbol{\theta}$}_{t-1}),$
(78)

showing a dynamic scaling of the momentum depending (essentially) on the ratio
of $\text{$\boldsymbol{s}$}_{t+1}$ and $\text{$\boldsymbol{s}$}_{t}$. Although
this adaptive scaling is not counter-intuitive, it is a result of the somewhat
arbitrary choices made. Can we justify this from the Bayesian principles, and
can derive the corresponding BLR? The problem we face is provoked by the use
of stochastic gradient approximations and the principles simply state that we
should compute exact gradients instead. This might appear as a deficiency in
our main argument but using our free will, we can resolves this issue in other
ways.

It is our view that the natural way to include a momentum term in the BLR, is
to this within the mirror descent framework in Sec. 2. This will result in an
update where the momentum term obey the geometry of the posterior
approximation. We propose to augment Eq. 22 with a momentum term

$\displaystyle\text{$\boldsymbol{\mu}$}_{t+1}\leftarrow\arg\min_{\boldsymbol{\mu}\in\mathcal{M}}\,\,\langle\nabla_{\boldsymbol{\mu}}\mathcal{L}(q_{t}),\text{$\boldsymbol{\mu}$}\rangle+\frac{1+\gamma_{t}}{\rho_{t}}\mathbb{D}_{A^{*}}(\text{$\boldsymbol{\mu}$}\|\text{$\boldsymbol{\mu}$}_{t})-\frac{\gamma_{t}}{\rho_{t}}\mathbb{D}_{A^{*}}(\text{$\boldsymbol{\mu}$}\|\text{$\boldsymbol{\mu}$}_{t-1})$
(79)

The last term penalizes the proximity to $\text{$\boldsymbol{\mu}$}_{t-1}$
where the proximity is measured according to the KL divergence, following the
suggestion by Khan et al. [2018] we may interpret this as the natural momentum
term. This gives a (revised) BLR learning rule with momentum,

$\displaystyle\text{$\boldsymbol{\lambda}$}_{t+1}\leftarrow\text{$\boldsymbol{\lambda}$}_{t}-\rho_{t}\widetilde{\nabla}_{\text{$\boldsymbol{\lambda}$}}\left[\mathbb{E}_{q_{t}}\left(\bar{\ell}(\text{$\boldsymbol{\theta}$})\right)-\mathcal{H}(q_{t})\right]+\gamma_{t}(\text{$\boldsymbol{\lambda}$}_{t}-\text{$\boldsymbol{\lambda}$}_{t-1}).$
(80)

This update will recover Eq. 75 and an alternative to Eq. 78. Choosing a
Gaussian candidate distribution $\text{${\cal
N}$}(\text{$\boldsymbol{\theta}$}|\text{$\boldsymbol{m}$},\text{$\boldsymbol{I}$})$
(the result is invariant to a fixed covariance matrix), we recover Eq. 75
after a derivation similar as in Sec. 1.3.1. Choosing the candidate
distribution $\text{${\cal
N}$}(\text{$\boldsymbol{\theta}$}|\text{$\boldsymbol{m}$},\text{{diag}}(\text{$\boldsymbol{s}$})^{-1})$,
follow again previous derivations (Newton’s method in Sec. 1.3.2) with mini-
batch gradients, and result in the following update

$\begin{split}\text{$\boldsymbol{\theta}$}_{t+1}&\leftarrow\text{$\boldsymbol{\theta}$}_{t}-\rho_{t}\frac{1}{\text{$\boldsymbol{s}$}_{t+1}}\circ\left[\widehat{\nabla}_{\mathbf{\theta}}\bar{\ell}(\text{$\boldsymbol{\theta}$}_{t})\right]+\frac{1}{\text{$\boldsymbol{s}$}_{t+1}}\circ(\text{$\boldsymbol{s}$}_{t}\text{$\boldsymbol{\theta}$}_{t}-\text{$\boldsymbol{s}$}_{t-1}\text{$\boldsymbol{\theta}$}_{t-1}),\\\
\text{$\boldsymbol{s}$}_{t+1}&\leftarrow(1-\rho_{t})\text{$\boldsymbol{s}$}_{t}+\rho_{t}\,\,\text{{diag}}\left[\widehat{\nabla}_{\mathbf{\theta}\mathbf{\theta}}^{2}\bar{\ell}(\text{$\boldsymbol{\theta}$}_{t})\right]+\gamma_{t}(\text{$\boldsymbol{s}$}_{t}-\text{$\boldsymbol{s}$}_{t-1}).\end{split}$
(81)

Assuming $\text{$\boldsymbol{s}$}_{t}\approx\text{$\boldsymbol{s}$}_{t-1}$ the
last terms in the above two equations simplifies, and then by replacing
$\text{$\boldsymbol{s}$}_{t}$ by $\sqrt{s}_{t}$ and adding a constant $c$, we
can recover Eq. 78. This momentum-version of Eq. 37, not only justifies the
use of scaling vectors for the momentum, but also to the use of exponential-
smoothing for the scaling vectors itself.

##  Appendix G Dropout from the BLR

Deriving updates for $\text{$\boldsymbol{m}$}_{jl}$ and
$\text{$\boldsymbol{S}$}_{jl}$ with dropout is similar to the Newton variant
derived for mixture of Gaussian discussed in App. E. With dropout there are
only two components where one of the component has fixed parameters, hence one
set of natural and expectation parameters per a weight vector
$\text{$\boldsymbol{\theta}$}_{jl}$ for the $j$’th unit in $l$’th layer,

$\begin{array}[]{ll}\begin{array}[]{ll}\text{$\boldsymbol{\lambda}$}_{jl}^{(1)}&=\text{$\boldsymbol{S}$}_{jl}\text{$\boldsymbol{m}$}_{jl},\\\
\text{$\boldsymbol{\lambda}$}^{(2)}_{jl}&=-\text{$\frac{1}{2}$}\text{$\boldsymbol{S}$}_{jl},\end{array}\begin{array}[]{ll}\quad\text{$\boldsymbol{\mu}$}_{jl}^{(1)}&=\mathbb{E}_{q}\left[1_{[z_{jl}=1]}\text{$\boldsymbol{\theta}$}_{jl}\right]=\pi_{1}\text{$\boldsymbol{m}$}_{jl},\\\
\quad\text{$\boldsymbol{\mu}$}_{jl}^{(2)}&=\mathbb{E}_{q}\left[1_{[z_{jl}=1]}\text{$\boldsymbol{\theta}$}_{jl}\text{$\boldsymbol{\theta}$}_{jl}^{T}\right]=\pi_{1}\left(\text{$\boldsymbol{S}$}_{jl}^{-1}+\text{$\boldsymbol{m}$}_{jl}\text{$\boldsymbol{m}$}_{jl}^{T}\right),\end{array}\end{array}$

We can reuse the update Eqs. 67 and 68 from App. E

$\displaystyle\text{$\boldsymbol{m}$}_{jl,t+1}$
$\displaystyle\leftarrow\text{$\boldsymbol{m}$}_{jl,t}-\frac{\rho}{\pi_{1}}\text{$\boldsymbol{S}$}_{jl,t+1}^{-1}\nabla_{\mathbf{m}_{jl}}\left\\{\mathbb{E}_{q_{t}}\left[\bar{\ell}(\text{$\boldsymbol{\theta}$})\right]+\mathbb{E}_{q_{t}}\left[\log
q(\text{$\boldsymbol{\theta}$}_{jl})\right]\right\\},$ (82)
$\displaystyle\text{$\boldsymbol{S}$}_{jl,t+1}$
$\displaystyle\leftarrow\text{$\boldsymbol{S}$}_{jl,t+1}+\frac{2\rho}{\pi_{1}}\nabla_{\mathbf{S}_{jl}^{-1}}\left\\{\mathbb{E}_{q_{t}}\left[\bar{\ell}(\text{$\boldsymbol{\theta}$})\right]+\mathbb{E}_{q_{t}}\left[\log
q(\text{$\boldsymbol{\theta}$}_{jl})\right]\right\\}.$ (83)

We have moved the expectation inside the gradient since the entropy term only
depends on $\text{$\boldsymbol{\theta}$}_{jl}$. The gradient of the loss
$\bar{\ell}(\text{$\boldsymbol{\theta}$})$ may depend on the whole
$\boldsymbol{\theta}$, however.

By using appropriate methods to compute gradients, we can recover the dropout
method. Specifically, for the loss term, we use the reparameterization trick
[Kingma and Welling, 2013] and approximate the gradients at the samples from
$q(\text{$\boldsymbol{\theta}$}_{jl})$,

$\text{$\boldsymbol{\theta}$}_{jl}=z_{jl}(\text{$\boldsymbol{m}$}_{jl}+\text{$\boldsymbol{S}$}_{jl}^{-1/2}\text{$\boldsymbol{\epsilon}$}_{1,jl})+(1-z_{jl})s_{0}^{-1/2}\text{$\boldsymbol{\epsilon}$}_{2,jl}$

where $z_{jl}$ is a sample from Bernoulli distribution with probability
$\pi_{1}$, and $\text{$\boldsymbol{\epsilon}$}_{1,jl}$ and
$\text{$\boldsymbol{\epsilon}$}_{2,jl}$ are two independent samples from
standard normal distribution. Since weights are deterministic in dropout, we
need to ignore $\text{$\boldsymbol{\epsilon}$}_{1,jl}$ and
$\text{$\boldsymbol{\epsilon}$}_{2,jl}$ to recover dropout from the BLR. The
gradients are then evaluated at the dropout mean
$\widetilde{\text{$\boldsymbol{m}$}}_{jl}=z_{jl}\text{$\boldsymbol{m}$}_{jl}$
(the delta method) which leads to the following gradient approximations:

$\begin{split}\nabla_{\mathbf{m}_{jl}}\mathbb{E}_{q_{t}}[\bar{\ell}(\text{$\boldsymbol{\theta}$})]&\approx\left.\nabla_{\text{$\boldsymbol{\theta}$}_{jl}}\bar{\ell}(\tilde{\text{$\boldsymbol{\theta}$}})\right|_{\tilde{\text{$\boldsymbol{\theta}$}}=\widetilde{\text{$\boldsymbol{m}$}}_{t}},\\\
2\nabla_{\mathbf{S}_{jl}^{-1}}\mathbb{E}_{q_{t}}\left[\bar{\ell}(\text{$\boldsymbol{\theta}$})\right]&=\nabla_{\mathbf{m}_{jl}\mathbf{m}_{jl}^{\top}}^{2}\mathbb{E}_{q_{t}}\left[\bar{\ell}(\text{$\boldsymbol{\theta}$})\right]\approx\left.\nabla_{\tilde{\text{$\boldsymbol{\theta}$}}_{jl}\tilde{\text{$\boldsymbol{\theta}$}}_{jl}^{\top}}^{2}\bar{\ell}(\tilde{\text{$\boldsymbol{\theta}$}})\right|_{\tilde{\text{$\boldsymbol{\theta}$}}=\widetilde{\text{$\boldsymbol{m}$}}_{t}}.\end{split}$
(84)

We will use these gradients for the loss term in Eqs. 82 and 83.

Gradients of $\log q(\text{$\boldsymbol{\theta}$}_{jl})$ only depends on the
variables involved in $q(\text{$\boldsymbol{\theta}$}_{jl})$, and we can use
the delta method similar to Eq. 33 in Sec. 3. We make the assumption that the
two mixture components are “not close” which is reasonable with dropout. With
this assumption then the gradient respect to $\text{$\boldsymbol{m}$}_{jl}$ is
approximately zero

$\displaystyle\nabla_{\mathbf{m}_{jl}}\mathbb{E}_{q}\left[\log
q(\text{$\boldsymbol{\theta}$}_{jl})\right]$
$\displaystyle=\nabla_{\mathbf{m}_{jl}}\int
d\text{$\boldsymbol{\theta}$}_{jl}\log
q(\text{$\boldsymbol{\theta}$}_{jl})\left[\pi_{1}\text{${\cal
N}$}(\text{$\boldsymbol{\theta}$}_{jl}|\text{$\boldsymbol{m}$}_{jl},\text{$\boldsymbol{S}$}_{jl}^{-1})+(1-\pi_{1})\text{${\cal
N}$}(\text{$\boldsymbol{\theta}$}_{jl}|\text{$\boldsymbol{0}$},s_{0}^{-1}\text{$\boldsymbol{I}$}_{n_{l}})\right]$
$\displaystyle=\pi_{1}\nabla_{\mathbf{m}_{jl}}\int
d\text{$\boldsymbol{\theta}$}_{jl}\log
q(\text{$\boldsymbol{\theta}$}_{jl})\text{${\cal
N}$}(\text{$\boldsymbol{\theta}$}_{jl}|\text{$\boldsymbol{m}$}_{jl},\text{$\boldsymbol{S}$}_{jl}^{-1})$
$\displaystyle=\pi_{1}\mathbb{E}_{\mathcal{\mathcal{N}}(\text{\text{$\boldsymbol{\theta}$}}_{jl}|\mathbf{m}_{jl,t},\mathbf{S}_{jl,t}^{-1})}\left[\nabla_{\mathbf{\theta}_{jl}}\log
q(\text{$\boldsymbol{\theta}$}_{jl})\right]\text{ (Using Bonnet's Theorem)}$
$\displaystyle\approx\pi_{1}\nabla_{\mathbf{m}_{jl}}\log
q(\text{$\boldsymbol{m}$}_{jl,t})\text{ (Using the Delta approximation) }$
$\displaystyle\approx 0.$ (85)

For the gradient with respect to $\text{$\boldsymbol{S}$}^{-1}$ we get

$\displaystyle 2\nabla_{\mathbf{S}_{jl}^{-1}}\mathbb{E}_{q}\left[\log
q(\text{$\boldsymbol{\theta}$}_{jl})\right]$
$\displaystyle=\nabla_{\mathbf{m}_{jl}\mathbf{m}_{jl}^{\top}}^{2}\mathbb{E}_{q}\left[\log
q(\text{$\boldsymbol{\theta}$}_{jl})\right]$
$\displaystyle=\nabla_{\mathbf{m}_{jl}}\left[\pi_{1}\mathbb{E}_{\mathcal{\mathcal{N}}(\text{\text{$\boldsymbol{\theta}$}}_{jl}|\mathbf{m}_{jl,t},\mathbf{S}_{jl,t}^{-1})}\left[\nabla_{\mathbf{\theta}_{jl}}\log
q(\text{$\boldsymbol{\theta}$}_{jl})\right]\right]\text{ (using first
derivative wrt $\mathbf{m}_{jl}$) }$
$\displaystyle=\pi_{1}\mathbb{E}_{\mathcal{\mathcal{N}}(\mathbf{\theta_{jl}}|\mathbf{m}_{jl,t},\mathbf{S}_{jl,t}^{-1})}\left[\nabla_{\mathbf{\theta}_{jl}\mathbf{\theta}_{jl}^{\top}}^{2}\log
q(\text{$\boldsymbol{\theta}$}_{jl})\right]\text{ (Using Bonnet's theorem) }$
$\displaystyle\approx\pi_{1}\nabla_{\mathbf{m}_{jl}\mathbf{m}_{jl}^{\top}}^{2}\log
q(\text{$\boldsymbol{m}$}_{jl,t})\text{ (Using Delta approximation) }$
$\displaystyle\approx-\pi_{1}\text{$\boldsymbol{S}$}_{jl,t},$ (86)

Using the gradient approximations Eqs. 84, 85 and 86 in Eqs. 82 and 83, we get
the

$\displaystyle\text{$\boldsymbol{m}$}_{jl,t+1}$
$\displaystyle\leftarrow\text{$\boldsymbol{m}$}_{jl,t}-\frac{\rho}{\pi_{1}}\text{$\boldsymbol{S}$}_{jl,t+1}^{-1}\nabla_{\mathbf{m}_{jl}}\left\\{\nabla_{\mathbf{m}_{jl}}\bar{\ell}(\widetilde{\text{$\boldsymbol{m}$}}_{t})+0\right\\},$
$\displaystyle\text{$\boldsymbol{S}$}_{jl,t+1}$
$\displaystyle\leftarrow\text{$\boldsymbol{S}$}_{jl,t+1}+\frac{\rho}{\pi_{1}}\nabla_{\mathbf{S}_{jl}^{-1}}\left\\{\nabla_{\mathbf{m}_{jl}\mathbf{m}_{jl}^{\top}}^{2}\bar{\ell}(\widetilde{\text{$\boldsymbol{m}$}}_{t})-\pi_{1}\text{$\boldsymbol{S}$}_{jl,t}\right\\}.$

which reduces to the updates Eqs. 42 and 43.


Tahereh Ramezani, Ernst Paunzen, Caiyun Xia, Kateřina Pivonková, and Prapti
Mondal 11institutetext: Department of Theoretical Physics and Astrophysics,
Masaryk University, Kotlářská 2, CZ-611 37 Brno, Czechia
11email<EMAIL_ADDRESS>Submitted on 11 Novemver 2023; Accepted on 05
April 2024)

# Estimation of radial velocities of BHB stars

Tahereh Ramezani 11 Ernst Paunzen 11 Caiyun Xia 11 Kateřina Pivonková 11 and
Prapti Mondal 11

###### Abstract

We studied blue horizontal branch stars (BHBs), and calculated their radial
velocities. Spectra of these stars have been obtained with moderate signal-to-
noise ratio for five blue horizontal-branch stars using the 2 metre telescope
and Echelle Spectrograph in Ondřejov observatory, Czech republic.

###### keywords:

Blue Horizontal Branch Stars- Radial Velocity- B type star.

## Introduction

Blue horizontal branch (BHB) stars are hot helium burning stars of the end of
blue edge of horizontal branch. BHBs are quite often used to study the
properties of galactic halo (Utkin & Dambis 2020; Kinman et al. 2000; Vickers
et al. 2021). One can think of these stars due to their relatively constant
absolute magnitude, with value around zero, which means we are able to predict
their distances with high precision, thus also the distance to the Galactic
centre (Utkin & Dambis 2020). In the catalogue of Vickers et al. (2021), the
absolute magnitude of almost 14 000 stars was predicted to be $M_{V}$
$\approx{0.31}$ mag.

Subdwarf stars of spectral type B (sdBs) are extremely hot stars found at the
end of the horizontal branch, related to the blue horizontal branch (BHB)
stars, which are quite often used to trace the galactic halo (Kinman et al.
2000). Many scenarios to explain the evolution of sdBs have been proposed
during last decades. Some such scenarios are strong mass loss rate occurring
in a close binary system (Bailer-Jones et al. 2018), mergering of a close
binary white dwarfs (Bernhard et al. 2015) or a merging of a red giant and a
low mass main sequence star during a common envelope phase leading to
formation of a rapidly rotating hot subdwarf star proposed by Politano et al.
(2008).

The stars of type sdB show similar spectral characteristics to the main
sequence stars of the same spectral type. A small difference between sdBs and
main sequence stars of the same spectral type is that sdB stars have weaker
helium lines, are less luminous and have stronger hydrogen Balmer lines caused
by their high surface gravity $\left({\rm log}~{}g=5.0-6.0\right)$, while at
the same time they are quite compact objects $\left(R_{\rm sdB}=0.1-0.3~{}{\rm
R_{\odot}}\right)$ with very thin hydrogen envelope and helium core (Geier &
Heber 2012). Irrgang et al. (2021) assumed a typical mass for BHB stars as $\
0.5\pm 0.1~{}{\rm M_{\odot}}$.

According to Philip et al. (1990), almost no rotational broadening in high
resolution spectra is visible and an appropriate Strömgren $\left(b-y\right)$
and $\ {\rm c}_{1}$ criteria must be fulfilled for BHB stars. They found out a
high rotation velocity for HB stars and low rotation for hotter stars.

## 1\. Observations

The observations were obtained during a few nights in September 2021 (8th and
10th) with the Perek 2 meter telescope located in Ondřejov, Czech republic.

The Perek 2 meter telescope has a resolving power of R$\sim$13000 at
$H\alpha$. Additionally, the telescope is equipped with two instruments, a
single-order spectrograph and an Echelle spectrograph.

Those objects have been observed spectroscopically with the Echelle
spectrograph connected to the Perek 2 meter telescope. Each spectrum was
processed using oesred.cl script designed for work in IRAF by Cabezas et al.
(2023) . Estimation of radial velocities was done using the SpaS software
designed for plotting and following analysis of 1D spectra.

Spectra of all five BHB stars investigated are shown in the Figure 1.

Figure 1: Spectra of the five BHB stars studied.

The journal of observations of our stars is given in Table 1.

Table 1: List of stars with nights of observation (first epoch and second epoch), number of observations per star, and the exposure time of each spectrum. star ID | first epoch | second epoch | number of observations | exposure time [s]
---|---|---|---|---
TYC3655-1179-1 | 08.09.2021 | 10.09.2021 | 2 | 5200
HD 8376 | 08.09.2021 | 10.09.2021 | 2 | 3600
BD+38574b | 08.09.2021 | 10.09.2021 | 2 | 5400
HD 167105 | 08.09.2021 | 10.09.2021 | 2 | 3600
HD 203563 | 08.09.2021 | 10.09.2021 | 2 | 1039

## 2\. Data analysis

We got the reddening by Schlafly & Finkbeiner (2011), distance from Bailer-
Jones et al. (2021), and coordinates from Gaia EDR3, (Gaia Collaboration 2020)
for each object and put them in the Table 2.

We estimated radial velocities for the first epoch and the second epoch with
their standard deviations for each epoch and put them in Table 3.

Table 2: List of stars for which we got reddening by Schlafly & Finkbeiner, (2011), and distance from Bailer-Jones et al. (2021). We got Longitude and Latitude values from Gaia EDR3 (Gaia Collaboration 2020). star ID | RA | DEC | Longitude | Latitude | E(B-V) [mag] | Distance [pc]
---|---|---|---|---|---|---
TYC3655-1179-1 | 00 44 43.32 | +53 10 39.22 | 121.911 | -9.681 | 0.156 | 739 $\pm$ 9
HD 8376 | 01 23 28.28 | +31 47 12.26 | 130.839 | -30.593 | 0.045 | 624 $\pm$ 6
BD+38574b | 02 51 44.00 | +38 53 47.95 | 147.178 | -18.246 | 0.052 | 652 $\pm$ 6
HD 167105 | 18 11 06.30 | +50 47 32.41 | 78.694 | +26.861 | 0.062 | 405 $\pm$ 2
HD 203563 | 21 22 59.96 | +02 14 37.29 | 54.686 | -31.892 | 0.026 | 441 $\pm$ 5
Table 3: List of stars for which we estimated radial velocities for the first epoch (second column) and the second epoch (fourth column) with their standard deviations for each epoch. star ID | $v_{\rm rad}$ [km/s] | stddev | $v_{\rm rad}$ [km/s] | stddev
---|---|---|---|---
TYC3655-1179-1 |  $-$189 | 0.10 |  $-$188 | 0.18
HD 8376 |  $+$144 | 0.08 |  $+$145 | 0.23
BD+38574b |  $-$101 | 0.08 |  $-$101 | 0.32
HD 167105 |  $-$157 | 0.03 |  $-$169 | 0.03
HD 203563 |  $-$101 | 0.05 |  $-$96 | 0.10

The Diffuse Interstellar Bands (DIBs) can be as weak as a couple of percent,
so this is simply not enough to detect them with certainty. ESO archive has a
spectrum of HD 203563, there it is an intensity of 5780A DIB of maybe 3%.

We found our targets using the TOPCAT software. Then we had a night
spectroscopy observation where we observed our targets all night and wrote
down the start time, which was 21:00, the end time and the exposure time. Then
we reduced our data in IRAF by the package oesred. Then we changed the format
from .fit to .spec. After that we found the radial velocity of Hβ, Hγ and Hδ
and fitted them, which they had convenient fits; also calculated signal to
noise which are in the Figures 2 to 6.

Figure 2: The dependence of S/N to wavelength for two different epochs of the
same object TYC 3655-1179-1. X axis represents intervals of wavelength: 1.
interval is 4000-5000 Å, 2. interval is 5000-6000 Å, 3. interval is 6000-7000
Å, 4. interval is 7000-8000 Å. Figure 3: The dependence of S/N to wavelength
for two different epochs of the same object HD 8376. X axis represents
intervals of wavelength: 1. interval is 4000-5000 Å, 2. interval is 5000-6000
Å, 3. interval is 6000-7000 Å, 4. interval is 7000-8000 Å. Figure 4: The
dependence of S/N to wavelength for two different epochs of the same object
BD+38574b. X axis represents intervals of wavelength: 1. interval is 4000-5000
Å, 2. interval is 5000-6000 Å, 3. interval is 6000-7000 Å, 4. interval is
7000-8000 Å. Figure 5: The dependence of S/N to wavelength for two different
epochs of the same object HD 167105. X axis represents intervals of
wavelength: 1. interval is 4000-5000 Å, 2. interval is 5000-6000 Å, 3.
interval is 6000-7000 Å, 4. interval is 7000-8000 Å. Figure 6: The dependence
of S/N to wavelength for two different epochs of the same object HD 203563. X
axis represents intervals of wavelength: 1. interval is 4000-5000 Å, 2.
interval is 5000-6000 Å, 3. interval is 6000-7000 Å, 4. interval is 7000-8000
Å. Figure 7: Hertzsprung-Russell diagram of the targets together with
isochrones from Bressan et al. (2012).

We searched in the catalogues: Anders et al. (2022), Paegert et al. (2021),
Stassun et al. (2019), McDonald et al. (2017), and from the parameters’ values
and use of this formula:

$log(\rm g/{\rm g_{\odot}})$ = $log(\rm M/{\rm M_{\odot}})$ \+ $4log(\rm
T_{\rm eff}$/${\rm T_{{\rm eff}_{\odot}}})$ \- $log(\rm L/{\rm L_{\odot}})$
(1)

Netopil et al. (2017), and found log$T_{\rm eff}$ and log$Lum$, and plotted
the isochrones of the stars which are in the Figure 7.

## 3\. Conclusion

In our work, we studied five blue horizontal branch stars (BHBs), and
estimated their radial velocity using the Echelle spectrograph mounted on the
2 meter telescope situated in Ondřejov observatory, Czech Academy of Sciences.
We give estimated values of the radial velocities of these stars in Table 3.

The Radial velocity of the HD 167105 which was measured by Gontcharov, (2006)
was -172.40 km/s, also for BD+38574b which was measured by Jönsson et al.
(2020) was -94.749496 km/s. We see that our measurements are similar to the
corresponding values of their measurements.

We used spectroscopic measurements to determine the stellar parameters of the
BHBs. Radial velocities were measured from spectra taken by 2 meter telescope
during the two nights in September 2021. In our samples, we considered only
those BHB stars for which the effective temperature$\ (T_{\rm eff})$ is less
than $\sim$10 000 K.

## Acknowledgements

This work was supported by Masaryk University, Faculty of Science, and Potsdam
university. We would like to express our special thanks of gratitude to Jiri
Kubat, who gave us the golden opportunity to do this wonderful project, also
another supporter, Brankica Kubatova in Ondřejov institute. From Astronomical
Institute of the Czech Academy of Sciences. Thanks to our lecturers, Harry
Dawson and Max Pritzkuliet from Potsdam University. We would like to thanks to
our colleagues Martin Piecka at the Vienna University, Gabriel Szász and Jan
Janik at the Masaryk University for their guidance.

## References

  * [] Anders, F., Khalatyan, A., Queiroz, A. B. A., Chiappini, C., Ardevol, J., Casamiquela, L., Figueras, F., Jimenez-Arranz, O., Jordi, C., Monguio, M., Romero-Gomez, M., Altamirano, D., Antoja, T., Assaad, R., Cantat-Gaudin, T., Castro-Ginard, A., Enke, H., Girardi, L., Guiglion, G., Khan, S., Luri, X., Miglio, A., Minchev, I., Ramos, P., Santiago, B. X., Steinmetz, M., 2022, A&A, 658, 27
  * [] Bailer-Jones, C. A. L., Rybizki, J., Fouesneau, M., Demleitner, M., Andrae, R., 2021, AJ, 161, 24
  * [] Bailer-Jones, C. A. L., Rybizki, J., Fouesneau, M., Mantelet, G., Andrae, R., 2018, AJ, 156, 58
  * [] Bernhard, K., Hümmerich, S., Otero, S., Paunzen, E., 2015, A&A, 581, A138
  * [] Bressan, A., Marigo, P., Girardi, L., Salasnich, B., Dal Cero, C., Rubele, S., Nanni, A., 2012, MNRAS, 427, 127-145
  * [] Cabezas, M., Šlechta, M., Škoda, P., Kubátová, B., 2023, Zenodo
  * [] Gaia Collaboration, 2020, A&A
  * [] Geier, S., Heber, U., 2012, A&A, 543, A149
  * [] Gontcharov, G. A., 2006, Astronomy Letters, 32, 759-771
  * [] Irrgang, A., Geier, S., Heber, U., Kupfer, T., El-Badry, K., Bloemen, S., 2021, A&A, 650, A102
  * [] Jönsson, H., Holtzman, Jon A., Allende Prieto, C., Cunha, K., García-Hernández, D. A., Hasselquist, S., Masseron, T., Osorio, Y., Shetrone, M., Smith, V., Stringfellow, Guy S., Bizyaev, D., Edvardsson, B., Majewski, Steven R., Mészáros, S., Souto, D., Zamora, O., Beaton, Rachael L., Bovy, J., Donor, J., Pinsonneault, Marc H., Poovelil, Vijith J., Sobeck, J., 2020, AJ, 160
  * [] Kinman, T., Castelli, F., Cacciari, C., Bragaglia, A., Harmer, D., Valdes, F., 2000, A&A, 364, 102
  * [] McDonald, I., Zijlstra, A. A., Watson, R. A., 2017, MNRAS, 471, 770-791
  * [] Netopil, M., Paunzen, E., Hümmerich, S., Bernhard, K., 2017, MNRAS, 468, 2745-2756
  * [] Paegert, M., Stassun, Keivan G., Collins, Karen A., Pepper, J., Torres, G., Jenkins, J., Twicken, Joseph D., Latham, David W., 2021, arXiv:2108.04778, 11
  * [] Philip, A. G. D., Hayes, D. S., Adelman, S. J., 1990, PASP, 102, 649
  * [] Politano, M., Taam, R. E., van der Sluys, M., Willems, B., 2008, ApJ, 687, L99
  * [] Schlafly, Edward F., Finkbeiner, Douglas P., 2011, ApJ, 737, 13
  * [] Stassun, Keivan G., Oelkers, Ryan J., Paegert, M., Torres, G., Pepper, J., De Lee, N., Collins, K., Latham, David W., Muirhead, Philip S., Chittidi, J., Rojas-Ayala, B., Fleming, Scott W., Rose, Mark E., Tenenbaum, P., Ting, Eric B., Kane, Stephen R., Barclay, T., Bean, Jacob L., Brassuer, C. E., Charbonneau, D., Ge, J., Lissauer, Jack J., Mann, Andrew W., McLean, B., Mullally, S., Narita, N., Plavchan, P., Ricker, George R., Sasselov, D., Seager, S., Sharma, S., Shiao, B., Sozzetti, A., Stello, D., Vanderspek, R., Wallace, G., Winn, Joshua N., 2019, AJ, 158, 21
  * [] Utkin, N. D., Dambis, A. K., 2020, MNRAS, 499, 1058
  * [] Vickers, J. J., Li, Z., Smith, M. C., Shen, J., 2021, ApJ, 912, 32
  * []
  * []
  * []