Learning and inference in latent variable models and theoretical guarantees of sampling algorithms

We analyse a class of algorithms, termed Proximal Interacting Particle Langevin Algorithms (PIPLA), for inference and learning in latent variable models whose joint probability density is non-differentiable. Leveraging proximal Markov chain Monte Carlo (MCMC) techniques and the recently introduced interacting particle Langevin algorithm (IPLA), we propose several variants within the novel proximal IPLA family, tailored to the problem of estimating parameters in a non-differentiable statistical model. We prove non-asymptotic bounds for the parameter estimates produced by multiple algorithms in the strongly log-concave setting and provide comprehensive numerical experiments on various models to demonstrate the effectiveness of the proposed methods. In particular, we demonstrate the utility of the proposed family of algorithms on a toy hierarchical example where our assumptions can be checked, as well as on the problems of sparse Bayesian logistic regression, sparse Bayesian neural network, and sparse matrix completion. Our theory and experiments together show that PIPLA family can be the de facto choice for parameter estimation problems in latent variable models for non-differentiable models.

On the other hand, generating samples from complex unnormalised probability distributions is an important problem in statistics and machine learning. One of the motivations for obtaining accurate samples is being able to estimate expectations of functions, $\mathbb{E}_{\pi(x)}[h(x)]\approx 1/N \sum_{i=1}^N h(x_i)$, where $x_i$ are i.i.d. samples from the target distribution $\pi\propto \exp(-V)$. A large number of works in the literature have explored this problem, providing new methods accompanied by theoretical guarantees under fairly general assumptions. In particular, score-based methods have become a very popular option for sampling. However, they only use local gradient information, which presents a challenge when sampling multi-modal distributions with disconnected...