Seminar on Advances in Probabilistic Machine Learning

Abstract: State space approximations and pseudo point approximations can be combined in a principled manner to yield scalable approximate inference algorithms for sums of separable Gaussian processes. In this talk, I will: 1. show how this combination can be performed for variational pseudo point approximations via a simple conditional independence result, 2. discuss how existing exact inference algorithms for state space models can be re-purposed for approximate inference, 3. interpret existing related work in light of our work, and 4. briefly discuss some experimental results in a spatio-temporal context. For more info, please see our recent UAI paper.

Abstract: Gaussian Processes are a tool of choice for modelling functions with uncertainties. However, the inference is only tractable analytically for the classical case of regression with Gaussian noise since all other likelihoods are not conjugate with the Gaussian prior. In this talk, I will show how one can transform a large class of likelihoods into conditional conjugate distributions by augmenting them with latent variables. These augmented models have the advantage that, while the posterior inference is still not fully analytic, the full conditionals are! Consequently, one can work easily (and efficiently!) with algorithms like Gibbs sampling or Coordinate Ascent VI (CAVI) and outperform existing inference methods.

Abstract: Scientists and engineers are often interested in learning the number of subpopulations (or components) present in a data set. A common suggestion is to use a finite mixture model (FMM) with a prior on the number of components. Past work has shown the resulting FMM component-count posterior is consistent; that is, the posterior concentrates on the true, generating number of components. But consistency requires the assumption that the component likelihoods are perfectly specified, which is unrealistic in practice. In this paper, we add rigour to data-analysis folk wisdom by proving that under even the slightest model misspecification, the FMM component-count posterior diverges: the posterior probability of any particular finite number of components converges to 0 in the limit of infinite data. Contrary to intuition, posterior-density consistency is not sufficient to establish this result. We develop novel sufficient conditions that are more realistic and easily checkable than those common in the asymptotics literature. We illustrate the practical consequences of our theory on simulated and real data.

Abstract: Solving decision-making problems in a variety of domains such as healthcare or operations research requires experimentation. By performing interventions, one can understand how a system behaves when an action is taken and thus infer the cause-effect relationships of a phenomenon. Experiments are usually expensive, time-consuming, and may present ethical issues. Therefore, researchers generally have to trade-off cost, time, and other practical considerations to decide which experiments to conduct in order to learn about a system. In this talk, I will present two methodologies that, by linking causal inference, experimental design and Gaussian process (GP) modelling, allow to efficiently learn the causal effects in a graph and identify the optimal intervention to perform. Firstly, I will show how to construct a multi-task causal GP model, the DAG-GP model, which captures the non-trivial correlation structure across different experimental outputs. By sharing experimental information, the DAG-GP model accurately estimates the causal effects in a variety of experimental settings while enabling proper uncertainty quantification. I will then demonstrate how this model, and more generally GP models, can be used within decision-making algorithm to choose experiments to perform. Particularly, I will introduce the Causal Bayesian Optimization algorithm, and I will show how incorporating the knowledge of the causal graph in Bayesian Optimization improves the ability to reason about optimal decision making while decreasing the optimization cost and avoiding suboptimal solutions.

Abstract: In this talk, I will discuss recent work on developing generative models for efficient sampling and inference by incorporating inductive biases in the form of equivariances. I will begin by introducing the Equivariant Stein Variational Gradient Descent algorithm – an equivariant sampling method based on Stein’s identity for sampling from densities with symmetries. Equivariant SVGD explicitly incorporates symmetry information in a density through equivariant kernels, which makes the resultant sampler efficient both in terms of sample complexity and the quality of generated samples. Subsequently, I will demonstrate the use of Equivariant SVGD by defining equivariant energy-based models to model invariant densities that are learned using contrastive divergence. I will then discuss the applications of these equivariant energy models for modelling joint densities in regression and classification tasks for image datasets, many-body particle systems and molecular structure generation. Finally, if time permits, I will touch on methods for sampling using diffusion models and neural transport augmented Monte Carlo methods for more efficient sampling in discrete spaces with applications in denoising, Bayesian posterior sampling, and training light-weight Bayesian quantised neural nets.

Abstract: Automated decision-making systems are increasingly being deployed in areas with personal and societal impacts: from personalized ads to medical diagnosis and criminal justice. This led to growing interest and need for trustworthy AI and ML systems - that is, models that are robust, explainable, fair, and so on. It is important to note that these guarantees only hold with respect to a certain model of the world, with inherent uncertainties. In this talk, I will present how probabilistic modelling and inference, by incorporating a distribution, offer a principled way to handle different kinds of uncertainties when reasoning about decision-making system behaviours. For example, labels in training data may be biased; I will show that probabilistic circuits, a class of tractable probabilistic models (TPMs), can be effective in enforcing and auditing fairness properties by explicitly modelling a latent unbiased label. Another common source of uncertainty is missing values at prediction time, which also leads to fairness and robustness queries that account for this to be computationally hard inference tasks. I will also demonstrate how TPMs can again tractably answer these complex queries.

Abstract: Probabilistic graphical models are flexible models for representing complex high-dimensional distributions and for incorporating domain knowledge in an intuitive and expressive way. For making probabilistic inference, one often relies on recursive message passing methods. While these methods are efficient for restricted model classes (e.g., for trees), they only serve as approximation methods for more complex models. In this talk, I will show how we can enhance the performance of message passing methods from two opposing angles: i.e., by simplifying the model itself with the utilized inference method in mind and by modifying the inference method with the underlying model in mind. Therefore, I will show how we can advance our understanding of message passing methods by considering them as a dynamic system and by applying tools from system theory. These insights will then suggest various improvements. Recently, we have also complemented message passing methods by neural networks. I will discuss how such hybrid models benefit from the flexibility of neural networks in combination with the implicit underlying model assumptions.

Abstract: Progressively applying Gaussian noise transforms complex data distributions to approximately Gaussian. Reversing this dynamic defines a generative model. When the forward noising process is given by a Stochastic Differential Equation (SDE), Song et al. (2021) demonstrate how the time inhomogeneous drift of the associated reverse-time SDE may be estimated using score-matching. A limitation of this approach is that the forward-time SDE must be run for a sufficiently long time for the final distribution to be approximately Gaussian. In contrast, solving the Schrödinger Bridge problem (SB), i.e. an entropy-regularized optimal transport problem on path spaces, yields diffusions which generate samples from the data distribution in finite time. We present Diffusion SB (DSB), an original approximation of the Iterative Proportional Fitting (IPF) procedure to solve the SB problem, and provide theoretical analysis along with generative modeling experiments. The first DSB iteration recovers the methodology proposed by Song et al. (2021), with the flexibility of using shorter time intervals, as subsequent DSB iterations reduce the discrepancy between the final-time marginal of the forward (resp. backward) SDE with respect to the prior (resp. data) distribution. Beyond generative modeling, DSB offers a widely applicable computational optimal transport tool as the continuous state-space analogue of the popular Sinkhorn algorithm (Cuturi, 2013). Joint work with Valentin De Bortoli, Jeremy Heng and Arnaud Doucet.

Abstract: Topographic generative models can be seen as a class of generative models where the latent variables have an underlying topographic (or spatial) organization which determines their correlation structure. Such structure is widely observed in biological neural networks, however, its computational value is still debated and thus lacks adoption by the deep learning community at large. In this talk, we will describe the statistical motivations behind early topographic generative models like Topographic ICA, and show how such priors can be integrated into modern deep neural networks by introducing the Topographic Variational Autoencoder (TVAE). Further, we will show how topographic representations can be seen as generalized structured representations, and demonstrate how topographic organization over space and time can be leveraged to induce the learning of equivariant sets of features we call capsules. Finally, we will show preliminary results comparing the representations learned by deep TVAEs with FMRI recordings, demonstrating the emergence of localized specialized regions similar to the face area observed in primates.

Abstract: Despite their ubiquitousness in modern data-driven decision-making systems, neural networks are not very well understood. A symptom of this is that network hyperparameters are almost always chosen via cross-validation, an expensive approach that scales poorly in the number of hyperparameters. Additionally, obtaining robust uncertainty estimates for neural network predictions remains an open problem. The probabilistic framework holds the promise of providing both an objective for model selection and reliable uncertainty estimates. However, for the case of neural networks exact probabilistic inference is intractable. This talk introduces the Linearised Laplace approximation for Bayesian deep learning. We examine the assumptions behind linearised Laplace, particularly in conjunction with model selection. We show that these interact poorly with some now-standard features of deep learning—stochastic approximation methods and normalisation layers—and make recommendations for how to better adapt this classic method to the modern setting. We provide Theoretical support of our recommendations and validate them empirically on MLPs, classic CNNs, residual networks with and without normalisation layers, generative autoencoders and transformers. As a case study, we deep dive into Bayesian deep learning methods for tomographic reconstruction. Using linearised Laplace, we construct a probabilistic Deep Image Prior over reconstructed images. Inference in this model allows us to choose U-Net architecture parameters without the need for cross-validation and yields state of the art uncertainty calibration for tomographic reconstruction.

Abstract: Coupling methods have recently been used to compute unbiased estimates of Markov chains Monte Carlo and particle smoothing expectations. However, in most cases, sampling from couplings has a random run time, the variance of which can be infinite. This behaviour, acceptable in distributed computing, is highly problematic in the parallel computing framework. We propose a limited variance coupled rejection sampling method for sampling from couplings of arbitrary distributions. We show how we can modify the coupled rejection method to propose an ensemble of proposals so as to asymptotically recover a maximal coupling while decreasing the total run time of the algorithm. We then discuss the important special case of coupling Gaussian distributions with different means and covariances, and show how the rejection sampling method can be optimised in this case. We then apply the method to sampling from couplings of Gaussian tails, perform coupled Gibbs sampling, couple parallel resampling algorithms in particle filtering, and couple manifold MALA.

Abstract: Well-calibrated predictive uncertainty of neural networks—essentially making them know when they don’t know—is paramount in safety-critical applications. However, deep neural networks are overconfident in the region both far away and near the training data. In our works, we study Bayesian neural networks (BNNs) and their extensions to mitigate this issue. First, we show that being Bayesian, even just at the last layer and in a post-hoc manner, helps mitigate overconfidence in deep ReLU classifiers. Then, we provide a cost-effective Gaussian-process extension to ReLU BNNs that provides a guarantee that ReLU nets will never be overconfident in the region far from the data. Furthermore, we propose two ways of improving the calibration of general BNNs in the out-of-distribution (OOD) regions near the data by (i) training the uncertainty of Laplace approximations and (ii) by leveraging OOD data during training. Finally, we provide a simple library, laplace-torch, to facilitate modern arts of Laplace approximations in deep learning. This library gives users a way to turn a standard pre-trained deep net into a BNN in a cost-efficient manner.

Abstract: Normalizing flows have shown great success as general-purpose density estimators. However, many real-world applications require the use of domain-specific knowledge, which normalizing flows cannot readily incorporate. We propose embedded-model flows (EMF), which alternate general-purpose transformations with structured layers that embed domain-specific inductive biases. These layers are automatically constructed by converting user-specified differentiable probabilistic models into equivalent bijective transformations. We also introduce gated structured layers, which allow bypassing the parts of the models that fail to capture the statistics of the data. We demonstrate that EMFs can be used to induce desirable properties such as multimodality, hierarchical coupling and continuity. Furthermore, we show that EMFs enable a high-performance form of variational inference where the structure of the prior model is embedded in the variational architecture. In our experiments, we show that this approach outperforms state-of-the-art methods in common structured inference problems.

Abstract: Bayesian joint parameter and state inference in non-linear state-space models is a difficult problem due to the often high-dimensional state sequence. Particle Gibbs (PG) is well-suited for solving this type of inference problem but produces correlated samples. In this talk, I describe how the correlation can be reduced by marginalizing out one or more parameters from the state update when conjugacy relations exist between the parameter prior and the complete data likelihood. Deriving the marginalized conjugacy relations is often time-consuming, but probabilistic programming can be employed to automate the process. I also introduce a marginalized PG sampler for multiple time series described by a common state-space model structure, where subsets of the parameters are shared between different models. The spread of mosquito-borne diseases, where some parameters are location-specific, and some are disease-specific, is one example. In theory, it is possible to update all models concurrently, but sequential Monte Carlo becomes inefficient as the number of time series increases. Our suggested marginalized PG sampler instead updates one model at a time, conditioned on the remaining datasets, and can be formulated in a modular fashion that greatly facilitates its implementation.

Abstract: Designing sequences of adaptive experiments to maximise the information gathered about an underlying process is a key challenge in science and engineering. Bayesian Experimental Design (BED) is a powerful mathematical framework for tackling the optimal design problem. Despite the huge potential of obtaining information more quickly and efficiently, the widespread adoption of adaptive BED has been severely limited by the costly computations required at each experiment iteration. In this talk, I’ll present a new method, called Deep Adaptive Design (DAD), that alleviates this problem. DAD marks a critical change from previous BED methods in that it optimises a policy instead of individual designs during the experiment. The policy is parametrised by a neural network, taking as inputs past data and returning the design to use at the next experiment iteration. Using a single pass through the network, DAD enables quick and adaptive design decisions in real time.

Abstract: Despite trends in modern medicine and epidemiological control, the risk for novel outbreaks and previously existing pathogens is currently greater than ever. Indeed, the current outbreak of SARS-CoV-2 has exposed the need for precise, robust, and principled mathematical modelling of disease outbreaks that can perform well with noisy and potentially biased data. To tackle these challenges, I will present a unifying view of modelling infectious diseases that contributes to the new understanding of the spread of the diseases and their epidemiological properties. The unified framework allows flexible probabilistic models that are capable of fitting complex and noisy data from different sources. I will touch upon how the new unified framework, built using Stan (numpyro), has helped us to characterize the initial spread of SARS-CoV-2 and quantify the altered epidemiological characteristics of various ‘variants of concerns’ (VOCs).

Abstract: In this talk, I will introduce the Transformed Gaussian Processes, a stochastic process specified by transforming samples from a Gaussian process using an invertible transformation (warping function). These processes can be easily made non-stationary by parameterizing the warping function through an input-dependent transformation. I show how this is achieved with a Bayesian Neural Network implemented with Monte Carlo dropout with the additional benefit of incorporating uncertainties, effectively regularizing the model. This new model can match the performance of a Deep Gaussian Process at a fraction of its cost and also allow us to incorporate inductive biases in the function that we are trying to model (e.g. positive constraints), among other benefits. Training and predictions can be scaled using a sparse variational inference algorithm. We also show how the basic idea of Transformed Gaussian Processes can be used to create a set of C dependent function priors which can provide similar or better results than an SVGP in classification problems with a big number of classes but one order of magnitude faster.

Abstract: I will describe a kernel-based nonparametric test of relative goodness of fit, where the goal is to compare two models, both of which may have unobserved latent variables, such that the marginal distributions of the observed variables are intractable. Given the premise that “all models are wrong,” the goal of the test is to determine whether one model significantly outperforms the other in respect of a reference data sample. The test generalises earlier kernel Stein discrepancy (KSD) tests to the case of latent variable models, a much more general class than the fully observed models treated previously. The new test, with a properly calibrated threshold, has a well-controlled type-I error. In the case of models with low-dimensional latent structure and high-dimensional observations, our test significantly outperforms the relative maximum mean discrepancy test, which is based on samples from the models, and does not exploit the latent structure. I will illustrate the test on probabilistic topic models of arXiv articles.

Abstract: Optimizing expensive-to-evaluate black-box functions of discrete (and potentially continuous) design parameters is a ubiquitous problem in science and engineering applications. Bayesian optimization (BO) is a popular sample-efficient method that selects promising designs to evaluate by optimizing an acquisition function (AF) over some domain based on a probabilistic surrogate model. However, maximizing the AF over mixed or high-cardinality discrete search spaces is challenging as we cannot use standard gradient-based methods or evaluate the AF at every point in the search space. To address this issue, we propose using probabilistic reparameterization (PR). Instead of directly optimizing the AF over the search space containing discrete parameters, we instead maximize the expectation of the AF over a probability distribution defined by continuous parameters. We prove that under suitable reparameterizations, the BO policy that maximizes the probabilistic objective is the same as that which maximizes the AF, and therefore, PR enjoys the same regret bounds as the underlying AF. Moreover, our approach admits provably converges to a stationary point of the probabilistic objective under gradient ascent using scalable, unbiased estimators of both the probabilistic objective and its gradient, and therefore, as the number of starting points and gradient steps increases, our approach will recover of a maximizer of the AF (an often-neglected requisite for commonly used BO regret bounds). We validate our approach empirically and demonstrate state-of-the-art optimization performance on many real-world applications. PR is complementary to (and benefits) recent work and naturally generalizes to settings with multiple objectives and black-box constraints.

Abstract: A number of variational autoencoders (VAEs) have recently emerged with the aim of modelling multimodal data, e.g., to jointly model images and their corresponding captions. Still, multimodal VAEs tend to focus solely on a subset of the modalities, e.g., by fitting the image while neglecting the caption. We refer to this limitation as modality collapse. In this presentation, I argue that this effect is a consequence of modality-specific gradients conflicting during the training of multimodal VAEs. After this talk, you will be able to detect which parts of your model’s computational graph can suffer from gradients conflict (which I call impartiality blocks), as well as how to leverage existing gradient-conflict solutions from multitask learning to mitigate modality collapse. In order words, you will learn how to encourage impartial optimization across modalities. The framework I introduce is general, and we have successfully applied it to several multimodal VAE models, losses, and datasets from the literature, and empirically showed that it significantly improves the reconstruction performance, conditional generation, and coherence of the latent space across modalities.

Abstract: The performance of many algorithms in the fields of hard combinatorial problem solving, machine learning or AI in general depends on hyperparameter tuning. Automated methods have been proposed to alleviate users from the tedious and error-prone task of manually searching for performance-optimized configurations. However, there is still a lot of untapped potential. Existing solution approaches often neglect the non-stationarity of hyperparameters where different hyperparameter values are optimal at different stages of an algorithms run. Taking the non-stationarity into account in the optimization procedure promises much better performances but also poses many new challenges. In this talk we will discuss existing solution approaches to classical hyperparameter optimization and explore ways of tackling the non-stationarity of hyperparameters in particular by means of reinforcement learning.

Abstract: State-of-the-art machine learning models can be vulnerable to very small input perturbations that are adversarially constructed. Adversarial attacks are a popular framework for studying these vulnerabilities. They consider worst-case input disturbances designed to maximize model error and got a lot of attention due to their impact on the performance of state-of-the-art models. Adversarial training considers extending model training with these examples and is an effective approach to defend against such attacks. This talk will explore adversarial attacks and training in linear regression. There is a strong reason for this focus, for linear regression, adversarial training can be formulated as a convex and quadratic problem. Moreover, many interesting phenomena that can be observed in nonlinear models are still present. The setup is used to study the role of high dimensionality in robustness. And to reveal the connection between adversarial training, parameter-shrinking methods and minimum-norm solutions.

Abstract: Score-based generative models (SGMs) are a powerful class of generative models that exhibit remarkable empirical performance. Score-based generative modelling consists of a noising stage, whereby a diffusion is used to gradually add Gaussian noise to data, and a generative model, which entails a denoising process defined by approximating the time-reversal of the diffusion. Existing SGMs assume that data is supported on a Euclidean space, i.e. a manifold with flat geometry. In many domains such as robotics, geoscience or protein modelling, data is often naturally described by distributions living on Riemannian manifolds and current SGM techniques are not appropriate. We introduce here Riemannian Score-based Generative Models (RSGMs), a class of generative models extending SGMs to Riemannian manifolds. We demonstrate our approach on a variety of manifolds, and in particular with earth and climate science spherical data.

Abstract: While Bayesian deep learning has been a popular field of research in recent years, most of the work has focused on improving inference methods for better performance and lower computational costs. Conversely, the priors have often been ignored and merely chosen to be isotropic Gaussian, for mathematical and computational convenience. In this talk, I will review recent work that calls this popular practice into question and highlights pathologies that can arise from prior misspecification in Bayesian neural networks. I will then present different methods that can aid the selection of better priors and I will discuss the advantages of function-space priors over weight-space ones.

Abstract: Simulator-based models are models for which the likelihood is intractable but simulation of synthetic data is possible. They are often used to describe complex real-world phenomena, and as such can often be misspecified in practice. In this talk, I will present a novel algorithm based on the posterior bootstrap and maximum mean discrepancy estimators. This leads to a highly-parallelisable Bayesian inference algorithm with strong robustness properties. This is demonstrated through an in-depth theoretical study which includes generalisation bounds, frequentist consistency and robustness of our posterior guarantees. The approach is then illustrated on a range of examples including a g-and-k distribution and a toggle-switch model.

Abstract: Causal structure learning aims to discover the causal directed acyclic graph (DAG) responsible for generating a given dataset. However, a point estimate can be flawed due to limited data as well as non-identifiability of the underlying DAG. Bayesian approaches to structure learning instead seek to characterize a full posterior distribution over DAGs, but are typically very computationally expensive in high dimensions. In this talk, I will present Tractable Uncertainty for STructure learning (TRUST), a framework for approximate posterior inference that relies on probabilistic circuits, a type of tractable probabilistic model, as the representation of our posterior belief over causal DAGs. In comparison to Monte-Carlo posterior approximations, our representation can capture a much richer space of DAGs, while also being able to tractably reason about the uncertainty; for example, inferring the most likely completion of a partial graph, or the expected linear causal effect. Finally, I will show how our posterior representations can be learned by exploiting existing structure learning algorithms together with variational inference, leading empirically to improvement in both the quality of inferred structures and posterior uncertainty.

Abstract: The numerical simulation of differential equations underpins many modelling decisions made in the natural sciences. Solving differential equations with probabilistic numerical algorithms promises better uncertainty quantification than with non-probabilistic approaches, but until recently, probabilistic solvers have been inefficient, unstable, and generally impractical. In this talk, I will explain the fundamentals of probabilistic numerical algorithms for the simulation of ordinary differential equations. Building on this, I will survey the stable and efficient implementation of probabilistic numerical solvers and discuss generalisations of the algorithm to partial differential equations.

Abstract: Comparative decisions, such as picking between two cars or deciding between two hiking trails, require the users to visit multiple webpages and contrast the choices along relevant aspects. Given the impressive capabilities of pre-trained large language models, we ask whether they can help automate such analysis. We refer to this task as extractive aspect-based contrastive summarization which involves constructing a structured summary that compares the choices along relevant aspects. In this paper, we propose a novel method called STRUM for this task that can generalize across domains without requiring any human-written summaries or fixed aspect list as supervision. Given a set of relevant input webpages, STRUM solves this problem using two pre-trained T5-based large language models: first one fine-tuned for aspect and value extraction, and second one fine-tuned for natural language inference. We showcase the abilities of our method across different domains, identify shortcomings, and discuss questions that we believe will be critical in this new line of research.

Abstract: In this talk, I present extensive evidence against the common belief that variational Bayesian learning is ineffective for large neural networks. First, I show that a recent deep learning method called sharpness-aware minimization (SAM) solves an optimal convex relaxation of the variational Bayesian objective. Then, I demonstrate that a direct optimization of the variational objective with an Improved Variational Online Newton method (IVON) can consistently match or outperforms Adam for training large networks such as GPT-2 and ResNets from scratch. IVON’s computational costs are nearly identical to Adam but its predictive uncertainty is better. The talk concludes with several new use cases of variational learning where we improve fine-tuning and model merging in Large Language Models, accurately predict generalization error, and faithfully estimate sensitivity to data.

Abstract: A natural way of estimating heteroscedastic label noise in regression is to model the observed (potentially noisy) target as a sample from a normal distribution, whose parameters can be learned by minimizing the negative log-likelihood. This formulation has desirable loss attenuation properties, as it reduces the contribution of high-error examples. Intuitively, this behaviour can improve robustness against label noise by reducing overfitting. We propose an extension of this simple and probabilistic approach to classification, that has the same desirable loss attenuation properties. We evaluate the effectiveness of the method by measuring its robustness against label noise in classification. In follow-up work, we improve the method’s robustness by modelling and estimating a shift (non-zero mean) in the Gaussian noise distribution, which we show makes it possible for the method to correct noisy labels.

Abstract: We lay out three avenues in which we think generative models are especially valuable for modeling biomolecules. 1) Hard prediction tasks can be better addressed with generative models that can suggest and rank multiple solutions (e.g., docking). 2) The dynamics and conformations of biomolecules can be captured with generative models (e.g., protein conformational ensembles and MD trajectories). 3) Designing new biomolecules can be accelerated, informed by samples or likelihoods from generative models (e.g., protein binder or regulatory DNA design).

Abstract: Modeling partial differential equations (PDEs) is of crucial importance in science and engineering. Some of the most common tasks include 1) forecasting, where the aim is to predict future states based on an initial one, as well as 2) inverse problems, such as data assimilation (DA), with the goal of reconstructing an aspect of the PDE (i.e. coefficient, initial condition, full trajectory, etc.) given some partial observations of the solution to the PDE. However, most previous numerical and machine learning approaches that target forecasting cannot be applied out-of-the-box for data assimilation. Recently, diffusion models have emerged as a powerful tool for conditional generation, being able to flexibly incorporate observations without retraining. In this talk, I will discuss our recent work in this domain, where we perform a comparative study of score-based diffusion models for forecasting and assimilation of sparse observations. In particular, we focus on diffusion models that are either presented with the conditional information during training, or conditioned after unconditional training. We address the shortcomings of previous work and develop methods that are able to successfully tackle the combination of forecasting and data assimilation, a task commonly encountered in real-world scenarios such as weather modeling.

Abstract: We introduce a software package, Pigeons.jl, that provides a way to leverage distributed computation to obtain samples from complicated probability distributions, such as multimodal posteriors arising in Bayesian inference and high-dimensional distributions in statistical mechanics. Pigeons.jl provides simple APIs to perform such computations single-threaded, multi-threaded, and/or distributed over thousands of MPI-communicating machines. Our software provides several Markov kernels, including our newly proposed algorithm, autoMALA. This MCMC algorithm, based on the Metropolis-adjusted Langevin algorithm, automatically sets its step size at each iteration based on the local geometry of the target distribution. Our experiments demonstrate that autoMALA is competitive with related state-of the-art MCMC methods, in terms of the number of log density evaluations per effective sample, and it outperforms state-of-the-art samplers on targets with varying geometries.

Abstract: Probabilistic circuits are prominent tractable probabilistic models which can provide exact answers to a wide range of probabilistic queries in a tractable way. Given their sparse nature and the structural constraints enabling exact inference, it is challenging to induce these models in high-dimensional real-world domains such as time series and raw images. In this talk, I show how we can leverage spectral modeling and the clear probabilistic semantics of probabilistic circuits to learn models able to provide efficient and reliable predictions in these challenging domains, and how to make these particular models more robust to distribution shift and out-of-distribution data.

Abstract: The past decade has witnessed a remarkable surge in deep learning applications, spanning from recommender systems to fundamental sciences like biology and chemistry. However, when applied to scientific domains, deep learning methods lack effective inductive biases, leading to suboptimal performance and limited generalizability. By bridging the gap between deep learning and physical inductive biases, we aim to develop robust, interpretable, and scientifically grounded AI methods. In this talk, I will present our recent works on developing physics-inspired deep learning approaches in AI4Science, such as modeling weather and climate, generating macromolecules, and designing topological GNNs for drug discovery.

Abstract: Deep learning has made significant progress, but robustness issues like overconfidence, hallucinations, and adversarial vulnerability remain critical challenges in safety-critical applications. Bayesian methods — such as variational inference or even ensembles — have shown promise in improving robustness and uncertainty estimation but are often computationally expensive, paving the way for more efficient methods. Furthermore, some are difficult to implement, a problem we address in the PyTorch library TorchUncertainty. Finally, since Bayesian methods on the weight space are difficult to scale to the current computer vision models, we suggest exploring other arguably more adapted lower-dimensional avenues.

Abstract: TBA