This website contains sample trajectory visualizations for our paper “Generative Modeling With Inverse Heat Dissipation”.
While diffusion models have shown great success in image generation, their noise-inverting generative process does not explicitly consider the inductive biases of natural images, such as their inherent multi-scale nature. Inspired by diffusion models and the desirability of coarse-to-fine modelling, we propose a new model that generates images through iteratively inverting the heat equation, a PDE that locally erases fine-scale information when run over the 2D plane of the image. In our novel methodology, the solution of the forward heat equation is interpreted as a variational approximation in a directed graphical model. We demonstrate promising image quality and point out emergent qualitative properties not seen in diffusion models, such as disentanglement of overall colour and shape in images and aspects of neural network interpretability. Spectral analysis on natural images positions our model as a type of dual to diffusion models.
Example of the information destroying forward process (during training) and the generative inverse process (sampling), both defined by the PDE.
We provide generation sequences for different data sets. Generation starts from a flat image and adds progressively more detail. The iterative generative process can be visualized as a video, showing the smooth change from effective low-resolution to high resolution. Effectively, the model redistributes the mass in the original image to form an image.
Sampling with a shared initial state
One way to visualize the stochasticity of the generative process is to keep the initial draw from the prior fixed and sample multiple trajectories based on it. Large-scale features are decided in the beginning of the process and fine-scale features at the end. If we split the sampling to two parts at specified moments, this results in a hierarchy over scales:
Starting from the same initial state results in a wide variety of images. Here are examples from a low-resolution version of AFHQ (64×64):