This blog post presents an interesting phenomenom that occurs in unconditional diffusion models used for inverse problem solving: a robustness to perturbations and noise to the input contexts. We claim that this phenomenom occurs due to the noisy context seen during training and show the results of some experiments on simple numerical trajectory examples that help justify this claim.

In regression problems, we often model the conditional mean of a conditional probability distribution $p(\textbf{t}|\textbf{x})$ by minimizing the least squares ($L^2$-loss) between the input and target variables (this can be shown through a brief derivation using the calculus of variations). For more complicated multimodal distributions, however, using merely the conditional mean of a univariate Gaussian can prove to be problematic.

In this blog post, we show how a neural network can represent more general distributions (in our case, we fit a Gaussian mixture model) and then show how one might implement such a network in code. Such models are known as mixture density networks, and we will see how to model the errors and corresponding outputs.

Hello World!