Diffusion model: Difference between revisions

Revision as of 06:12, 14 April 2025

Revision as of 03:27, 16 April 2025

Line 111:

==== Learning the score function ====

Given a density <math>q</math>, we wish to learn a score function approximation <math>f_\theta \approx \nabla \ln q</math>. This is '''score matching'''''.''<ref>{{Cite web |title=Sliced Score Matching: A Scalable Approach to Density and Score Estimation {{!}} Yang Song |url=https://yang-song.net/blog/2019/ssm/ |access-date=2023-09-24 |website=yang-song.net}}</ref> Typically, score matching is formalized as minimizing '''Fisher divergence''' function <math>E_q[\|f_\theta(x) - \nabla \ln q(x)\|^2]</math>. By expanding the integral, and performing an integration by parts, <math display="block">E_q[\|f_\theta(x) - \nabla \ln q(x)\|^2] = E_q[\|f_\theta\|^2 + 2\nabla^2\cdot f_\theta] + C</math>giving us a loss function, also known as the [[Scoring rule#Hyvärinen scoring rule|Hyvärinen scoring rule]], that can be minimized by stochastic gradient descent.

==== Annealing the score function ====