Proximal gradient methods for learning: Difference between revisions

Revision as of 04:47, 7 December 2013

Revision as of 19:44, 7 December 2013

Line 85:

===Adaptive step size===

Consider a problem of the form <math>\min_w F(w) + R(w),</math> where <math>F(w)</math> is convex and differentiable and <math>R(w)</math> is convex (for example, for differentiable [[Loss_function|loss functions]] empirical lossrisk minimization with regularization takes this form). This gives rise to the fixed point iteration scheme

:<math>w^{k+1} = \operatorname{prox}_{\gamma R}\left(w^k-\gamma \nabla F\left(w^k\right)\right).</math>

One standard modification is to let <math>\gamma</math> vary with <math>k</math>, hence we have the scheme