Convex optimization: Difference between revisions

Revision as of 04:39, 10 February 2025

Latest revision as of 21:50, 25 August 2025

(7 intermediate revisions by 6 users not shown)

Line 1:

{{short description|Subfield of mathematical optimization}}

'''Convex optimization''' is a subfield of [[mathematical optimization]] that studies the problem of minimizing [[convex function]]s over [[convex set]]s (or, equivalently, maximizing [[concave functions]] over convex sets). Many classes of convex optimization problems admit polynomial-time algorithms,<ref name="Nesterov 1994">{{harvnb|Nesterov|Nemirovskii|1994}}</ref> whereas mathematical optimization is in general [[NP-hard]].<ref>

{{cite journal | last1 = Murty | first1 = Katta | last2 = Kabadi | first2 = Santosh | title = Some NP-complete problems in quadratic and nonlinear programming | journal = Mathematical Programming | volume = 39 | issue = 2 | pages = 117–129 | year = 1987 | doi = 10.1007/BF02592948| hdl = 2027.42/6740 | s2cid = 30500771 | hdl-access = free}}</ref><ref>Sahni, S. "Computationally related problems," in SIAM Journal on Computing, 3, 262--279, 1974.</ref><ref>{{cite journal | url=https://link.springer.com/article/10.1007/BF00120662 | doi=10.1007/BF00120662 | title=Quadratic programming with one negative eigenvalue is NP-hard | date=1991 | last1=Pardalos | first1=Panos M. | last2=Vavasis | first2=Stephen A. | journal=Journal of Global Optimization | volume=1 | pages=15–22 | url-access=subscription }}</ref>

==Definition==

Line 72:

{{cite journal |last1=Agrawal |first1=Akshay |last2=Verschueren |first2=Robin |last3=Diamond |first3=Steven |last4=Boyd |first4=Stephen |year=2018 |title=A rewriting system for convex optimization problems |url=https://web.stanford.edu/~boyd/papers/pdf/cvxpy_rewriting.pdf |journal=Control and Decision |volume=5 |issue=1 |pages=42–60 |arxiv=1709.04494 |doi=10.1080/23307706.2017.1397554 |s2cid=67856259}}</ref>

[[File:Hierarchy compact convex.pngsvg|thumb|A hierarchy of convex optimization problems. (LP: [[linear programming]], QP: [[quadratic programming]], SOCP [[Second-order cone programming|second-order cone program]], SDP: [[semidefinite programming]], CP: [[conic optimization]].)]]

*[[Linear programming]] problems are the simplest convex programs. In LP, the objective and constraint functions are all linear.

Line 84:

*[[Geometric programming]]

*[[Entropy maximization]] with appropriate constraints.

==Properties==

The following are useful properties of convex optimization problems:<ref name="rockafellar93">{{cite journal | author = Rockafellar, R. Tyrrell | title = Lagrange multipliers and optimality | journal = SIAM Review | volume = 35 | issue = 2 | year = 1993 | pages = 183–238 |url = http://web.williams.edu/Mathematics/sjmiller/public_html/105Sp10/handouts/Rockafellar_LagrangeMultAndOptimality.pdf | doi=10.1137/1035044| citeseerx = 10.1.1.161.7209}}</ref><ref name=":2" />{{Rp|___location=chpt.4}}

* every point that is [[local minimum]] is also a [[global minimum]];

* the optimal set is convex;

* if the objective function is ''strictly'' convex, then the problem has at most one optimal point.

Line 117 ⟶ 118:

*[[Subgradient method]]

*[[Drift plus penalty|Dual subgradients and the drift-plus-penalty method]]

Subgradient methods can be implemented simply and so are widely used.<ref>{{Cite journal |date=2006 |title=Numerical Optimization |url=https://link.springer.com/book/10.1007/978-0-387-40065-5 |journal=Springer Series in Operations Research and Financial Engineering |language=en |doi=10.1007/978-0-387-40065-5|isbn=978-0-387-30303-1 |url-access=subscription }}</ref> Dual subgradient methods are subgradient methods applied to a [[Duality (optimization)|dual problem]]. The [[Drift plus penalty|drift-plus-penalty]] method is similar to the dual subgradient method, but takes a time average of the primal variables.{{Citation needed|date=April 2021}}

== Lagrange multipliers ==

Line 150 ⟶ 151:

Solvers implement the algorithms themselves and are usually written in C. They require users to specify optimization problems in very specific formats which may not be natural from a modeling perspective. Modeling tools are separate pieces of software that let the user specify an optimization in higher-level syntax. They manage all transformations to and from the user's high-level model and the solver's input/output format.

TheBelow tableare belowtwo showstables. aThe mixfirst ofshows shows modelingmodelling tools (such as CVXPY and ConvexJuMP.jl) and the second solvers (such as CVXOPTSCS and MOSEK). This tableThey isare by no means exhaustive.

{| class="wikitable sortable"

|+

Line 164 ⟶ 165:

!{{Yes}}

|<ref name=":3">{{Cite web|last=Borchers|first=Brian|title=An Overview Of Software For Convex Optimization|url=http://infohost.nmt.edu/~borchers/presentation.pdf|url-status=dead|archive-url=https://web.archive.org/web/20170918180026/http://infohost.nmt.edu/~borchers/presentation.pdf|archive-date=2017-09-18|access-date=12 Apr 2021}}</ref>

|-

|CVXMOD

|[[Python (programming language)|Python]]

|Interfaces with the [https://cvxopt.org/ CVXOPT] solver.

!{{Yes}}

|<ref name=":3" />

|-

|CVXPY

|Python

|

!{{Yes}}

|

|<ref>{{Cite web|title=Welcome to CVXPY 1.1 — CVXPY 1.1.11 documentation|url=https://www.cvxpy.org/|access-date=2021-04-12|website=www.cvxpy.org}}</ref>

|-

Line 189 ⟶ 184:

|<ref>{{Cite web|title=Disciplined Convex Optimiation - CVXR|url=https://www.cvxgrp.org/CVXR/|access-date=2021-06-17|website=www.cvxgrp.org}}</ref>

|-

|GAMS

|YALMIP

|

|MATLAB, Octave

|Modeling system for linear, nonlinear, mixed integer linear/nonlinear, and second-order cone programming problems.

|Interfaces with CPLEX, GUROBI, MOSEK, SDPT3, SEDUMI, CSDP, SDPA, PENNON solvers; also supports integer and nonlinear optimization, and some nonconvex optimization. Can perform [[robust optimization]] with uncertainty in LP/SOCP/SDP constraints.

!{{Yes}}

|<ref name=":3" />

|-

|LMI lab

|MATLAB

|Expresses and solves semidefinite programming problems (called "linear matrix inequalities")

!{{No}}

|<ref name=":3" />

|-

|GloptiPoly 3

|LMIlab translator

|MATLAB,

|

Octave

|Transforms LMI lab problems into SDP problems.

|Modeling system for polynomial optimization.

!{{Yes}}

|<ref name=":3" />

|-

|JuMP.jl

|xLMI

|[[PythonJulia (programming language)|PythonJulia]]

|MATLAB

|Supports many solvers. Also supports integer and nonlinear optimization, and some nonconvex optimization.

|Similar to LMI lab, but uses the SeDuMi solver.

!{{Yes}}

|<ref>{{cite journal |last1=Lubin |first1=Miles |last2=Dowson |first2=Oscar |last3=Dias Garcia |first3=Joaquim |last4=Huchette |first4=Joey |last5=Legat |first5=Benoît |last6=Vielma |first6=Juan Pablo |date=2023 |title=JuMP 1.0: Recent improvements to a modeling language for mathematical optimization | journal = Mathematical Programming Computation | doi = 10.1007/s12532-023-00239-3 |eprint=2206.03866 }}</ref>

|<ref name=":3" />

|-

|AIMMS

|

|Can do robust optimization on linear programming (with MOSEK to solve second-order cone programming) and [[mixed integer linear programming]]. Modeling package for LP + SDP and robust versions.

!{{No}}

|<ref name=":3" />

|-

|ROME

|

|Modeling system for robust optimization. Supports distributionally robust optimization and [[uncertainty set]]s.

!{{Yes}}

|<ref name=":3" />

|-

|GloptiPoly 3

|MATLAB,

Octave

|Modeling system for polynomial optimization.

!{{Yes}}

|<ref name=":3" />

Line 242 ⟶ 219:

|Modeling system for polynomial optimization. Uses the SDPA or SeDuMi solvers.

!{{Yes}}

|<ref name=":3" />

|-

|YALMIP

|MATLAB, Octave

|Interfaces with CPLEX, GUROBI, MOSEK, SDPT3, SEDUMI, CSDP, SDPA, PENNON solvers; also supports integer and nonlinear optimization, and some nonconvex optimization. Can perform [[robust optimization]] with uncertainty in LP/SOCP/SDP constraints.

!{{Yes}}

|<ref name=":3" />

|-}

{| class="wikitable sortable"

|+

!Program

!Language

!Description

![[Free and open-source software|FOSS]]?

!Ref

|-

|AIMMS

|

|Can do robust optimization on linear programming (with MOSEK to solve second-order cone programming) and [[mixed integer linear programming]]. Modeling package for LP + SDP and robust versions.

!{{No}}

|<ref name=":3" />

|-

Line 314 ⟶ 311:

|Supports general-purpose codes. Uses low-rank factorization with an augmented Lagrangian method.

!{{Yes}}

|<ref name=":3" />

|-

|GAMS

|

|Modeling system for linear, nonlinear, mixed integer linear/nonlinear, and second-order cone programming problems.

!{{No}}

|<ref name=":3" />

|-

|Optimization Services

|

|XML standard for encoding optimization problems and solutions.

|

|<ref name=":3" />

|}

== Applications ==

Convex optimization can be used to model problems in a wide range of disciplines, such as automatic [[control systems]], estimation and [[signal processing]], communications and networks, electronic [[circuit design]],<ref name=":2" />{{Rp|___location=|page=17}} data analysis and modeling, [[finance]], [[statistics]] ([[optimal design|optimal experimental design]]),<ref>ChritensenChristensen/Klarbring, chpt. 4.</ref> and [[structural optimization]], where the approximation concept has proven to be efficient.<ref name=":2" /><ref>Schmit, L.A.; Fleury, C. 1980: ''Structural synthesis by combining approximation concepts and dual methods''. J. Amer. Inst. Aeronaut. Astronaut 18, 1252-1260</ref> Convex optimization can be used to model problems in the following fields:

* [[Portfolio optimization]].<ref name=":0">{{Cite web |last1=Boyd |first1=Stephen |last2=Diamond |first2=Stephen |last3=Zhang |first3=Junzi |last4=Agrawal |first4=Akshay |title=Convex Optimization Applications |url=https://web.stanford.edu/~boyd/papers/pdf/cvx_applications.pdf |url-status=live |archive-url=https://web.archive.org/web/20151001185038/http://web.stanford.edu/~boyd/papers/pdf/cvx_applications.pdf |archive-date=2015-10-01 |access-date=12 Apr 2021}}</ref>

Line 348 ⟶ 333:

==See also==

* [[Duality (optimization)|Duality]]

* [[Hierarchical Risk Parity]]

* [[Karush–Kuhn–Tucker conditions]]

* [[Optimization problem]]