Quadratic programming: Difference between revisions

Revision as of 16:24, 24 August 2023

Latest revision as of 05:55, 18 July 2025

(23 intermediate revisions by 15 users not shown)

Line 25:

where {{math|'''x'''^T}} denotes the vector [[transpose]] of {{math|'''x'''}}, and the notation {{math|''A'''''x''' ⪯ '''b'''}} means that every entry of the vector {{math|''A'''''x'''}} is less than or equal to the corresponding entry of the vector {{math|'''b'''}} (component-wise inequality).

===LeastConstrained least squares===

As a special case when {{math|''Q''}} is [[positive definite matrix|symmetric positive-definite]], the cost function reduces to least squares:

:{| cellspacing="10"

|-

Line 36:

| <math>A \mathbf{x} \preceq \mathbf{b},</math>

|}

where {{math|1=''Q'' = ''R''^T''R''}} follows from the [[Cholesky decomposition]] of {{math|''Q''}} and {{math|1='''c''' = −''R''^T '''d'''}}. Conversely, any such [[constrained least squares]] program can be equivalently framed as a quadratic programming problem, even for a generic non-square {{math|''R''}} matrix.

===Generalizations===

Line 100:

:<math>Z^\top Q Z \mathbf{y} = -Z^\top \mathbf{c}</math>

Under certain conditions on {{mvar|Q}}, the reduced matrix {{math|''Z''^T''QZ''}} will be positive definite. It is possible to write a variation on the [[conjugate gradient method]] which avoids the explicit calculation of {{mvar|Z}}.<ref>{{Cite journal | last1 = Gould| first1 = Nicholas I. M.| last2 = Hribar| first2 = Mary E.| last3 = Nocedal| first3 = Jorge|date=April 2001| title = On the Solution of Equality Constrained Quadratic Programming Problems Arising in Optimization| journal = SIAM J. Sci. Comput.| pages = 1376–1395| volume = 23| issue = 4| citeseerx = 10.1.1.129.7555| doi = 10.1137/S1064827598345667| bibcode = 2001SJSC...23.1376G}}</ref>

==Lagrangian duality==

Line 119:

Besides the Lagrangian duality theory, there are other duality pairings (e.g. [[Wolfe duality|Wolfe]], etc.).

==Run-time complexity==

==Complexity==

=== Convex quadratic programming ===

For [[positive-definite matrix|positive definite]] {{mvar|Q}}, the [[ellipsoid method]] solves the problem in (weakly) [[polynomial time]].<ref>{{cite journal| last=Kozlov | first=M. K. |author2=S. P. Tarasov | author3-link=Leonid Khachiyan |author3=Leonid G. Khachiyan | year=1979 | title=[Polynomial solvability of convex quadratic programming] | journal=[[Doklady Akademii Nauk SSSR]] | volume=248 | pages=1049–1051}} Translated in: {{cite journal| journal=Soviet Mathematics - Doklady | volume=20 | pages=1108–1111}}</ref> If, on the other hand, {{mvar|Q}} is indefinite, then the problem is [[NP-hard]].<ref>{{cite journal | last = Sahni | first = S. | title = Computationally related problems | journal = SIAM Journal on Computing | volume = 3 | issue = 4 | pages = 262–279 | year = 1974 | doi=10.1137/0203021| url = http://www.cise.ufl.edu/~sahni/papers/comp.pdf | citeseerx = 10.1.1.145.8685 }}</ref>

ThereFor can be several stationary points and local minima for these non[[positive-convexdefinite problems.matrix|positive In fact, even ifdefinite]] {{mvar|Q}}, haswhen onlythe oneproblem negativeis convex, the [[eigenvalueellipsoid method]], solves the problem isin (stronglyweakly) [[NP-hardpolynomial time]].<ref>{{cite journal | title last=Kozlov Quadratic| programmingfirst=M. withK. one|author2=S. negativeP. eigenvalueTarasov is (strongly)| NPauthor3-hardlink=Leonid Khachiyan | first1 author3=Leonid Panos MG. Khachiyan | last1 year= Pardalos1979 | first2 title=[Polynomial Stephensolvability A.of |convex last2quadratic = Vavasisprogramming] | journal =[[Doklady JournalAkademii ofNauk Global OptimizationSSSR]] | volume = 1248 | issue pages=1049–1051}} 1Translated |in: year = 1991{{cite journal| pages journal=Soviet 15–22Mathematics |- Doklady doi=10.1007/bf00120662| s2cid volume=20 12602885| pages=1108–1111}}</ref>

Ye and Tse<ref>{{Cite journal |last1=Ye |first1=Yinyu |last2=Tse |first2=Edison |date=1989-05-01 |title=An extension of Karmarkar's projective algorithm for convex quadratic programming |url=https://doi.org/10.1007/BF01587086 |journal=Mathematical Programming |language=en |volume=44 |issue=1 |pages=157–179 |doi=10.1007/BF01587086 |s2cid=35753865 |issn=1436-4646|url-access=subscription }}</ref> present a polynomial-time algorithm, which extends [[Karmarkar's algorithm]] from linear programming to convex quadratic programming. On a system with ''n'' variables and ''L'' input bits, their algorithm requires O(L n) iterations, each of which can be done using O(L n³) arithmetic operations, for a total runtime complexity of O(''L''² ''n''⁴).

== Integer constraints ==

Kapoor and Vaidya<ref>{{Cite book |last1=Kapoor |first1=S |last2=Vaidya |first2=P M |chapter=Fast algorithms for convex quadratic programming and multicommodity flows |date=1986-11-01 |title=Proceedings of the eighteenth annual ACM symposium on Theory of computing - STOC '86 |chapter-url=https://dl.acm.org/doi/10.1145/12130.12145 |___location=New York, NY, USA |publisher=Association for Computing Machinery |pages=147–159 |doi=10.1145/12130.12145 |isbn=978-0-89791-193-1|s2cid=18108815 }}</ref> present another algorithm, which requires O(''L'' * log ''L'' ''* n''^3.67 * log ''n'') arithmetic operations.

=== Non-convex quadratic programming ===

If {{mvar|Q}} is indefinite, (so the problem is non-convex) then the problem is [[NP-hard]].<ref>{{cite journal | last = Sahni | first = S. | title = Computationally related problems | journal = SIAM Journal on Computing | volume = 3 | issue = 4 | pages = 262–279 | year = 1974 | doi=10.1137/0203021| url = http://www.cise.ufl.edu/~sahni/papers/comp.pdf | citeseerx = 10.1.1.145.8685 }}</ref> A simple way to see this is to consider the non-convex quadratic constraint ''x_i''² = ''x_i''. This constraint is equivalent to requiring that ''x_i'' is in {0,1}, that is, ''x_i'' is a binary integer variable. Therefore, such constraints can be used to model any [[Integer programming|integer program]] with binary variables, which is known to be NP-hard.

Moreover, these non-convex problems might have several stationary points and local minima. In fact, even if {{mvar|Q}} has only one negative [[eigenvalue]], the problem is (strongly) [[NP-hard]].<ref>{{cite journal | title = Quadratic programming with one negative eigenvalue is (strongly) NP-hard | first1 = Panos M. | last1 = Pardalos | first2 = Stephen A. | last2 = Vavasis | journal = Journal of Global Optimization | volume = 1 | issue = 1 | year = 1991 | pages = 15–22 | doi=10.1007/bf00120662| s2cid = 12602885 }}</ref>

Moreover, finding a KKT point of a non-convex quadratic program is CLS-hard.<ref>{{Cite arXiv |eprint=2311.13738 |last1=Fearnley |first1=John |last2=Goldberg |first2=Paul W. |last3=Hollender |first3=Alexandros |last4=Savani |first4=Rahul |title=The Complexity of Computing KKT Solutions of Quadratic Programs |date=2023 |class=cs.CC }}</ref>

== Mixed-integer quadratic programming ==

There are some situations where one or more elements of the vector {{math|'''x'''}} will need to take on [[integer]] values. This leads to the formulation of a mixed-integer quadratic programming (MIQP) problem.<ref>{{Cite journal|last=Lazimy|first=Rafael|date=1982-12-01|title=Mixed-integer quadratic programming|journal=Mathematical Programming| language=en| volume=22| issue=1| pages=332–349| doi=10.1007/BF01581047| s2cid=8456219|issn=1436-4646}}</ref> Applications of MIQP include [[water resources]]<ref>{{Cite journal|last1=Propato Marco|last2=Uber James G.|date=2004-07-01|title=Booster System Design Using Mixed-Integer Quadratic Programming|journal=Journal of Water Resources Planning and Management|volume=130|issue=4|pages=348–352|doi=10.1061/(ASCE)0733-9496(2004)130:4(348)}}</ref> and the [[Tracking error#Index fund creation|construction of index funds]].<ref>{{Cite book|last1=Cornuéjols|first1=Gérard|url=https://www.cambridge.org/core/books/optimization-methods-in-finance/8A4996C5DB2006224E4D983B5BC95E3B|title=Optimization Methods in Finance|last2=Peña|first2=Javier|last3=Tütüncü|first3=Reha|publisher=Cambridge University Press|year=2018|isbn=9781107297340|edition=2nd|___location=Cambridge, UK|pages=167–168}}</ref>

Line 159 ⟶ 170:

|-

|[[IPOPT]]|| IPOPT (Interior Point OPTimizer) is a software package for large-scale nonlinear optimization.

|-

|[[Julia (programming language)|Julia]]

|A high-level programming language with notable solving package being [https://jump.dev/JuMP.jl/stable/ JuMP]

|-

|[[Maple (software)|Maple]]|| General-purpose programming language for mathematics. Solving a quadratic problem in Maple is accomplished via its [http://www.maplesoft.com/support/help/Maple/view.aspx?path=Optimization/QPSolve QPSolve] command.

Line 176 ⟶ 190:

|[[SAS System|SAS]]/OR|| A suite of solvers for Linear, Integer, Nonlinear, Derivative-Free, Network, Combinatorial and Constraint Optimization; the [[Algebraic modeling language]] OPTMODEL; and a variety of vertical solutions aimed at specific problems/markets, all of which are fully integrated with the [[SAS System]].

|-

|[[SuanShu_numerical_librarySuanShu numerical library|SuanShu]]|| an open-source suite of optimization algorithms to solve [[Linear programming|LP]], QP, [[SOCP]], [[Semidefinite_programmingSemidefinite programming|SDP]], [[Sequential_quadratic_programmingSequential quadratic programming|SQP]] in Java

|-

|[[TK Solver]]|| Mathematical modeling and problem solving software system based on a declarative, rule-based language, commercialized by Universal Technical Systems, Inc..

Line 184 ⟶ 198:

|[[FICO Xpress|XPRESS]]||Solver for large-scale linear programs, quadratic programs, general nonlinear and mixed-integer programs. Has API for several programming languages, also has a modelling language Mosel and works with AMPL, [[General Algebraic Modeling System|GAMS]]. Free for academic use.

|}

== Extensions ==

'''Polynomial optimization'''<ref>{{Citation |last=Tuy |first=Hoang |title=Polynomial Optimization |date=2016 |url=https://doi.org/10.1007/978-3-319-31484-6_12 |work=Convex Analysis and Global Optimization |pages=435–452 |editor-last=Tuy |editor-first=Hoang |access-date=2023-12-16 |series=Springer Optimization and Its Applications |volume=110 |place=Cham |publisher=Springer International Publishing |language=en |doi=10.1007/978-3-319-31484-6_12 |isbn=978-3-319-31484-6|url-access=subscription }}</ref> is a more general framework, in which the constraints can be [[polynomial function]]s of any degree, not only 2.

==See also==

Line 202 ⟶ 219:

|publisher=RAL Numerical Analysis Group Internal Report 2000-1

|url=ftp://ftp.numerical.rl.ac.uk/pub/qpbook/qp.pdf

|archive-url=https://web.archive.org/web/20170705054523/ftp://ftp.numerical.rl.ac.uk/pub/qpbook/qp.pdf|archive-date=2017-07-05|url-status=dead|date=2000

|date=2000

}}

==External links==

*[http://www.numerical.rl.ac.uk/qp/qp.html A page about QPquadratic programming] {{Webarchive|url=https://web.archive.org/web/20110607212051/http://www.numerical.rl.ac.uk/qp/qp.html |date=2011-06-07 }}

*[https://neos-guide.org/content/quadratic-programming NEOS Optimization Guide: Quadratic Programming]

*[https://www.math.uh.edu/~rohop/fall_06/Chapter3.pdf Quadratic Programming] {{Webarchive|url=https://web.archive.org/web/20230408093722/https://www.math.uh.edu/~rohop/fall_06/Chapter3.pdf |date=2023-04-08 }}

*[https://or.stackexchange.com/q/989/2576 Cubic programming and beyond], in Operations Research stack exchange

{{Mathematical programming}}

{{Optimization algorithms}}

{{Authority control}}

[[Category:Optimization algorithms and methods]]