Interval predictor model: Difference between revisions

Revision as of 14:12, 16 July 2019

Latest revision as of 08:03, 7 July 2025

(46 intermediate revisions by 13 users not shown)

Line 1:

In [[Regressionregression analysis]], an '''Intervalinterval Predictorpredictor Modelmodel''' ('''IPM''') is an approach to regression where bounds on the function to be approximated are obtained.

{{Machine learning bar}}

This differs from other techniques in [[Machinemachine Learninglearning]], where usually one wishes to estimate point values or an entire probability distribution.

In [[Regression analysis]], an '''Interval Predictor Model''' (IPM) is an approach to regression where bounds on the function to be approximated are obtained.

This differs from other techniques in [[Machine Learning]], where usually one wishes to estimate point values or an entire probability distribution.

Interval Predictor Models are sometimes referred to as a [[nonparametric regression]] technique, because a potentially infinite set of functions are contained by the IPM, and no specific distribution is implied for the regressed variables.

As a consequence of the theory of [[Scenario Optimization]], in many cases rigorous predictions can be made regarding the performance of the model at test time <ref name="CampiCalafiore2009">{{cite journal|last1=Campi|first1=M.C.|last2=Calafiore|first2=G.|last3=Garatti|first3=S.|title=Interval predictor models: Identification and reliability|journal=Automatica|volume=45|issue=2|year=2009|pages=382–392|issn=00051098|doi=10.1016/j.automatica.2008.09.004}}</ref>.

Hence an Interval Predictor Model can be seen as a guaranteed bound on [[quantile regression]].

Multiple-input multiple-output IPMs for multi-point data commonly used to represent functions have been recently developed.<ref>{{cite book | doi=10.1109/CDC45484.2021.9683582 | chapter=Interval Predictor Models for Robust System Identification | title=2021 60th IEEE Conference on Decision and Control (CDC) | year=2021 | last1=Crespo | first1=Luis G. | last2=Kenny | first2=Sean P. | last3=Colbert | first3=Brendon K. | last4=Slagel | first4=Tanner | pages=872–879 | isbn=978-1-6654-3659-5 | s2cid=246479771 }}</ref> These IPM prescribe the parameters of the model as a path-connected, semi-algebraic set using sliced-normal <ref>{{cite journal |last1=Crespo |first1=Luis |last2=Colbert |first2=Brendon |last3=Kenny |first3=Sean |last4=Giesy |first4=Daniel |title=On the quantification of aleatory and epistemic uncertainty using Sliced-Normal distributions |journal=Systems and Control Letters |date=2019 |volume=34 |page=104560 |doi=10.1016/j.sysconle.2019.104560 |s2cid=209339118 |url=https://doi.org/10.1016/j.sysconle.2019.104560|url-access=subscription }}</ref> or sliced-exponential distributions.<ref>{{cite book | doi=10.1109/CDC45484.2021.9683584 | chapter=Robust Estimation of Sliced-Exponential Distributions^⋆ | title=2021 60th IEEE Conference on Decision and Control (CDC) | year=2021 | last1=Crespo | first1=Luis G. | last2=Colbert | first2=Brendon K. | last3=Slager | first3=Tanner | last4=Kenny | first4=Sean P. | pages=6742–6748 | isbn=978-1-6654-3659-5 | s2cid=246476974 }}</ref> A key advantage of this approach is its ability to characterize complex parameter dependencies to varying fidelity levels. This practice enables the analyst to adjust the desired level of conservatism in the prediction.

== Convex Interval Predictor Models ==

Typically the Interval Predictor Model is created by specifying a parametric function, which is usually chosen to be the product of a parameter vector and a basis.

As a consequence of the theory of [[Scenarioscenario Optimizationoptimization]], in many cases rigorous predictions can be made regarding the performance of the model at test time .<ref name="CampiCalafiore2009">{{cite journal|last1=Campi|first1=M.C.|last2=Calafiore|first2=G.|last3=Garatti|first3=S.|title=Interval predictor models: Identification and reliability|journal=Automatica|volume=45|issue=2|year=2009|pages=382–392|issn=000510980005-1098|doi=10.1016/j.automatica.2008.09.004}}</ref>.

Hence an Intervalinterval Predictorpredictor Modelmodel can be seen as a guaranteed bound on [[quantile regression]].

Interval predictor models can also be seen as a way to prescribe the [[Support_(mathematics)#Support_of_a_distribution|support]] of random predictor models, of which a [[Gaussian process]] is a specific case

.<ref name="CrespoKenny2018">{{cite journal|last1=Crespo|first1=Luis G.|last2=Kenny|first2=Sean P.|last3=Giesy|first3=Daniel P.|title=Staircase predictor models for reliability and risk analysis|journal=Structural Safety|volume=75|year=2018|pages=35–44|issn=016747300167-4730|doi=10.1016/j.strusafe.2018.05.002|s2cid=126167977 }}</ref>

== Convex interval predictor models ==

Typically the Intervalinterval Predictorpredictor Modelmodel is created by specifying a parametric function, which is usually chosen to be the product of a parameter vector and a basis.

Usually the basis is made up of polynomial features or a radial basis is sometimes used.

Then a convex set is assigned to the parameter vector, and the size of the convex set is minimized such that every possible data point can be predicted by one possible value of the parameters.

Ellipsoidal parameters sets were used by Campi (2009), which yield a convex optimization program to train the IPM .<ref name="CampiCalafiore2009" />.

Crespo (2016) proposed the use of a hyperrectangular parameter set, which results in a convenient, linear form for the bounds of the IPM.<ref name="CrespoKenny2016">{{cite journal|last1=Crespo|first1=Luis G.|last2=Kenny|first2=Sean P.|last3=Giesy|first3=Daniel P.|title=Interval Predictor Models With a Linear Parameter Dependency|journal=Journal of Verification, Validation and Uncertainty Quantification|volume=1|issue=2|year=2016|pages=021007|issn=2377-2158|doi=10.1115/1.4032070}}</ref>.

This results in a very convenient form for the bounds of the IPM, and henceHence the IPM can be trained with a linear optimization program:

:<math>

\operatorname{arg\,min}_p{ \left\{\mathbb{E}_x(\bar{y}_p(x) - \underline{y}_p(x)) : \bar{y}_p(x^{(i)}) > y^{(i)} > \underline{y}_p(x^{(i)}), i=1,...\ldots,N \right\}}

</math>

where the training data examples are <math> y^{(i)}</math> and <math> x^{(i)}</math>, and the Interval Predictor Model bounds <math> \underline{y}_p(x)</math> and <math>\overline{y}_p(x) </math> are parameterised by the parameter vector <math> p </math>.

The reliability of such an IPM is obtained by noting that for a convex IPM the number of support constraints is less than the dimensionality of the [[trainable parametersparameter]]s, and hence the scenario approach can be applied.

Lacerda (2017) demonstrated that this approach can be extended to situations where the training data is interval valued rather than point valued .<ref name="LacerdaCrespo2017">{{cite journalbook|last1=Lacerda|first1=Marcio J.|title=2017 American Control Conference (ACC)|last2=Crespo|first2=Luis G.|titlechapter=Interval predictor models for data with measurement uncertainty|year=2017|pages=1487–1492|doi=10.23919/ACC.2017.7963163|isbn=978-1-5090-5992-8|hdl=2060/20170005690|s2cid=3713493 }}</ref>.

== Non-convex Intervalinterval Predictorpredictor Modelsmodels ==

In Campi (2015) a non-convex theory of scenario optimization was proposed .<ref name="CampiGaratti2015">{{cite journalbook|last1=Campi|first1=Marco C.|title=2015 54th IEEE Conference on Decision and Control (CDC)|last2=Garatti|first2=Simone|last3=Ramponi|first3=Federico A.|titlechapter=Non-convex scenario optimization with application to system identification|year=2015|pages=4023–4028|doi=10.1109/CDC.2015.7402845|isbn=978-1-4799-7886-1|s2cid=127406 }}</ref>.

This involves measuring the number of support constraints, <math>S</math>, for the Interval Predictor Model after training and hence making predictions about the reliability of the model.

This enables non-convex IPMs to be created, such as a single layer neural network.

Campi (2015) demonstrates that an algorithm withwhere costthe scenario optimization program is only solved <math>\mathcal{O}(S)</math> times which can determine the reliability of the model at test time in anwithout a-priori fashion (i.e. without anprior evaluation on a validation set) .<ref name="CampiGaratti2015" />.

This is achieved by minimizingsolving the max-erroroptimisation loss function given byprogram

:<math>

\mathcal{L}_operatorname{arg\text{max-error,min}} =_p \max_left\{i}h : |y^{(i)}-\hat{y}_p(x^{(i)}) - y^{(i)}| < h, i=1,\ldots,N\right\},

</math>

where the Intervalinterval Predictorpredictor Modelmodel center line <math> \hat{y}_p(x) = (\overline{y}_p(x) + \underline{y}_p(x)) \times 1/2</math>, and the model width <math> h = (\overline{y}_p(x) - \underline{y}_p(x)) \times 1/2 </math>. This results in an IPM which makes predictions with homoscedastic uncertainty.

This results in an IPM which makes predictions with homoscedastic uncertainty.

Sadeghi (2019) demonstrates that the non-convex scenario approach from Campi (2015) can be extended to train deeper Neuralneural Networksnetworks which predict intervals with hetreoscedastic uncertainty on datasets with Imprecision imprecision.<ref name="Sadeghi2019">{{cite journal|last1=Sadeghi|first1=Jonathan C.|last2=De Angelis|first2=Marco|last3=Patelli|first3=Edoardo|title=Efficient Training of Interval Neural Networks for Imprecise Training Data|year=2019|journal=Neural Networks|volume=118|pages=338–351|doi=10.1016/j.neunet.2019.07.005|pmid=31369950|s2cid=199383010 |url=https://strathprints.strath.ac.uk/71230/ |url-access=subscription}}</ref>.

This is achieved by proposing generalizations to the max-error loss function given by

:<math>

\mathcal{L}_{\text{max-error}} = \max_i |y^{(i)}-\hat{y}_p(x^{(i)})|,

</math>

which is equivalent to solving the optimisation program proposed by Campi (2015).

== Applications ==

Initially, [[Scenarioscenario Optimizationoptimization]] was applied to robust control problems .<ref name="CampiGaratti2009">{{cite journal|last1=Campi|first1=Marco C.|last2=Garatti|first2=Simone|last3=Prandini|first3=Maria|author3-link= Maria Prandini |title=The scenario approach for systems and control design|journal=Annual Reviews in Control|volume=33|issue=2|year=2009|pages=149–157|issn=136757881367-5788|doi=10.1016/j.arcontrol.2009.07.001}}</ref>

.

Crespo (2015) applied Interval Predictor Models to the design of space radiation shielding <ref name="CrespoKenny2016a">{{cite journal|last1=Crespo|first1=Luis G.|last2=Kenny|first2=Sean P.|last3=Giesy|first3=Daniel P.|last4=Norman|first4=Ryan B.|last5=Blattnig|first5=Steve|title=Application of Interval Predictor Models to Space Radiation Shielding|year=2016|doi=10.2514/6.2016-0431}}</ref>

.

In Patelli (2017), Faes (2019), and Crespo (2018), Interval Predictor models were applied to the [[Structural reliability]] analysis problem <ref name="PatelliBroggi2017">{{cite journal|last1=Patelli|first1=Edoardo|last2=Broggi|first2=Matteo|last3=Tolo|first3=Silvia|last4=Sadeghi|first4=Jonathan|title=COSSAN SOFTWARE: A MULTIDISCIPLINARY AND COLLABORATIVE SOFTWARE FOR UNCERTAINTY QUANTIFICATION|year=2017|pages=212–224|doi=10.7712/120217.5364.16982}}</ref>

<ref name="CrespoKenny2018">{{cite journal|last1=Crespo|first1=Luis G.|last2=Kenny|first2=Sean P.|last3=Giesy|first3=Daniel P.|title=Staircase predictor models for reliability and risk analysis|journal=Structural Safety|volume=75|year=2018|pages=35–44|issn=01674730|doi=10.1016/j.strusafe.2018.05.002}}</ref>

<ref name="FaesSadeghi2019">{{cite journal|last1=Faes|first1=Matthias|last2=Sadeghi|first2=Jonathan|last3=Broggi|first3=Matteo|last4=De Angelis|first4=Marco|last5=Patelli|first5=Edoardo|last6=Beer|first6=Michael|last7=Moens|first7=David|title=On the robust estimation of small failure probabilities for strong non-linear models|journal=ASCE-ASME J. Risk and Uncert. in Engrg. Sys., Part B: Mech. Engrg.|year=2019|issn=2332-9017|doi=10.1115/1.4044044}}</ref>.

Brandt (2017) applies Interval Predictor Models to fatigue damage estimation of offshore wind turbines jacket substructures <ref name="BrandtBroggi2017">{{cite journal|last1=Brandt|first1=Sebastian|last2=Broggi|first2=Matteo|last3=Hafele|first3=Jan|last4=Guillermo Gebhardt|first4=Cristian|last5=Rolfes|first5=Raimund|last6=Beer|first6=Michael|title=Meta-models for fatigue damage estimation of offshore wind turbines jacket substructures|journal=Procedia Engineering|volume=199|year=2017|pages=1158–1163|issn=18777058|doi=10.1016/j.proeng.2017.09.292}}</ref>.

Crespo (2015) and (2021) applied Interval Predictor Models to the design of space radiation shielding <ref name="CrespoKenny2016a">{{cite book|last1=Crespo|first1=Luis G.|title=18th AIAA Non-Deterministic Approaches Conference|last2=Kenny|first2=Sean P.|last3=Giesy|first3=Daniel P.|last4=Norman|first4=Ryan B.|last5=Blattnig|first5=Steve|chapter=Application of Interval Predictor Models to Space Radiation Shielding|year=2016|doi=10.2514/6.2016-0431|isbn=978-1-62410-397-1|hdl=2060/20160007750|s2cid=124192684 }}</ref> and to system identification.<ref>{{cite book | doi=10.1109/CDC45484.2021.9683582 | chapter=Interval Predictor Models for Robust System Identification | title=2021 60th IEEE Conference on Decision and Control (CDC) | year=2021 | last1=Crespo | first1=Luis G. | last2=Kenny | first2=Sean P. | last3=Colbert | first3=Brendon K. | last4=Slagel | first4=Tanner | pages=872–879 | isbn=978-1-6654-3659-5 | s2cid=246479771 }}</ref>

== Software Implementations ==

PyIPM provides an [[Open source]] Matlab and Python implementation of the work of Crespo (2015) <ref>{{cite journal|doi=10.5281/zenodo.2784750|title="PyIPM"}}</ref>.

In Patelli (2017), Faes (2019), and Crespo (2018), Interval Predictor models were applied to the [[Structuralstructural reliability]] analysis problem .<ref name="PatelliBroggi2017">{{cite journalbook|last1=Patelli|first1=Edoardo|title=Proceedings of the 2nd International Conference on Uncertainty Quantification in Computational Sciences and Engineering (UNCECOMP 2017)|last2=Broggi|first2=Matteo|last3=Tolo|first3=Silvia|last4=Sadeghi|first4=Jonathan|title=COSSAN SOFTWARE: A MULTIDISCIPLINARY AND COLLABORATIVE SOFTWARE FOR UNCERTAINTY QUANTIFICATION|year=2017|pages=212–224|doi=10.7712/120217.5364.16982|chapter=Cossan Software: A Multidisciplinary and Collaborative Software for Uncertainty Quantification|isbn=978-618-82844-4-9}}</ref>

A similar implementation is available in the [http://cossan.co.uk/software/open-cossan-engine.php OpenCOSSAN] software <ref name="PatelliBroggi2017" />.

<ref name="CrespoKenny2018"/>

<ref name="FaesSadeghi2019">{{cite journal|last1=Faes|first1=Matthias|last2=Sadeghi|first2=Jonathan|last3=Broggi|first3=Matteo|last4=De Angelis|first4=Marco|last5=Patelli|first5=Edoardo|last6=Beer|first6=Michael|last7=Moens|first7=David|title=On the robust estimation of small failure probabilities for strong non-linear models|journal=ASCE-ASME J.Journal of Risk and Uncert.Uncertainty in Engrg.Engineering Sys.Systems, Part B: Mech.Mechanical Engrg.Engineering|volume=5|issue=4|year=2019|issn=2332-9017|doi=10.1115/1.4044044|s2cid=197472507 |url=https://lirias.kuleuven.be/handle/123456789/633621 |url-access=subscription}}</ref>.

Brandt (2017) applies Intervalinterval Predictorpredictor Modelsmodels to fatigue damage estimation of offshore wind turbines jacket substructures .<ref name="BrandtBroggi2017">{{cite journal|last1=Brandt|first1=Sebastian|last2=Broggi|first2=Matteo|last3=Hafele|first3=Jan|last4=Guillermo Gebhardt|first4=Cristian|last5=Rolfes|first5=Raimund|last6=Beer|first6=Michael|title=Meta-models for fatigue damage estimation of offshore wind turbines jacket substructures|journal=Procedia Engineering|volume=199|year=2017|pages=1158–1163|issn=187770581877-7058|doi=10.1016/j.proeng.2017.09.292|doi-access=free}}</ref>.

Garatti (2019) proved that Chebyshev layers (i.e., the minimax layers around functions fitted by linear <math>\ell_\infty</math>-regression) belong to a particular class of Interval Predictor Models, for which the reliability is invariant with respect to the distribution of the data.<ref name="GarCamCar2019">{{cite journal|last1=Garatti|first1=S.|last2=Campi|first2=M.C.|last3=Carè|first3=A.|title=On a class of Interval Predictor Models with universal reliability|journal=Automatica|volume=110|year=2019|page=108542|issn=0005-1098|doi=10.1016/j.automatica.2019.108542|hdl=11311/1121161 |s2cid=204188183 |hdl-access=free}}</ref>

== Software Implementationsimplementations ==

OpenCOSSAN provides a Matlab implementation of the work of Crespo (2015).<ref name="PatelliBroggi2017" />

== References ==

{{Reflist}}

{{least squares and regression analysis}}

{{Statistics|correlation|state=collapsed}}

[[Category:Regression analysis| ]]