Data validation and reconciliation: Difference between revisions

Revision as of 20:45, 17 March 2018

Latest revision as of 02:24, 18 August 2025

(21 intermediate revisions by 17 users not shown)

Line 1:

{{Short description|A technology to correct measurements in industrial processes}}

'''DVR''' is a program on Computer Store, airing from 4:00 to 5:00 a.m. Eastern Time weekends, 9:00 to 11:00 p.m. Eastern Time on Saturdays, 5:30 to 8:00 p.m. Eastern on Sundays and 10:00 p.m. to 2:00 a.m. (11:00 p.m. to 3:00 a.m. during March and October when [[New York City|New York]] is four hours behind [[London]]) on Wednesdays. It debuted in 2011, then moved to weekend primetime in 2015 before moving to early weekends on October 10, 2015, but the sucesos has put the show in timeslots outside of Saturday and Sunday.

'''Industrial process data validation and reconciliation''', or more briefly, '''process data reconciliation (PDR)''', is a technology that uses process information and mathematical methods in order to automatically ensure [[data validation]] and reconciliation by correcting measurements in industrial processes. The use of PDR allows for extracting accurate and reliable information about the state of industry processes from raw measurement [[data]] and produces a single consistent set of data representing the most likely process operation.

==Models, data and measurement errors==

Line 12 ⟶ 13:

File:Normal_with_bias.jpg|Normally distributed measurements with bias.

</gallery>

Data originates typically from [[measurements]] taken at different places throughout the industrial site, for example temperature, pressure, volumetric flow rate measurements etc. To understand the basic principles of DVRPDR, it is important to first recognize that plant measurements are never 100% correct, i.e. raw measurement <math>y\,</math> is not a solution of the nonlinear system <math>F(y)=0\,\!</math>. When using measurements without correction to generate plant balances, it is common to have incoherencies. [[Observational error|Measurement errors]] can be categorized into two basic types:

# [[random error]]s due to intrinsic [[sensor]] [[accuracy]] and

# [[systematic errors]] (or gross errors) due to sensor [[calibration]] or faulty data transmission.

Line 31 ⟶ 32:

==History==

DVRPDR has become more and more important due to industrial processes that are becoming more and more complex. DVRPDR started in the early 1960s with applications aiming at closing [[mass balance|material balances]] in production processes where raw measurements were available for all [[variable (mathematics)|variables]].<ref>D.R. Kuehn, H. Davidson, ''Computer Control II. Mathematics of Control'', Chem. Eng. Process 57: 44–47, 1961.</ref> At the same time the problem of [[systematic error|gross error]] identification and elimination has been presented.<ref>V. Vaclavek, ''Studies on System Engineering I. On the Application of the Calculus of the Observations of Calculations of Chemical Engineering Balances'', Coll. Czech Chem. Commun. 34: 3653, 1968.</ref> In the late 1960s and 1970s unmeasured variables were taken into account in the data reconciliation process.,<ref>V. Vaclavek, M. Loucka, ''Selection of Measurements Necessary to Achieve Multicomponent Mass Balances in Chemical Plant'', Chem. Eng. Sci. 31: 1199–1205, 1976.</ref><ref name="Mah-Stanley-Downing-1976">[http://gregstanleyandassociates[Richard S.com/ReconciliationRectificationProcessData-1976 H.pdf Mah|R.S.H. Mah]], G.M. Stanley, D.W. Downing, [http://gregstanleyandassociates.com/ReconciliationRectificationProcessData-1976.pdf ''Reconciliation and Rectification of Process Flow and Inventory Data'', Ind. & Eng. Chem. Proc. Des. Dev. 15: 175–183, 1976.]</ref> DVRPDR also became more mature by considering general nonlinear equation systems coming from thermodynamic models.,<ref>J.C. Knepper, J.W. Gorman, ''Statistical Analysis of Constrained Data Sets'', AiChE Journal 26: 260–164, 1961.</ref>

,<ref name="Stanley-Mah-1977">G.M. Stanley and R.S.H. Mah, [http://gregstanleyandassociates.com/AIChEJ-1977-EstimationInProcessNetworks.pdf ''Estimation of Flows and Temperatures in Process Networks'', AIChE Journal 23: 642–650, 1977.]</ref><ref>P. Joris, B. Kalitventzeff, ''Process measurements analysis and validation'', Proc. CEF’87: Use Comput. Chem. Eng., Italy, 41–46, 1987.</ref> Quasi steady state dynamics for filtering and simultaneous parameter estimation over time were introduced in 1977 by Stanley and Mah.<ref name="Stanley-Mah-1977"/> Dynamic DVRPDR was formulated as a nonlinear optimization problem by Liebman et al. in 1992.<ref>M.J. Liebman, T.F. Edgar, L.S. Lasdon, ''Efficient Data Reconciliation and Estimation for Dynamic Processes Using Nonlinear Programming Techniques'', Computers Chem. Eng. 16: 963–986, 1992.</ref>

,<ref name="Stanley-Mah-1977">[http://gregstanleyandassociates.com/AIChEJ-1977-EstimationInProcessNetworks.pdf G.M. Stanley and R.S.H. Mah, ''Estimation of Flows and Temperatures in Process Networks'', AIChE Journal 23: 642–650, 1977.]</ref>

<ref>P. Joris, B. Kalitventzeff, ''Process measurements analysis and validation'', Proc. CEF’87: Use Comput. Chem. Eng., Italy, 41–46, 1987.</ref> Quasi steady state dynamics for filtering and simultaneous parameter estimation over time were introduced in 1977 by Stanley and Mah.<ref name="Stanley-Mah-1977"/> Dynamic DVR was formulated as a nonlinear optimization problem by Liebman et al. in 1992.<ref>M.J. Liebman, T.F. Edgar, L.S. Lasdon, ''Efficient Data Reconciliation and Estimation for Dynamic Processes Using Nonlinear Programming Techniques'', Computers Chem. Eng. 16: 963–986, 1992.</ref>

==Data reconciliation==

Line 55:

The term <math>\left(\frac{y_i^*-y_i}{\sigma_i}\right)^2\,\!</math> is called the ''penalty'' of measurement ''i''. The objective function is the sum of the penalties, which will be denoted in the following by <math>f(y^*)=\sum_{i=1}^n\left(\frac{y_i^*-y_i}{\sigma_i}\right)^2</math>.

In other words, one wants to minimize the overall correction (measured in the least squares term) that is needed in order to satisfy the [[constraint (mathematics)|system constraints]]. Additionally, each least squares term is weighted by the [[standard deviation]] of the corresponding measurement. The standard deviation is related to the accuracy of the measurement. For example, at a 95% confidence level, the standard deviation is about half the accuracy.

===Redundancy===

Line 66:

Redundancy can be due to [[redundancy (engineering)|sensor redundancy]], where sensors are duplicated in order to have more than one measurement of the same quantity. Redundancy also arises when a single variable can be estimated in several independent ways from separate sets of measurements at a given time or time averaging period, using the algebraic constraints.

Redundancy is linked to the concept of [[observability]]. A variable (or system) is observable if the models and sensor measurements can be used to uniquely determine its value (system state). A sensor is redundant if its removal causes no loss of observability. Rigorous definitions of observability, calculability, and redundancy, along with criteria for determining it, were established by Stanley and Mah,<ref name="Stanley-Mah-1981a">

[httphttps://gregstanleyandassociates.com/whitepapers/DataRec/CES-1981a-ObservabilityRedundancy.pdf Stanley G.M. and Mah, R.S.H., "Observability and Redundancy in Process Data Estimation, Chem. Engng. Sci. 36, 259 (1981)]</ref> for these cases with set constraints such as algebraic equations and inequalities. Next, we illustrate some special cases:

Topological redundancy is intimately linked with the [[degrees of freedom (physics and chemistry)|degrees of freedom]] (<math>dof\,\!</math>) of a mathematical system,<ref name="vdi">VDI-Gesellschaft Energie und Umwelt, "Guidelines - VDI 2048 Blatt 1 - Uncertainties“Control and quality improvement of measurementsprocess atdata acceptanceand teststheir foruncertainties energyby means of correction calculation for conversionoperation and poweracceptance plantstests”; -VDI 2048 Part 1; September Fundamentals2017", ''[http://www.vdi.de/401.0.html Association of German Engineers] {{Webarchive|url=https://web.archive.org/web/20100325223512/http://www.vdi.de/401.0.html |date=2010-03-25 }}'', 20002017.</ref> i.e. the minimum number of pieces of information (i.e. measurements) that are required in order to calculate all of the system variables. For instance, in the example above the flow conservation requires that <math>a=b+c\,</math>. One needs to know the value of two of the 3 variables in order to calculate the third one. The degrees of freedom for the model in that case is equal to 2. At least 2 measurements are needed to estimate all the variables, and 3 would be needed for redundancy.

When speaking about topological redundancy we have to distinguish between measured and unmeasured variables. In the following let us denote by <math>x\,\!</math> the unmeasured variables and <math>y\,\!</math> the measured variables. Then the system of the process constraints becomes <math>F(x,y)=0\,\!</math>, which is a nonlinear system in <math>y\,\!</math> and <math>x\,\!</math>.

Line 90:

We incorporate only flow conservation constraints and obtain <math>a+b=c\,\!</math> and <math>c=d\,\!</math>. It is possible that the system <math>F(x,y)=0\,\!</math> is not calculable, even though <math>p-m\ge 0\,\!</math>.

If we have measurements for <math>c\,\!</math> and <math>d\,\!</math>, but not for <math>a\,\!</math> and <math>b\,\!</math>, then the system cannot be calculated (knowing <math>c\,\!</math> does not give information about <math>a\,\!</math> and <math>b\,\!</math>). On the other hand, if <math>a\,\!</math> and <math>cd\,\!</math> are known, but not <math>b\,\!</math> and <math>dc\,\!</math>, then the system can be calculated.

In 1981, observability and redundancy criteria were proven for these sorts of flow networks involving only mass and energy balance constraints.<ref name="Stanley-Mah-1981b">[httphttps://gregstanleyandassociates.com/whitepapers/DataRec/CES-1981b-ObservabilityRedundancyProcessNetworks.pdf Stanley G.M., and Mah R.S.H., "Observability and Redundancy Classification in Process Networks", Chem. Engng. Sci. 36, 1941 (1981) ]</ref> After combining all the plant inputs and outputs into an "environment node", loss of observability corresponds to cycles of unmeasured streams. That is seen in the second case above, where streams a and b are in a cycle of unmeasured streams. Redundancy classification follows, by testing for a path of unmeasured streams, since that would lead to an unmeasured cycle if the measurement was removed. Measurements c and d are redundant in the second case above, even though part of the system is unobservable.

===Benefits===

Line 115:

The individual test compares each penalty term in the objective function with the critical values of the normal distribution. If the <math>i</math>-th penalty term is outside the 95% confidence interval of the normal distribution, then there is reason to believe that this measurement has a gross error.

==Advanced process data validation and reconciliation==

Advanced process data validation and reconciliation (DVRPDR) is an integrated approach of combining data reconciliation and data validation techniques, which is characterized by

* complex models incorporating besides mass balances also thermodynamics, momentum balances, equilibria constraints, hydrodynamics etc.

* gross error remediation techniques to ensure meaningfulness of the reconciled values,

Line 126:

===Gross error remediation===

[[image:scheme reconciliation.jpg|thumb|350px|The workflow of an advanced data validation and reconciliation process.]]

Gross errors are measurement systematic errors that may [[bias]] the reconciliation results. Therefore, it is important to identify and eliminate these gross errors from the reconciliation process. After the reconciliation [[statistical tests]] can be applied that indicate whether or not a gross error does exist somewhere in the set of measurements. These techniques of gross error remediation are based on two concepts:

* gross error elimination

* gross error relaxation.

Line 133:

Gross error relaxation targets at relaxing the estimate for the uncertainty of suspicious measurements so that the reconciled value is in the 95% confidence interval. Relaxation typically finds application when it is not possible to determine which measurement around one unit is responsible for the gross error (equivalence of gross errors). Then measurement uncertainties of the measurements involved are increased.

It is important to note that the remediation of gross errors reduces the quality of the reconciliation, either the redundancy decreases (elimination) or the uncertainty of the measured data increases (relaxation). Therefore, it can only be applied when the initial level of redundancy is high enough to ensure that the data reconciliation can still be done (see Section 2,<ref name="vdi" />).

===Workflow===

Advanced DVRPDR solutions offer an integration of the techniques mentioned above:

# data acquisition from data historian, data base or manual inputs

# data validation and filtering of raw measurements

Line 144:

#* gross error remediation (and go back to step 3)

# result storage (raw measurements together with reconciled values)

The result of an advanced DVRPDR procedure is a coherent set of validated and reconciled process data.

==Applications==

DVRPDR finds application mainly in industry sectors where either measurements are not accurate or even non-existing, like for example in the [[upstream (fossil-fuel industry)|upstream sector]] where [[flow measurement|flow meters]] are difficult or expensive to position (see <ref>P. Delava, E. Maréchal, B. Vrielynck, B. Kalitventzeff (1999), ''Modelling of a Crude Oil Distillation Unit in Term of Data Reconciliation with ASTM or TBP Curves as Direct Input – Application : Crude Oil Preheating Train'', Proceedings of ESCAPE-9 conference, Budapest, May 31-June 2, 1999, supplementary volume, p. 17-20.</ref>); or where accurate data is of high importance, for example for security reasons in [[nuclear power plants]] (see <ref>M. Langenstein, J. Jansky, B. Laipple (2004), ''Finding Megawatts in nuclear power plants with process data validation'', Proceedings of ICONE12, Arlington, USA, April 25–29, 2004.</ref>). Another field of application is [[Performance test (assessment)|performance and process monitoring]] (see <ref>Th. Amand, G. Heyen, B. Kalitventzeff, ''Plant Monitoring and Fault Detection: Synergy between Data Reconciliation and Principal Component Analysis'', Comp. and Chem, Eng. 25, p. 501-507, 2001.</ref>) in oil refining or in the chemical industry.

As DVRPDR enables to calculate estimates even for unmeasured variables in a reliable way, the German Engineering Society (VDI Gesellschaft Energie und Umwelt) has accepted the technology of DVRPDR as a means to replace expensive sensors in the nuclear power industry (see VDI norm 2048,<ref name="vdi" />).

==See also==

Line 161:

{{Reflist}}

* Alexander, Dave, Tannar, Dave & Wasik, Larry "Mill Information System uses Dynamic Data Reconciliation for Accurate Energy Accounting" TAPPI Fall Conference 2007.[http://www.tappi.org/Downloads/Conference-Papers/2007/07EPE/07epe87.aspx]{{Dead link|date=July 2019 |bot=InternetArchiveBot |fix-attempted=yes }}

* Rankin, J. & Wasik, L. "Dynamic Data Reconciliation of Batch Pulping Processes (for On-Line Prediction)" PAPTAC Spring Conference 2009.

* S. Narasimhan, C. Jordache, ''Data reconciliation and gross error detection: an intelligent use of process data'', Golf Publishing Company, Houston, 2000.

* V. Veverka, F. Madron, ''Material and Energy Balancing in the Process Industries'', Elsevier Science BV, Amsterdam, 1997.

* J. Romagnoli, M.C. Sanchez, ''Data processing and reconciliation for chemical process operations'', Academic Press, 2000.

{{DEFAULTSORT:Data Validation And Reconciliation}}

[[Category:Data management]]