Test oracle: Difference between revisions

Revision as of 02:29, 12 December 2019

Revision as of 10:27, 16 February 2020

Line 1:

{{other uses2|oracle}}

In [[computing]], [[software engineering]], and [[software testing]], a '''test oracle''', (or just '''oracle''',) is a mechanism for determining whether a test has passed or failed.<ref>Kaner, Cem; [http://www.testingeducation.org/k04/OracleExamples.htm ''A Course in Black Box Software Testing''], 2004</ref> The use of oracles involves comparing the output(s) of the system under test, for a given [[test case|test-case]] input, to the output(s) that the oracle determines that product should have. The term "test oracle" was first introduced in a paper by William E. Howden.<ref>{{cite journal |last1=Howden |first1=W.E. |date=July 1978 |title=Theoretical and Empirical Studies of Program Testing |journal=IEEE Transactions on Software Engineering |volume=4 |issue=4 |pages=293–298 |doi=10.1109/TSE.1978.231514 }}</ref> Additional work on different kinds of oracles was explored by [[Elaine Weyuker]].<ref>Weyuker, Elaine J.; "The Oracle Assumption of Program Testing", in ''Proceedings of the 13th International Conference on System Sciences (ICSS), Honolulu, HI, January 1980'', pp. 44-49</ref>

Oracles often operate separately from the system under test.<ref name="038720881X">Jalote, Pankaj; ''An Integrated Approach to Software Engineering'', Springer/Birkhäuser, 2005, {{ISBN|0-387-20881-X}}</ref> However, [[Method (computer programming)|method]] postconditions are part of the system under test, as automated oracles in [[design by contract]] models.<ref>{{cite journal |last1=Meyer |first1=Bertrand |last2=Fiva |first2=Arno |last3=Ciupa |first3=Ilinca |last4=Leitner |first4=Andreas |last5=Wei |first5=Yi |last6=Stapf |first6=Emmanuel |date=September 2009 |title=Programs That Test Themselves |journal=Computer |volume=42 |issue=9 |pages=46–55 |doi= 10.1109/MC.2009.296 }}</ref> Determining the correct output for a given input (and a set of program/system states) is known as the '''oracle problem''' or '''test oracle problem'''<ref name="Oracle survey"/>{{rp|507}}, which is a much harder problem than it seems, and involves working with problems related to controllability and observability.<ref name="ammann-intro">Ammann, Paul; and Offutt, Jeff; "Introduction to Software Testing", ''Cambridge University Press'', 2008, {{ISBN|978-0-521-88038-1}}</ref> Various methods have been proposed to alleviate the test oracle problem. A popular technique is [[metamorphic testing]].<ref>{{cite journal | first1=Sergio | last1=Segura | first2=Gordon | last2=Fraser | first3=Ana B. | last3=Sanchez | first4=Antonio | last4=Ruiz-Cortes | title=A survey on metamorphic testing | journal=[[IEEE Transactions on Software Engineering]] | volume=42 | issue=9 | pages=805–824 | date=2016 | doi=10.1109/TSE.2016.2532875| hdl=11441/38271 }}</ref><ref>{{cite journal | first1=Tsong Yueh | last1=Chen | first2=Fei-Ching | last2=Kuo | first3=Huai | last3=Liu | first4=Pak-Lok | last4=Poon | first5=Dave | last5=Towey | first6=T.H. | last6=Tse | first7= Zhi Quan | last7=Zhou | title=Metamorphic testing: A review of challenges and opportunities | journal=[[ACM Computing Surveys]] | volume=51 | issue=1 | pages=4:1–4:27 | date=2018 | doi=10.1145/3143561| url=http://eprints.nottingham.ac.uk/51607/1/__MTChallOpporCSUR.accepted.20170922.pdf }}</ref>