Content deleted Content added
m Undid my revision 941061048. I am sorry to have made a mistake. |
m Open access bot: hdl, doi added to citation with #oabot. |
||
Line 3:
In [[computing]], [[software engineering]], and [[software testing]], a '''test oracle''' (or just '''oracle''') is a mechanism for determining whether a test has passed or failed.<ref>Kaner, Cem; [http://www.testingeducation.org/k04/OracleExamples.htm ''A Course in Black Box Software Testing''], 2004</ref> The use of oracles involves comparing the output(s) of the system under test, for a given [[test case|test-case]] input, to the output(s) that the oracle determines that product should have. The term "test oracle" was first introduced in a paper by William E. Howden.<ref>{{cite journal |last1=Howden |first1=W.E. |date=July 1978 |title=Theoretical and Empirical Studies of Program Testing |journal=IEEE Transactions on Software Engineering |volume=4 |issue=4 |pages=293–298 |doi=10.1109/TSE.1978.231514 }}</ref> Additional work on different kinds of oracles was explored by [[Elaine Weyuker]].<ref>Weyuker, Elaine J.; "The Oracle Assumption of Program Testing", in ''Proceedings of the 13th International Conference on System Sciences (ICSS), Honolulu, HI, January 1980'', pp. 44-49</ref>
Oracles often operate separately from the system under test.<ref name="038720881X">Jalote, Pankaj; ''An Integrated Approach to Software Engineering'', Springer/Birkhäuser, 2005, {{ISBN|0-387-20881-X}}</ref> However, [[Method (computer programming)|method]] postconditions are part of the system under test, as automated oracles in [[design by contract]] models.<ref>{{cite journal |last1=Meyer |first1=Bertrand |last2=Fiva |first2=Arno |last3=Ciupa |first3=Ilinca |last4=Leitner |first4=Andreas |last5=Wei |first5=Yi |last6=Stapf |first6=Emmanuel |date=September 2009 |title=Programs That Test Themselves |journal=Computer |volume=42 |issue=9 |pages=46–55 |doi= 10.1109/MC.2009.296 }}</ref> Determining the correct output for a given input (and a set of program/system states) is known as the '''oracle problem''' or '''test oracle problem'''<ref name="Oracle survey"/>{{rp|507}}, which is a much harder problem than it seems, and involves working with problems related to controllability and observability.<ref name="ammann-intro">Ammann, Paul; and Offutt, Jeff; "Introduction to Software Testing", ''Cambridge University Press'', 2008, {{ISBN|978-0-521-88038-1}}</ref> Various methods have been proposed to alleviate the test oracle problem. A popular technique is [[metamorphic testing]].<ref>{{cite journal | first1=Sergio | last1=Segura | first2=Gordon | last2=Fraser | first3=Ana B. | last3=Sanchez | first4=Antonio | last4=Ruiz-Cortes | title=A survey on metamorphic testing | journal=[[IEEE Transactions on Software Engineering]] | volume=42 | issue=9 | pages=805–824 | date=2016 | doi=10.1109/TSE.2016.2532875| hdl=11441/38271 | hdl-access=free }}</ref><ref>{{cite journal | first1=Tsong Yueh | last1=Chen | first2=Fei-Ching | last2=Kuo | first3=Huai | last3=Liu | first4=Pak-Lok | last4=Poon | first5=Dave | last5=Towey | first6=T.H. | last6=Tse | first7= Zhi Quan | last7=Zhou | title=Metamorphic testing: A review of challenges and opportunities | journal=[[ACM Computing Surveys]] | volume=51 | issue=1 | pages=4:1–4:27 | date=2018 | doi=10.1145/3143561| url=http://eprints.nottingham.ac.uk/51607/1/__MTChallOpporCSUR.accepted.20170922.pdf }}</ref>
== Categories ==
Line 16:
A derived test oracle differentiates correct and incorrect behaviour by using information derived from artefacts of the system. These may include documentation, system execution results and characteristics of versions of the system under test<ref name="Oracle survey"/>{{rp|514}}. Regression test suites (or reports) are an example of a derived test oracle - they are built on the assumption that the result from a previous system version can be used as aid (oracle) for a future system version. Previously measured performance characteristics may be used as an oracle for future system versions, for example, to trigger a question about observed potential performance degradation. Textual documentation from previous system versions may be used as a basis to guide expectations in future system versions.
A '''pseudo-oracle''' falls into the category<ref name="Oracle survey"/>{{rp|515}} of derived test oracle. A pseudo-oracle, as defined by Weyuker,<ref name="pseudo-oracle">{{cite journal |last1=Weyuker |first1=E.J. |date=November 1982 |title=On Testing Non-Testable Programs |journal=The Computer Journal |volume=25 |issue=4 |pages=465–470 |doi=10.1093/comjnl/25.4.465 |doi-access=free }}</ref> is a separately written program which can take the same input as the program/system under test so that their outputs may be compared to understand if there might be a problem to investigate.
=== Implicit ===
| |||