Test oracle: Difference between revisions

Revision as of 10:37, 19 March 2024

Revision as of 13:40, 26 April 2024

Line 1:

{{Use American English|date=January 2021}}

{{other uses|Oracle (disambiguation)}}

In [[software testing]], a '''test oracle''' (or just '''oracle''') is a provider of information that describes correct output based on the input of a [[test case|test case]]. Testing with an oracle involves comparing actual results of the system under test (SUT) with the expected results as provided by the oracle. <ref>Earl T. Barr et al; ''[https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6963470 The Oracle Problem in Software Testing: A Survey]'', 2015</ref>

In [[computing]], [[software engineering]], and [[software testing]],

given an input for a system a '''test oracle''' (or just '''oracle''') is a procedure that distinguishes between the correct and incorrect behaviors of the System Under Test (SUT).<ref>Earl T. Barr et al; ''[https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6963470 The Oracle Problem in Software Testing: A Survey]'', 2015</ref> The use of oracles involves comparing the output(s) of the system under test, for a given [[test case|test-case]] input, to the output(s) that the oracle determines that product should have. The term "test oracle" was first introduced in a paper by William E. Howden.<ref>{{cite journal |last1=Howden |first1=W.E. |date=July 1978 |title=Theoretical and Empirical Studies of Program Testing |journal=IEEE Transactions on Software Engineering |volume=4 |issue=4 |pages=293–298 |doi=10.1109/TSE.1978.231514 }}</ref> Additional work on different kinds of oracles was explored by [[Elaine Weyuker]].<ref>Weyuker, Elaine J.; "The Oracle Assumption of Program Testing", in ''Proceedings of the 13th International Conference on System Sciences (ICSS), Honolulu, HI, January 1980'', pp. 44-49</ref>

given an input for a system a '''test oracle''' (or just '''oracle''') is a procedure that distinguishes between the correct and incorrect behaviors of the System Under Test (SUT).<ref>Earl T. Barr et al; ''[https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6963470 The Oracle Problem in Software Testing: A Survey]'', 2015</ref> The use of oracles involves comparing the output(s) of the system under test, for a given [[test case|test-case]] input, to the output(s) that the oracle determines that product should have. The term "test oracle" was first introduced in a paper by William E. Howden.<ref>{{cite journal |last1=Howden |first1=W.E. |date=July 1978 |title=Theoretical and Empirical Studies of Program Testing |journal=IEEE Transactions on Software Engineering |volume=4 |issue=4 |pages=293–298 |doi=10.1109/TSE.1978.231514 }}</ref> Additional work on different kinds of oracles was explored by [[Elaine Weyuker]].<ref>Weyuker, Elaine J.; "The Oracle Assumption of Program Testing", in ''Proceedings of the 13th International Conference on System Sciences (ICSS), Honolulu, HI, January 1980'', pp. 44-49</ref>

Oracles often operate separately from the system under test.<ref name="038720881X">Jalote, Pankaj; ''An Integrated Approach to Software Engineering'', Springer/Birkhäuser, 2005, {{ISBN|0-387-20881-X}}</ref> However, [[Method (computer programming)|method]] postconditions are part of the system under test, as automated oracles in [[design by contract]] models.<ref>{{cite journal |last1=Meyer |first1=Bertrand |last2=Fiva |first2=Arno |last3=Ciupa |first3=Ilinca |last4=Leitner |first4=Andreas |last5=Wei |first5=Yi |last6=Stapf |first6=Emmanuel |date=September 2009 |title=Programs That Test Themselves |journal=Computer |volume=42 |issue=9 |pages=46–55 |doi= 10.1109/MC.2009.296 }}</ref> Determining the correct output for a given input (and a set of program or system states) is known as the '''oracle problem''' or test oracle problem,<ref name="Oracle survey"/>{{rp|507}} which is a much harder problem than it seems, and involves working with problems related to controllability and observability.<ref name="ammann-intro">Ammann, Paul; and Offutt, Jeff; "Introduction to Software Testing, 2nd edition", ''Cambridge University Press'', 2016, {{ISBN|978-1107172012}}</ref>

An oracle can operate separately from the SUT; accessed at test [[Runtime (program lifecycle phase)|runtime]], or it can be used before a test is run with expected results encoded into the test logic.<ref name="038720881X">Jalote, Pankaj; ''An Integrated Approach to Software Engineering'', Springer/Birkhäuser, 2005, {{ISBN|0-387-20881-X}}</ref>

However, [[Method (computer programming)|method]] postconditions are part of the SUT, as automated oracles in [[design by contract]] models.<ref>{{cite journal |last1=Meyer |first1=Bertrand |last2=Fiva |first2=Arno |last3=Ciupa |first3=Ilinca |last4=Leitner |first4=Andreas |last5=Wei |first5=Yi |last6=Stapf |first6=Emmanuel |date=September 2009 |title=Programs That Test Themselves |journal=Computer |volume=42 |issue=9 |pages=46–55 |doi= 10.1109/MC.2009.296 }}</ref>

Determining the correct output for a given input (and a set of program or system states) is known as the ''oracle problem'' or ''test oracle problem'',<ref name="Oracle survey"/>{{rp|507}} which some consider a relatively hard problem, and involves working with problems related to controllability and observability.<ref name="ammann-intro">Ammann, Paul; and Offutt, Jeff; "Introduction to Software Testing, 2nd edition", ''Cambridge University Press'', 2016, {{ISBN|978-1107172012}}</ref>

== Categories ==

Line 12 ⟶ 17:

=== Specified ===

TheseA oraclesspecified areoracle is typically associated with formalized approaches to software modeling and software code construction. TheyIt areis connected to [[formal specification]],<ref>{{cite book |last1=Börger |first1=E |title=Applied Formal Methods — FM-Trends 98 |chapter=High Level System Design and Analysis Using Abstract State Machines |editor-last1=Hutter |editor-first1=D |editor-last2=Stephan |editor-first2=W |editor-last3=Traverso |editor-first3=P |editor-last4=Ullman |editor-first4=M |date=1999|volume=1641 |pages=1–43 |doi=10.1007/3-540-48257-1_1 |series=Lecture Notes in Computer Science |isbn=978-3-540-66462-8 |citeseerx=10.1.1.470.3653 }}</ref> [[model-based design]] which may be used to generate test oracles,<ref>{{cite journal |last1=Peters |first1=D.K. |date=March 1998 |title=Using test oracles generated from program documentation |journal=IEEE Transactions on Software Engineering |volume=24 |issue=3 |pages=161–173 |doi=10.1109/32.667877 |citeseerx=10.1.1.39.2890 }}</ref> state transition specification for which oracles can be derived to aid [[model-based testing]]<ref>{{cite journal| author-last1=Utting |author-first1=Mark |author-last2=Pretschner |author-first2=Alexander |author-last3=Legeard |author-first3=Bruno |title = A taxonomy of model-based testing approaches |journal = Software Testing, Verification and Reliability |volume= 22|issue= 5 |issn= 1099-1689|doi=10.1002/stvr.456 |pages= 297–312|year=2012 |url=https://eprints.qut.edu.au/57853/1/master_pdflatex.pdf }}</ref> and [[conformance testing|protocol conformance testing]],<ref>{{cite book|author-link1=Marie-Claude Gaudel |last1=Gaudel |first1=Marie-Claude |title=Reliable SoftwareTechnologies — Ada-Europe 2001 |chapter=Testing from Formal Specifications, a Generic Approach |editor-last1=Craeynest |editor-first1=D.|editor-last2=Strohmeier |editor-first2=A|date=2001 |volume=2043 |pages=35–48 |doi=10.1007/3-540-45136-6_3 |series=Lecture Notes in Computer Science |isbn=978-3-540-42123-8 }}</ref> and [[design by contract]] for which the equivalent test oracle is an [[assertion (software development)|assertion]].

Specified Testtest Oraclesoracles have a number of challenges. Formal specification relies on abstraction, which in turn may naturally have an element of imprecision as all models cannot capture all behavior.<ref name="Oracle survey"/>{{rp|514}}

=== Derived ===

A derived test oracle differentiates correct and incorrect behavior by using information derived from artifacts of the system. These may include documentation, system execution results and characteristics of versions of the system under testSUT.<ref name="Oracle survey"/>{{rp|514}} Regression test suites (or reports) are an example of a derived test oracle - they are built on the assumption that the result from a previous system version can be used as aid (oracle) for a future system version. Previously measured performance characteristics may be used as an oracle for future system versions, for example, to trigger a question about observed potential performance degradation. Textual documentation from previous system versions may be used as a basis to guide expectations in future system versions.

A pseudo-oracle<ref name="Oracle survey"/>{{rp|515}} falls into the category of derived test oracle. A pseudo-oracle, as defined by Weyuker,<ref name="pseudo-oracle">{{cite journal |last1=Weyuker |first1=E.J. |date=November 1982 |title=On Testing Non-Testable Programs |journal=The Computer Journal |volume=25 |issue=4 |pages=465–470 |doi=10.1093/comjnl/25.4.465 |doi-access=free }}</ref> is a separately written program which can take the same input as the program or system under testSUT so that their outputs may be compared to understand if there might be a problem to investigate.

A partial oracle<ref name="Oracle survey"/>{{rp|515}} is a hybrid between specified test oracle and derived test oracle. It specifies important (but not complete) properties of the system under testSUT. For example, [[metamorphic testing]] exploits such properties, called metamorphic relations, across multiple executions of the system.

=== Implicit ===

Line 32 ⟶ 37:

=== Human ===

When specified, derived or implicit test oracles cannot be used, thenA human inputcan toact determineas thea test oracles is requiredoracle.<ref name="ammann-intro" /> TheseThis approach can be thought ofcategorized as quantitative andor qualitative approaches.<ref name="Oracle survey"/>{{rp|519–520}} A quantitative approach aims to find the right amount of information to gather on a system under testSUT (e.g., test results) for a stakeholder to be able to make decisions on fit-for-purpose or the release of the software. A qualitative approach aims to find the representativeness and suitability of the input test data and context of the output from the system under testSUT. An example is using realistic and representative test data and making sense of the results (if they are realistic). These can be guided by [[heuristic]] approaches, such as gut instincts, rules of thumb, checklist aids, and experience to help tailor the specific combination selected for the program/system under testSUT.

== Examples ==