Test oracle: Difference between revisions

Revision as of 10:27, 16 February 2020

Latest revision as of 07:11, 28 August 2025

(36 intermediate revisions by 19 users not shown)

Line 1:

{{Short description|Provider of information for software testing}}

{{other uses2|oracle}}

{{Use American English|date=January 2021}}

{{other uses|Oracle (disambiguation)}}

In [[software testing]], a '''test oracle''' (or just '''oracle''') is a provider of information that describes [[Correctness (computer science)|correct]] output based on the input of a [[test case (software)|test case]]. Testing with an oracle involves comparing actual results of the system under test (SUT) with the expected results as provided by the oracle.<ref>Earl T. Barr et al; ''[https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6963470 The Oracle Problem in Software Testing: A Survey]'', 2015</ref>

In [[computing]], [[software engineering]], and [[software testing]], a '''test oracle''' (or just '''oracle''') is a mechanism for determining whether a test has passed or failed.<ref>Kaner, Cem; [http://www.testingeducation.org/k04/OracleExamples.htm ''A Course in Black Box Software Testing''], 2004</ref> The use of oracles involves comparing the output(s) of the system under test, for a given [[test case|test-case]] input, to the output(s) that the oracle determines that product should have. The term "test oracle" was first introduced in a paper by William E. Howden.<ref>{{cite journal |last1=Howden |first1=W.E. |date=July 1978 |title=Theoretical and Empirical Studies of Program Testing |journal=IEEE Transactions on Software Engineering |volume=4 |issue=4 |pages=293–298 |doi=10.1109/TSE.1978.231514 }}</ref> Additional work on different kinds of oracles was explored by [[Elaine Weyuker]].<ref>Weyuker, Elaine J.; "The Oracle Assumption of Program Testing", in ''Proceedings of the 13th International Conference on System Sciences (ICSS), Honolulu, HI, January 1980'', pp. 44-49</ref>

An oracle can operate separately from the SUT; accessed at test [[Runtime (program lifecycle phase)|runtime]], or it can be used before a test is run with expected results encoded into the test logic.<ref name="038720881X">Jalote, Pankaj; ''An Integrated Approach to Software Engineering'', Springer/Birkhäuser, 2005, {{ISBN|0-387-20881-X}}</ref>

Oracles often operate separately from the system under test.<ref name="038720881X">Jalote, Pankaj; ''An Integrated Approach to Software Engineering'', Springer/Birkhäuser, 2005, {{ISBN|0-387-20881-X}}</ref> However, [[Method (computer programming)|method]] postconditions are part of the system under test, as automated oracles in [[design by contract]] models.<ref>{{cite journal |last1=Meyer |first1=Bertrand |last2=Fiva |first2=Arno |last3=Ciupa |first3=Ilinca |last4=Leitner |first4=Andreas |last5=Wei |first5=Yi |last6=Stapf |first6=Emmanuel |date=September 2009 |title=Programs That Test Themselves |journal=Computer |volume=42 |issue=9 |pages=46–55 |doi= 10.1109/MC.2009.296 }}</ref> Determining the correct output for a given input (and a set of program/system states) is known as the '''oracle problem''' or '''test oracle problem'''<ref name="Oracle survey"/>{{rp|507}}, which is a much harder problem than it seems, and involves working with problems related to controllability and observability.<ref name="ammann-intro">Ammann, Paul; and Offutt, Jeff; "Introduction to Software Testing", ''Cambridge University Press'', 2008, {{ISBN|978-0-521-88038-1}}</ref> Various methods have been proposed to alleviate the test oracle problem. A popular technique is [[metamorphic testing]].<ref>{{cite journal | first1=Sergio | last1=Segura | first2=Gordon | last2=Fraser | first3=Ana B. | last3=Sanchez | first4=Antonio | last4=Ruiz-Cortes | title=A survey on metamorphic testing | journal=[[IEEE Transactions on Software Engineering]] | volume=42 | issue=9 | pages=805–824 | date=2016 | doi=10.1109/TSE.2016.2532875| hdl=11441/38271 }}</ref><ref>{{cite journal | first1=Tsong Yueh | last1=Chen | first2=Fei-Ching | last2=Kuo | first3=Huai | last3=Liu | first4=Pak-Lok | last4=Poon | first5=Dave | last5=Towey | first6=T.H. | last6=Tse | first7= Zhi Quan | last7=Zhou | title=Metamorphic testing: A review of challenges and opportunities | journal=[[ACM Computing Surveys]] | volume=51 | issue=1 | pages=4:1–4:27 | date=2018 | doi=10.1145/3143561| url=http://eprints.nottingham.ac.uk/51607/1/__MTChallOpporCSUR.accepted.20170922.pdf }}</ref>

However, [[Method (computer programming)|method]] postconditions are part of the SUT, as automated oracles in [[design by contract]] models.<ref>{{cite journal |last1=Meyer |first1=Bertrand |last2=Fiva |first2=Arno |last3=Ciupa |first3=Ilinca |last4=Leitner |first4=Andreas |last5=Wei |first5=Yi |last6=Stapf |first6=Emmanuel |date=September 2009 |title=Programs That Test Themselves |journal=Computer |volume=42 |issue=9 |pages=46–55 |doi= 10.1109/MC.2009.296 }}</ref>

Determining the correct output for a given input (and a set of program or system states) is known as the ''oracle problem'' or ''test oracle problem'',<ref name="Oracle survey"/>{{rp|507}} which some consider a relatively hard problem, and involves working with problems related to controllability and observability.<ref name="ammann-intro">Ammann, Paul; and Offutt, Jeff; "Introduction to Software Testing, 2nd edition", ''Cambridge University Press'', 2016, {{ISBN|978-1107172012}}</ref>

== Categories ==

A research literature survey covering 1978 to 2012<ref name="Oracle survey">{{cite journal |last1=Barr |first1=Earl T. |last2=Harman |first2=Mark |last3=McMinn |first3=Phil |last4=Shahbaz |first4=Muzammil |last5=Yoo |first5=Shin |date=November 2014 |title=The Oracle Problem in Software Testing: A Survey |journal=IEEE Transactions on Software Engineering |volume=41 |issue=5 |pages=507–525 |doi=10.1109/TSE.2014.2372785 |url=http://discovery.ucl.ac.uk/1471263/1/06963470.pdf |doi-access=free }}</ref> found several potential categorisationscategories forof test oracles.

=== Specified ===

These oracles are typically associated with formalised approaches to software modelling and software code construction. They are connected to [[formal specification]],<ref>{{cite book |last1=Börger |first1=E |editor-last1=Hutter |editor-first1=D |editor-last2=Stephan |editor-first2=W |editor-last3=Traverso |editor-first3=P |editor-last4=Ullman |editor-first4=M |date=1999|title=High Level System Design and Analysis Using Abstract State Machines |journal=Applied Formal Methods — FM-Trends 98 |volume=1641 |pages=1–43 |doi=10.1007/3-540-48257-1_1 |series=Lecture Notes in Computer Science |isbn=978-3-540-66462-8 |citeseerx=10.1.1.470.3653 }}</ref> [[model-based design]] which may be used to generate test oracles,<ref>{{cite journal |last1=Peters |first1=D.K. |date=March 1998 |title=Using test oracles generated from program documentation |journal=IEEE Transactions on Software Engineering |volume=24 |issue=3 |pages=161–173 |doi=10.1109/32.667877 |citeseerx=10.1.1.39.2890 }}</ref> state transition specification for which oracles can be derived to aid [[model-based testing]]<ref>{{cite journal| author-last1=Utting |author-first1=Mark |author-last2=Pretschner |author-first2=Alexander |author-last3=Legeard |author-first3=Bruno |title = A taxonomy of model-based testing approaches |journal = Software Testing, Verification and Reliability |volume= 22|issue= 5 |issn= 1099-1689|doi=10.1002/stvr.456 |pages= 297–312|year=2012 |url=https://eprints.qut.edu.au/57853/1/master_pdflatex.pdf }}</ref> and [[conformance testing|protocol conformance testing]],<ref>{{cite book|authorlink1=Marie-Claude Gaudel |last1=Gaudel |first1=Marie-Claude |editor-last1=Craeynest |editor-first1=D.|editor-last2=Strohmeier |editor-first2=A|date=2001 |title=Testing from Formal Specifications, a Generic Approach |journal= Reliable SoftwareTechnologies — Ada-Europe 2001 |volume=2043 |pages=35–48 |doi=10.1007/3-540-45136-6_3 |series=Lecture Notes in Computer Science |isbn=978-3-540-42123-8 }}</ref> and [[design by contract]] for which the equivalent test oracle is an [[assertion (software development)|assertion]].

TheseA oraclesspecified oracle areis typically associated with formalisedformalized approaches to software modellingmodeling and software code construction. TheyIt areis connected to [[formal specification]],<ref>{{cite book |last1=Börger |first1=E |title=Applied Formal Methods — FM-Trends 98 |chapter=High Level System Design and Analysis Using Abstract State Machines |editor-last1=Hutter |editor-first1=D |editor-last2=Stephan |editor-first2=W |editor-last3=Traverso |editor-first3=P |editor-last4=Ullman |editor-first4=M |date=1999|title=High Level System Design and Analysis Using Abstract State Machines |journal=Applied Formal Methods — FM-Trends 98 |volume=1641 |pages=1–43 |doi=10.1007/3-540-48257-1_1 |series=Lecture Notes in Computer Science |isbn=978-3-540-66462-8 |citeseerx=10.1.1.470.3653 }}</ref> [[model-based design]] which may be used to generate test oracles,<ref>{{cite journal |last1=Peters |first1=D.K. |date=March 1998 |title=Using test oracles generated from program documentation |journal=IEEE Transactions on Software Engineering |volume=24 |issue=3 |pages=161–173 |doi=10.1109/32.667877 |citeseerx=10.1.1.39.2890 }}</ref> state transition specification for which oracles can be derived to aid [[model-based testing]]<ref>{{cite journal| author-last1=Utting |author-first1=Mark |author-last2=Pretschner |author-first2=Alexander |author-last3=Legeard |author-first3=Bruno |title = A taxonomy of model-based testing approaches |journal = Software Testing, Verification and Reliability |volume= 22|issue= 5 |issn= 1099-1689|doi=10.1002/stvr.456 |pages= 297–312|year=2012 |url=https://eprints.qut.edu.au/57853/1/master_pdflatex.pdf }}</ref> and [[conformance testing|protocol conformance testing]],<ref>{{cite book|authorlink1author-link1=Marie-Claude Gaudel |last1=Gaudel |first1=Marie-Claude |title=Reliable SoftwareTechnologies — Ada-Europe 2001 |chapter=Testing from Formal Specifications, a Generic Approach |editor-last1=Craeynest |editor-first1=D.|editor-last2=Strohmeier |editor-first2=A|date=2001 |title=Testing from Formal Specifications, a Generic Approach |journal= Reliable SoftwareTechnologies — Ada-Europe 2001 |volume=2043 |pages=35–48 |doi=10.1007/3-540-45136-6_3 |series=Lecture Notes in Computer Science |isbn=978-3-540-42123-8 }}</ref> and [[design by contract]] for which the equivalent test oracle is an [[assertion (software development)|assertion]].

Specified Test Oracles have a number of challenges. Formal specification relies on abstraction, which in turn may naturally have an element of imprecision as all models cannot capture all behaviour.<ref name="Oracle survey"/>{{rp|514}}

Specified Testtest Oraclesoracles have a number of challenges. Formal specification relies on abstraction, which in turn may naturally have an element of imprecision as all models cannot capture all behaviourbehavior.<ref name="Oracle survey"/>{{rp|514}}

=== Derived ===

A derived test oracle differentiates correct and incorrect behaviour by using information derived from artefacts of the system. These may include documentation, system execution results and characteristics of versions of the system under test<ref name="Oracle survey"/>{{rp|514}}. Regression test suites (or reports) are an example of a derived test oracle - they are built on the assumption that the result from a previous system version can be used as aid (oracle) for a future system version. Previously measured performance characteristics may be used as an oracle for future system versions, for example, to trigger a question about observed potential performance degradation. Textual documentation from previous system versions may be used as a basis to guide expectations in future system versions.

A derived test oracle differentiates correct and incorrect behaviourbehavior by using information derived from artefactsartifacts of the system. These may include documentation, system execution results and characteristics of versions of the system under testSUT.<ref name="Oracle survey"/>{{rp|514}}. Regression test suites (or reports) are an example of a derived test oracle - they are built on the assumption that the result from a previous system version can be used as aid (oracle) for a future system version. Previously measured performance characteristics may be used as an oracle for future system versions, for example, to trigger a question about observed potential performance degradation. Textual documentation from previous system versions may be used as a basis to guide expectations in future system versions.

A '''pseudo-oracle''' falls into the category<ref name="Oracle survey"/>{{rp|515}} of derived test oracle. A pseudo-oracle, as defined by Weyuker,<ref name="pseudo-oracle">{{cite journal |last1=Weyuker |first1=E.J. |date=November 1982 |title=On Testing Non-Testable Programs |journal=The Computer Journal |volume=25 |issue=4 |pages=465–470 |doi=10.1093/comjnl/25.4.465 }}</ref> is a separately written program which can take the same input as the program/system under test so that their outputs may be compared to understand if there might be a problem to investigate.

A '''pseudo-oracle''' falls into the category<ref name="Oracle survey"/>{{rp|515}} falls into the category of derived test oracle. A pseudo-oracle, as defined by Weyuker,<ref name="pseudo-oracle">{{cite journal |last1=Weyuker |first1=E.J. |date=November 1982 |title=On Testing Non-Testable Programs |journal=The Computer Journal |volume=25 |issue=4 |pages=465–470 |doi=10.1093/comjnl/25.4.465 |doi-access=free }}</ref> is a separately written program which can take the same input as the program/system underor testSUT so that their outputs may be compared to understand if there might be a problem to investigate.

A partial oracle<ref name="Oracle survey"/>{{rp|515}} is a hybrid between specified test oracle and derived test oracle. It specifies important (but not complete) properties of the SUT. For example, [[metamorphic testing]] exploits such properties, called metamorphic relations, across multiple executions of the system.

=== Implicit ===

An implicit test oracle relies on implied information and assumptions<ref name="Oracle survey"/>{{rp|518}}. For example, there may be some implied conclusion from a program crash, i.e. unwanted behaviour - an oracle to determine that there may be a problem. There are a number of ways to search and test for unwanted behaviour, whether some call it negative testing, where there are specialized subsets such as [[fuzzing]].

There are limitations inAn implicit test oraclesoracle - as they relyrelies on implied conclusionsinformation and assumptions.<ref name="Oracle survey"/>{{rp|518}} For example, there may be some implied conclusion from a program/process crash, mayi.e. notunwanted bebehavior a- priorityan issueoracle ifto thedetermine systemthat isthere amay fault-tolerantbe systema andproblem. soThere operating underare a formnumber of self-healing/[[self-managementways (computerto science)|self-management]].search Implicitand test oraclesfor mayunwanted bebehavior, susceptiblewhether tosome falsecall positivesit duenegative totesting, environmentwhere dependenciesthere are specialized subsets such as [[fuzzing]].

There are limitations in implicit test oracles - as they rely on implied conclusions and assumptions. For example, a program or process crash may not be a priority issue if the system is a fault-tolerant system and so operating under a form of self-healing/[[self-management (computer science)|self-management]]. Implicit test oracles may be susceptible to false positives due to environment dependencies. Property based testing relies on implicit oracles.

=== Human ===

When specified, derived or implicit test oracles cannot be used, then human input to determine the test oracles is required. These can be thought of as quantitative and qualitative approaches.<ref name="Oracle survey"/>{{rp|519–520}}

*A human can act as a test oracle.<ref name="ammann-intro" /> This approach can be categorized as quantitative or qualitative.<ref name="Oracle survey"/>{{rp|519–520}} A quantitative approach aims to find the right amount of information to gather on a system under testSUT (e.g., test results) for a stakeholder to be able to make decisions on fit-for-purpose /or the release of the software. A qualitative approach aims to find the representativeness and suitability of the input test data and context of the output from the SUT. An example is using realistic and representative test data and making sense of the results (if they are realistic). These can be guided by [[heuristic]] approaches, such as gut instincts, rules of thumb, checklist aids, and experience to help tailor the specific combination selected for the SUT.

* A qualitative approach aims to find the representativeness and suitability of the input test data and context of the output from the system under test. An example is using realistic and representative test data and making sense of the results (if they are realistic).

These can be guided by [[heuristic]] approaches, i.e. gut instinct, rule of thumb, checklist aids and experience to help tailor the specific combination selected for the program/system under test.

== Examples ==

Common oracles include:

*Test oracles are most commonly based on [[specificationSpecification (technical standard)|specifications]] and [[software documentation|documentation]].<ref>{{cite thesis | last = Peters | first = Dennis K. | citeseerx = 10.1.1.69.4331 | title = Generating a Test Oracle from Program Documentation | publisher = McMaster University | degree = M. Eng. | year = 1995 }}</ref><ref>{{cite conference | last1 = Peters | first1 = Dennis K. | last2 = Parnas | first2 = David L. | url = http://www.engr.mun.ca/~dpeters/papers/issta.pdf | title = Generating a Test Oracle from Program Documentation | booktitlebook-title = Proceedings of the 1994 International Symposium on Software Testing and Analysis | conference = ISSTA | publisher = ACM Press | pages = 58–65 }}</ref> A formal specification used as input to [[model-based design]] and [[model-based testing]] would be an example of a ''specified test oracle''. DocumentationThe that''model-based wasoracle'' notuses athe [[formalsame specification]]model ofto thegenerate productand wouldverify typicallysystem bebehavior.<ref>Robinson, Harry; [http://www.harryrobinson.net/MBT-on-a-shoestring.pdf ''derivedFinite testState oracleModel-Based Testing on a Shoestring'']{{Dead link|date=August 2025 |bot=InternetArchiveBot |fix-attempted=yes }}, e.g.STAR West 1999</ref> Documentation that is not a full specification of the product, such as a usage or installation guide, or a record of performance characteristics or minimum machine requirements for the software, would typically be a derived test oracle.

* other products (for instance, an oracle for a software program might be a second program that uses a different [[algorithm]] to evaluate the same mathematical expression as the product under test). This is an example of a ''derived test oracle'', a pseudo-oracle.<ref name="pseudo-oracle"/>{{rp|466}}

* a ''heuristic oracle'' that provides approximate results or exact results for a set of a few test inputs<ref>

Hoffman, Douglas; [http://www.softwarequalitymethods.com/Papers/STQE%20Heuristic.pdf Heuristic Test Oracles], Software Testing & Quality Engineering Magazine, 1999</ref>

* a ''statistical oracle'' that uses statistical characteristics,<ref>{{cite conference |url=http://www.mathematik.uni-ulm.de/sai/mayer/publications/oracles.pdf |title=Test Oracles Using Statistical Methods |last1=Mayer |first1=Johannes |last2=Guderlei |first2=Ralph |date=2004 |publisher=Springer |book-title=Proceedings of the First International Workshop on Software Quality, Lecture Notes in Informatics |pages=179–189 |conference=First International Workshop on Software Quality}}</ref> for example with image analysis where a range of certainty/uncertainty is defined for the test oracle to pronounce a match or not. This would be an example of a ''human test oracle''.

* a ''consistency oracle'' that compares the results of one test execution to another for similarity.<ref>

Hoffman, Douglas; [http://www.softwarequalitymethods.com/Papers/OracleTax.pdf ''Analysis of a Taxonomy for Test Oracles''], Quality Week, 1998

</ref> This is an example of a ''derived test oracle''.

* a ''model-based oracle'' that uses the same model to generate and verify system behavior,<ref>Robinson, Harry; [http://www.harryrobinson.net/MBT-on-a-shoestring.pdf ''Finite State Model-Based Testing on a Shoestring''], STAR West 1999</ref> an example of a ''specified test oracle''.

* a human oracle (i.e. the correctness of the system under test is determined by manual analysis)<ref name="ammann-intro" />

A consistency oracle compares the results of one test execution to another for similarity.<ref>Hoffman, Douglas; [http://www.softwarequalitymethods.com/Papers/OracleTax.pdf ''Analysis of a Taxonomy for Test Oracles''] {{Webarchive|url=https://web.archive.org/web/20120310100741/http://www.softwarequalitymethods.com/Papers/OracleTax.pdf |date=2012-03-10 }}, Quality Week, 1998</ref> This is another example of a derived test oracle.

==References==

* other products (for instance, anAn oracle for a software program might be a second program that uses a different [[algorithm]] to evaluate the same mathematical expression as the product under test). This is an example of a ''derived test pseudo-oracle'', which is a pseudo-derived test oracle.<ref name="pseudo-oracle"/>{{rp|466}}

During [[Google]] search, we do not have a complete oracle to verify whether the number of returned results is correct. We may define a metamorphic relation<ref>{{cite journal |first1=Z.Q. |last1=Zhou |first2=S. |last2=Zhang |first3=M. |last3=Hagenbuchner |first4=T.H. |last4=Tse |first5=F.-C. | last5=Kuo |first6=T.Y. |last6=Chen |date=2012 |title=Automated functional testing of online search services |journal=Software Testing, Verification and Reliability |volume=22 |issue=4 |pages=221–243 |doi=10.1002/stvr.437 |hdl=10722/123864 |hdl-access=free }}</ref> such that a follow-up narrowed-down search will produce fewer results. This is an example of a partial oracle, which is a hybrid between specified test oracle and derived test oracle.

*A a ''statistical oracle'' that uses statisticalprobabilistic characteristics,<ref>{{cite conference |url=http://www.mathematik.uni-ulm.de/sai/mayer/publications/oracles.pdf |title=Test Oracles Using Statistical Methods |last1=Mayer |first1=Johannes |last2=Guderlei |first2=Ralph |date=2004 |publisher=Springer |book-title=Proceedings of the First International Workshop on Software Quality, Lecture Notes in Informatics |pages=179–189 |conference=First International Workshop on Software Quality |access-date=2009-09-09 |archive-date=2017-08-09 |archive-url=https://web.archive.org/web/20170809023446/http://www.mathematik.uni-ulm.de/sai/mayer/publications/oracles.pdf |url-status=dead }}</ref> for example with image analysis where a range of certainty/ and uncertainty is defined for the test oracle to pronounce a match or nototherwise. This would be an example of a ''quantitative approach in human test oracle''.

A heuristic oracle provides representative or approximate results over a class of test inputs.<ref>Hoffman, Douglas; [http://www.softwarequalitymethods.com/Papers/STQE%20Heuristic.pdf Heuristic Test Oracles] {{Webarchive|url=https://web.archive.org/web/20160314144359/http://www.softwarequalitymethods.com/Papers/STQE%20Heuristic.pdf |date=2016-03-14 }}, Software Testing & Quality Engineering Magazine, 1999</ref> This would be an example of a qualitative approach in human test oracle.

== References ==

{{reflist}}

== Bibliography ==

* Binder, Robert V. (1999). "Chapter 18 - Oracles" in ''Testing Object-Oriented Systems: Models, Patterns, and Tools'', Addison-Wesley Professional, 7 November 1999, {{ISBN|978-0-201-80938-1}}

[[Category:Software testing]]

[[Category:Computation oracles]]