Test oracle: Difference between revisions

Revision as of 04:42, 18 January 2021

Revision as of 04:46, 19 January 2021

Line 26:

=== Human ===

When specified, derived or implicit test oracles cannot be used, then human input to determine the test oracles is required.<ref name="ammann-intro" /> These can be thought of as quantitative and qualitative approaches.<ref name="Oracle survey"/>{{rp|519–520}}

* A quantitative approach aims to find the right amount of information to gather on a system under test (e.g., test results) for a stakeholder to be able to make decisions on fit-for-purpose / release of the software.

* A qualitative approach aims to find the representativeness and suitability of the input test data and context of the output from the system under test. An example is using realistic and representative test data and making sense of the results (if they are realistic).

These can be guided by [[heuristic]] approaches, i.e.such as gut instinctinstincts, rulerules of thumb, checklist aids, and experience to help tailor the specific combination selected for the program/system under test.

== Examples ==

Line 35:

* [[Specification (technical standard)|Specifications]] and [[software documentation|documentation]].<ref>{{cite thesis | last = Peters | first = Dennis K. | citeseerx = 10.1.1.69.4331 | title = Generating a Test Oracle from Program Documentation | publisher = McMaster University | degree = M. Eng. | year = 1995 }}</ref><ref>{{cite conference | last1 = Peters | first1 = Dennis K. | last2 = Parnas | first2 = David L. | url = http://www.engr.mun.ca/~dpeters/papers/issta.pdf | title = Generating a Test Oracle from Program Documentation | book-title = Proceedings of the 1994 International Symposium on Software Testing and Analysis | conference = ISSTA | publisher = ACM Press | pages = 58–65 }}</ref> A formal specification used as input to [[model-based design]] and [[model-based testing]] would be an example of a ''specified test oracle''. Documentation that was not a [[formal specification]] of the product would typically be a ''derived test oracle'', e.g. a usage or installation guide, or a record of performance characteristics or minimum machine requirements for the software.

* A ''model-based oracle'' that uses the same model to generate and verify system behavior,<ref>Robinson, Harry; [http://www.harryrobinson.net/MBT-on-a-shoestring.pdf ''Finite State Model-Based Testing on a Shoestring''], STAR West 1999</ref>. This is an example of a ''specified test oracle''.

* An oracle for a software program might be a second program that uses a different [[algorithm]] to evaluate the same mathematical expression as the product under test. This is an example of a pseudo-oracle, which is a ''derived test oracle'', a pseudo-oracle.<ref name="pseudo-oracle"/>{{rp|466}}

* During [[Google]] search, we do not have a complete oracle to verify whether the number of returned results is correct. We may define a metamorphic relation<ref>{{cite journal |first1=Z.Q. |last1=Zhou |first2=S. |last2=Zhang |first3=M. |last3=Hagenbuchner |first4=T.H. |last4=Tse |first5=F.-C. | last5=Kuo |first6=T.Y. |last6=Chen |date=2012 |title=Automated functional testing of online search services |journal=Software Testing, Verification and Reliability |volume=22 |issue=4 |pages=221–243 }}</ref> such that a follow-up narrowed-down search will produce fewer results. This is an example of a ''partial oracle'', which is a hybrid between ''specified test oracle'' and ''derived test oracle''.

* A ''heuristic oracle'' that provides approximate results or exact results for a set of a few test inputs.<ref>

* A ''consistency oracle'' that compares the results of one test execution to another for similarity.<ref>

Hoffman, Douglas; [http://www.softwarequalitymethods.com/Papers/OracleTax.pdf ''Analysis of a Taxonomy for Test Oracles''], Quality Week, 1998</ref> This is another example of a ''derived test oracle''.

* A ''heuristic oracle'' that provides approximate results or exact results for a set of a few test inputs.<ref>

Hoffman, Douglas; [http://www.softwarequalitymethods.com/Papers/STQE%20Heuristic.pdf Heuristic Test Oracles], Software Testing & Quality Engineering Magazine, 1999</ref>

* A ''statistical oracle'' that uses statistical characteristics,<ref>{{cite conference |url=http://www.mathematik.uni-ulm.de/sai/mayer/publications/oracles.pdf |title=Test Oracles Using Statistical Methods |last1=Mayer |first1=Johannes |last2=Guderlei |first2=Ralph |date=2004 |publisher=Springer |book-title=Proceedings of the First International Workshop on Software Quality, Lecture Notes in Informatics |pages=179–189 |conference=First International Workshop on Software Quality}}</ref> for example with image analysis where a range of certainty/uncertainty is defined for the test oracle to pronounce a match or not. This would be an example of a ''human test oracle''.

* A ''consistency oracle'' that compares the results of one test execution to another for similarity.<ref>

Hoffman, Douglas; [http://www.softwarequalitymethods.com/Papers/OracleTax.pdf ''Analysis of a Taxonomy for Test Oracles''], Quality Week, 1998

</ref> This is an example of a ''derived test oracle''.

* A ''model-based oracle'' that uses the same model to generate and verify system behavior,<ref>Robinson, Harry; [http://www.harryrobinson.net/MBT-on-a-shoestring.pdf ''Finite State Model-Based Testing on a Shoestring''], STAR West 1999</ref> an example of a ''specified test oracle''.

* A ''human oracle'' (i.e. the correctness of the system under test is determined by manual analysis).<ref name="ammann-intro" />

==References==