Test oracle: Difference between revisions

Revision as of 04:46, 19 January 2021

Revision as of 10:19, 19 January 2021

Line 27:

=== Human ===

When specified, derived or implicit test oracles cannot be used, then human input to determine the test oracles is required.<ref name="ammann-intro" /> These can be thought of as quantitative and qualitative approaches.<ref name="Oracle survey"/>{{rp|519–520}}

* A '''quantitative approach''' aims to find the right amount of information to gather on a system under test (e.g., test results) for a stakeholder to be able to make decisions on fit-for-purpose / release of the software.

* A '''qualitative approach aims''' to find the representativeness and suitability of the input test data and context of the output from the system under test. An example is using realistic and representative test data and making sense of the results (if they are realistic).

These can be guided by [[heuristic]] approaches, such as gut instincts, rules of thumb, checklist aids, and experience to help tailor the specific combination selected for the program/system under test.

== Examples ==

ExamplesThe offollowing are commonsome applicationsexamples of thevarious categories of oracles:

* Test oracles are most commonly based on [[Specification (technical standard)|Specificationsspecifications]] and [[software documentation|documentation]].<ref>{{cite thesis | last = Peters | first = Dennis K. | citeseerx = 10.1.1.69.4331 | title = Generating a Test Oracle from Program Documentation | publisher = McMaster University | degree = M. Eng. | year = 1995 }}</ref><ref>{{cite conference | last1 = Peters | first1 = Dennis K. | last2 = Parnas | first2 = David L. | url = http://www.engr.mun.ca/~dpeters/papers/issta.pdf | title = Generating a Test Oracle from Program Documentation | book-title = Proceedings of the 1994 International Symposium on Software Testing and Analysis | conference = ISSTA | publisher = ACM Press | pages = 58–65 }}</ref> A formal specification used as input to [[model-based design]] and [[model-based testing]] would be an example of a ''specified test oracle''. DocumentationThe that''model-based wasoracle'' notuses athe [[formalsame specification]]model ofto thegenerate productand wouldverify typicallysystem bebehavior.<ref>Robinson, Harry; [http://www.harryrobinson.net/MBT-on-a-shoestring.pdf ''derivedFinite testState oracleModel-Based Testing on a Shoestring''], e.g.STAR West 1999</ref> Documentation that is not a full specification of the product, such as a usage or installation guide, or a record of performance characteristics or minimum machine requirements for the software, would typically be a ''derived test oracle''.

* A ''model-basedconsistency oracle'' usescompares the sameresults modelof toone generatetest andexecution verifyto systemanother behavior,for similarity.<ref>RobinsonHoffman, HarryDouglas; [http://www.harryrobinsonsoftwarequalitymethods.netcom/MBT-on-a-shoestringPapers/OracleTax.pdf ''FiniteAnalysis Stateof Model-Baseda TestingTaxonomy onfor aTest ShoestringOracles''], STARQuality WestWeek, 19991998</ref>. This is ananother example of a ''specifiedderived test oracle''.

* An oracle for a software program might be a second program that uses a different [[algorithm]] to evaluate the same mathematical expression as the product under test. This is an example of a ''pseudo-oracle'', which is a ''derived test oracle''.<ref name="pseudo-oracle"/>{{rp|466}}

* During [[Google]] search, we do not have a complete oracle to verify whether the number of returned results is correct. We may define a metamorphic relation<ref>{{cite journal |first1=Z.Q. |last1=Zhou |first2=S. |last2=Zhang |first3=M. |last3=Hagenbuchner |first4=T.H. |last4=Tse |first5=F.-C. | last5=Kuo |first6=T.Y. |last6=Chen |date=2012 |title=Automated functional testing of online search services |journal=Software Testing, Verification and Reliability |volume=22 |issue=4 |pages=221–243 |doi=10.1002/stvr.437 }}</ref> such that a follow-up narrowed-down search will produce fewer results. This is an example of a ''partial oracle'', which is a hybrid between ''specified test oracle'' and ''derived test oracle''.

* A ''statistical oracle'' uses statisticalprobabilistic characteristics,<ref>{{cite conference |url=http://www.mathematik.uni-ulm.de/sai/mayer/publications/oracles.pdf |title=Test Oracles Using Statistical Methods |last1=Mayer |first1=Johannes |last2=Guderlei |first2=Ralph |date=2004 |publisher=Springer |book-title=Proceedings of the First International Workshop on Software Quality, Lecture Notes in Informatics |pages=179–189 |conference=First International Workshop on Software Quality}}</ref> for example with image analysis where a range of certainty/uncertainty is defined for the test oracle to pronounce a match or nototherwise. This would be an example of a ''quantitative approach'' in ''human test oracle''.

* A ''consistency oracle'' compares the results of one test execution to another for similarity.<ref>

* A ''heuristic oracle'' provides representative or approximate results over a class of test inputs.<ref>Hoffman, Douglas; [http://www.softwarequalitymethods.com/Papers/OracleTaxSTQE%20Heuristic.pdf ''Analysis of a Taxonomy forHeuristic Test Oracles''], Software Testing & Quality WeekEngineering Magazine, 19981999</ref> This iswould anotherbe an example of a ''derivedqualitative approach'' in ''human test oracle''.

* A ''heuristic oracle'' provides approximate results or exact results for a set of a few test inputs.<ref>

Hoffman, Douglas; [http://www.softwarequalitymethods.com/Papers/STQE%20Heuristic.pdf Heuristic Test Oracles], Software Testing & Quality Engineering Magazine, 1999</ref>

* A ''statistical oracle'' uses statistical characteristics,<ref>{{cite conference |url=http://www.mathematik.uni-ulm.de/sai/mayer/publications/oracles.pdf |title=Test Oracles Using Statistical Methods |last1=Mayer |first1=Johannes |last2=Guderlei |first2=Ralph |date=2004 |publisher=Springer |book-title=Proceedings of the First International Workshop on Software Quality, Lecture Notes in Informatics |pages=179–189 |conference=First International Workshop on Software Quality}}</ref> for example with image analysis where a range of certainty/uncertainty is defined for the test oracle to pronounce a match or not. This would be an example of a ''human test oracle''.

==References==