Test oracle: Difference between revisions

Revision as of 20:55, 19 January 2021

Revision as of 20:56, 19 January 2021

Line 34:

The following are some examples of various categories of oracles:

* Test oracles are most commonly based on [[Specification (technical standard)|specifications]] and [[software documentation|documentation]].<ref>{{cite thesis | last = Peters | first = Dennis K. | citeseerx = 10.1.1.69.4331 | title = Generating a Test Oracle from Program Documentation | publisher = McMaster University | degree = M. Eng. | year = 1995 }}</ref><ref>{{cite conference | last1 = Peters | first1 = Dennis K. | last2 = Parnas | first2 = David L. | url = http://www.engr.mun.ca/~dpeters/papers/issta.pdf | title = Generating a Test Oracle from Program Documentation | book-title = Proceedings of the 1994 International Symposium on Software Testing and Analysis | conference = ISSTA | publisher = ACM Press | pages = 58–65 }}</ref> A formal specification used as input to [[model-based design]] and [[model-based testing]] would be an example of a ''specified test oracle''. The ''model-based oracle'' uses the same model to generate and verify system behavior.<ref>Robinson, Harry; [http://www.harryrobinson.net/MBT-on-a-shoestring.pdf ''Finite State Model-Based Testing on a Shoestring''], STAR West 1999</ref> Documentation that is not a full specification of the product, such as a usage or installation guide, or a record of performance characteristics or minimum machine requirements for the software, would typically be a ''derived test oracle''.

* A ''consistency oracle'' compares the results of one test execution to another for similarity.<ref>Hoffman, Douglas; [http://www.softwarequalitymethods.com/Papers/OracleTax.pdf ''Analysis of a Taxonomy for Test Oracles''], Quality Week, 1998</ref> This is another example of a ''derived test oracle''.

* An oracle for a software program might be a second program that uses a different [[algorithm]] to evaluate the same mathematical expression as the product under test. This is an example of a ''pseudo-oracle'', which is a ''derived test oracle''.<ref name="pseudo-oracle"/>{{rp|466}}

* During [[Google]] search, we do not have a complete oracle to verify whether the number of returned results is correct. We may define a metamorphic relation<ref>{{cite journal |first1=Z.Q. |last1=Zhou |first2=S. |last2=Zhang |first3=M. |last3=Hagenbuchner |first4=T.H. |last4=Tse |first5=F.-C. | last5=Kuo |first6=T.Y. |last6=Chen |date=2012 |title=Automated functional testing of online search services |journal=Software Testing, Verification and Reliability |volume=22 |issue=4 |pages=221–243 |doi=10.1002/stvr.437 }}</ref> such that a follow-up narrowed-down search will produce fewer results. This is an example of a ''partial oracle'', which is a hybrid between ''specified test oracle'' and ''derived test oracle''.

* A ''statistical oracle'' uses probabilistic characteristics,<ref>{{cite conference |url=http://www.mathematik.uni-ulm.de/sai/mayer/publications/oracles.pdf |title=Test Oracles Using Statistical Methods |last1=Mayer |first1=Johannes |last2=Guderlei |first2=Ralph |date=2004 |publisher=Springer |book-title=Proceedings of the First International Workshop on Software Quality, Lecture Notes in Informatics |pages=179–189 |conference=First International Workshop on Software Quality}}</ref> for example with image analysis where a range of certainty/uncertainty is defined for the test oracle to pronounce a match or otherwise. This would be an example of a ''quantitative approach'' in ''human test oracle''.

* A ''heuristic oracle'' provides representative or approximate results over a class of test inputs.<ref>Hoffman, Douglas; [http://www.softwarequalitymethods.com/Papers/STQE%20Heuristic.pdf Heuristic Test Oracles], Software Testing & Quality Engineering Magazine, 1999</ref> This would be an example of a ''qualitative approach'' in ''human test oracle''.

==References==