Content deleted Content added
Stevebroshar (talk | contribs) Use more correct words; clarify wording; simplify |
Rescuing 3 sources and tagging 1 as dead.) #IABot (v2.0.9.5 |
||
| (4 intermediate revisions by 4 users not shown) | |||
Line 1:
{{Short description|Provider of information for software testing}}
{{Use American English|date=January 2021}}
{{other uses|Oracle (disambiguation)}}
In [[software testing]], a '''test oracle''' (or just '''oracle''') is a provider of information that describes [[Correctness (computer science)|correct]] output based on the input of a [[test case (software)|test case]]. Testing with an oracle involves comparing actual results of the system under test (SUT) with the expected results as provided by the oracle.
The term "test oracle" was first introduced in a paper by William E. Howden.<ref>{{cite journal |last1=Howden |first1=W.E. |date=July 1978 |title=Theoretical and Empirical Studies of Program Testing |journal=IEEE Transactions on Software Engineering |volume=4 |issue=4 |pages=293–298 |doi=10.1109/TSE.1978.231514 }}</ref> Additional work on different kinds of oracles was explored by [[Elaine Weyuker]].<ref>Weyuker, Elaine J.; "The Oracle Assumption of Program Testing", in ''Proceedings of the 13th International Conference on System Sciences (ICSS), Honolulu, HI, January 1980'', pp. 44-49</ref>
Line 41 ⟶ 42:
== Examples ==
Test oracles are most commonly based on [[Specification (technical standard)|specifications]] and [[software documentation|documentation]].<ref>{{cite thesis | last = Peters | first = Dennis K. | citeseerx = 10.1.1.69.4331 | title = Generating a Test Oracle from Program Documentation | publisher = McMaster University | degree = M. Eng. | year = 1995 }}</ref><ref>{{cite conference | last1 = Peters | first1 = Dennis K. | last2 = Parnas | first2 = David L. | url = http://www.engr.mun.ca/~dpeters/papers/issta.pdf | title = Generating a Test Oracle from Program Documentation | book-title = Proceedings of the 1994 International Symposium on Software Testing and Analysis | conference = ISSTA | publisher = ACM Press | pages = 58–65 }}</ref> A formal specification used as input to [[model-based design]] and [[model-based testing]] would be an example of a ''specified test oracle''. The ''model-based oracle'' uses the same model to generate and verify system behavior.<ref>Robinson, Harry; [http://www.harryrobinson.net/MBT-on-a-shoestring.pdf ''Finite State Model-Based Testing on a Shoestring'']{{Dead link|date=August 2025 |bot=InternetArchiveBot |fix-attempted=yes }}, STAR West 1999</ref> Documentation that is not a full specification of the product, such as a usage or installation guide, or a record of performance characteristics or minimum machine requirements for the software, would typically be a derived test oracle.
A consistency oracle compares the results of one test execution to another for similarity.<ref>Hoffman, Douglas; [http://www.softwarequalitymethods.com/Papers/OracleTax.pdf ''Analysis of a Taxonomy for Test Oracles''] {{Webarchive|url=https://web.archive.org/web/20120310100741/http://www.softwarequalitymethods.com/Papers/OracleTax.pdf |date=2012-03-10 }}, Quality Week, 1998</ref> This is another example of a derived test oracle.
An oracle for a software program might be a second program that uses a different [[algorithm]] to evaluate the same mathematical expression as the product under test. This is an example of a pseudo-oracle, which is a derived test oracle.<ref name="pseudo-oracle"/>{{rp|466}}
Line 49 ⟶ 50:
During [[Google]] search, we do not have a complete oracle to verify whether the number of returned results is correct. We may define a metamorphic relation<ref>{{cite journal |first1=Z.Q. |last1=Zhou |first2=S. |last2=Zhang |first3=M. |last3=Hagenbuchner |first4=T.H. |last4=Tse |first5=F.-C. | last5=Kuo |first6=T.Y. |last6=Chen |date=2012 |title=Automated functional testing of online search services |journal=Software Testing, Verification and Reliability |volume=22 |issue=4 |pages=221–243 |doi=10.1002/stvr.437 |hdl=10722/123864 |hdl-access=free }}</ref> such that a follow-up narrowed-down search will produce fewer results. This is an example of a partial oracle, which is a hybrid between specified test oracle and derived test oracle.
A statistical oracle uses probabilistic characteristics,<ref>{{cite conference |url=http://www.mathematik.uni-ulm.de/sai/mayer/publications/oracles.pdf |title=Test Oracles Using Statistical Methods |last1=Mayer |first1=Johannes |last2=Guderlei |first2=Ralph |date=2004 |publisher=Springer |book-title=Proceedings of the First International Workshop on Software Quality, Lecture Notes in Informatics |pages=179–189 |conference=First International Workshop on Software Quality |access-date=2009-09-09 |archive-date=2017-08-09 |archive-url=https://web.archive.org/web/20170809023446/http://www.mathematik.uni-ulm.de/sai/mayer/publications/oracles.pdf |url-status=dead }}</ref> for example with image analysis where a range of certainty and uncertainty is defined for the test oracle to pronounce a match or otherwise. This would be an example of a quantitative approach in human test oracle.
A heuristic oracle provides representative or approximate results over a class of test inputs.<ref>Hoffman, Douglas; [http://www.softwarequalitymethods.com/Papers/STQE%20Heuristic.pdf Heuristic Test Oracles] {{Webarchive|url=https://web.archive.org/web/20160314144359/http://www.softwarequalitymethods.com/Papers/STQE%20Heuristic.pdf |date=2016-03-14 }}, Software Testing & Quality Engineering Magazine, 1999</ref> This would be an example of a qualitative approach in human test oracle.
== References ==
| |||