Exploratory testing: Difference between revisions

Revision as of 18:35, 5 May 2008

Latest revision as of 11:26, 16 August 2025

(152 intermediate revisions by 100 users not shown)

Line 1:

{{Short description|Software testing approach}}

'''Exploratory testing''' is the tactical pursuit of software faults and defects driven by challenging assumptions. It is an approach in [[software testing]] with simultaneous learning, [[test design]] and test execution. While the software is being tested, the tester learns things that together with experience and [[creativity]] generates new good tests to run.

'''Exploratory testing''' is an approach to [[software testing]] that is concisely described as simultaneous learning, [[test design]] and test execution. [[Cem Kaner]], who coined the term in 1984,<ref>Cem Kaner, "[http://www.kaner.com/pdfs/QAIExploring.pdf A Tutorial in Exploratory Testing] {{Webarchive|url=https://web.archive.org/web/20130612043734/http://www.kaner.com/pdfs/QAIExploring.pdf |date=2013-06-12 }}", p.2</ref> defines exploratory testing as "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project."<ref>Cem Kaner, ''[http://www.kaner.com/pdfs/QAIExploring.pdf A Tutorial in Exploratory Testing] {{Webarchive|url=https://web.archive.org/web/20130612043734/http://www.kaner.com/pdfs/QAIExploring.pdf |date=2013-06-12 }}'', p. 36.</ref>

While the software is being tested, the tester learns things that together with experience and [[creativity]] generates new good tests to run. Exploratory testing is often thought of as a [[black box testing]] technique. Instead, those who have studied it consider it a test ''approach'' that can be applied to any test technique, at any stage in the development process. The key is not the test technique nor the item being tested or reviewed; the key is the cognitive engagement of the tester, and the tester's responsibility for managing his or her time.<ref>Cem Kaner, ''[http://www.kaner.com/pdfs/QAIExploring.pdf A Tutorial in Exploratory Testing] {{Webarchive|url=https://web.archive.org/web/20130612043734/http://www.kaner.com/pdfs/QAIExploring.pdf |date=2013-06-12 }}'', p. 37-39, 40- .</ref>

{{portal|Software Testing}}

==History==

Exploratory testing has been performed for a long time, and has similarities to [[ad hoc testing]]. In the early [[1990s]], [[ad hoc]] was too often synonymous with sloppy and careless work. As a result, a group of test methodologists (now calling themselves the [[Context-Driven School]]) began using the term "exploratory" seeking to emphasize the dominant thought process involved in unscripted testing, and to begin to develop the practice into a teachable discipline. This new terminology was first published by [[Cem Kaner]] in his book ''Testing Computer Software''. Exploratory testing can be as disciplined as any other intellectual activity.

==Description History ==

Exploratory testing has always been performed forby askilled long time, and has similarities to [[ad hoc testing]]testers. In the early [[1990s]], [[ad hoc]] was too often synonymous with sloppy and careless work. As a result, a group of test methodologists (now calling themselves the [[Software testing controversies|Context-Driven School]]) began using the term "exploratory" seeking to emphasize the dominant thought process involved in unscripted testing, and to begin to develop the practice into a teachable discipline. This new terminology was first published by [[Cem Kaner]] in his book ''Testing Computer Software'' <ref name=Kaner7-11>Cem Kaner, ''Testing Computer Software'', TAB Books, Blue Ridge Summit, PA, 1988. Exploratoryp. testing6, can7-11.</ref> beand asexpanded disciplinedupon asin any''Lessons otherLearned intellectualin activitySoftware Testing''.<ref>

Exploratory testing seeks to find out how the software actually works, and to ask questions about how it will handle difficult and easy cases.

{{cite book

The testing is dependent on the tester's skill of inventing [[test case]]s and finding [[software bug|defects]]. The more the tester knows about the product and different [[test method]]s, the better the testing will be.

|last = Kaner

|first = Cem

|author-link = Cem Kaner |author2=Bach, James |author3=Pettichord, Bret

|title = Lessons Learned in Software Testing

|publisher = [[John Wiley & Sons]]

|year = 2001

|isbn = 978-0-471-08112-8}}

</ref><!--The entire book discusses techniques used in Exploratory testing. They can also be used in other types of testing.--> Exploratory testing can be as disciplined as any other intellectual activity.

== Description ==

To further explain, comparison can be made of freestyle exploratory testing to the antithesis [[scripted testing]], which basically means that test cases are designed in advance, including steps to reproduce and expected results. These tests are later performed by a tester who compares the actual result with the expected.

Exploratory testing seeks to find out how the software actually works, and to ask questions about how it will handle difficult and easy cases. The quality of the testing is dependent on the tester's skill of inventing [[Test case (software)|test case]]s and finding [[software bug|defects]]. The more the tester knows about the product and different [[test method]]s, the better the testing will be.

To further explain, comparison can be made of freestyle exploratory testing to its antithesis [[test script|scripted testing]]. In the latter activity test cases are designed in advance. This includes both the individual steps and the expected results. These tests are later performed by a tester who compares the actual result with the expected. When performing exploratory testing, expectations are open. Some results may be predicted and expected; others may not. The tester configures, operates, observes, and evaluates the product and its behaviour, critically investigating the result, and reporting information that seems likely to be a bug (which threatens the value of the product to some person) or an issue (which threatens the quality of the testing effort).

When performing exploratory testing, there are no exact expected results; it is the tester that decides what will be verified, critically investigating the correctness of the result.

In reality, testing almost always is a combination of exploratory and scripted testing, but with a tendency towards either one, depending on context.

According to Cem Kaner &and [[James Marcus Bach]], Exploratoryexploratory testing is more a [[mindset]] or "...a way of thinking about testing" than a methodology.<ref>1Cem Kaner, James Bach, ''Exploratory & Risk Based Testing'', [http://www.testingeducation.org/a/nature.pdf www.testingeducation.org] {{Webarchive|url=https://web.archive.org/web/20080511190045/http://testingeducation.org/a/nature.pdf |date=2008-05-11 }}, 2004, p. 10</ref> They also say show that it crosses a continuum from slightly exploratory (slightly ambiguous or vaguely scripted testing) to highly exploratory (freestyle exploratory testing).<ref>2Cem Kaner, James Bach, ''Exploratory & Risk Based Testing'', [http://www.testingeducation.org/a/nature.pdf www.testingeducation.org] {{Webarchive|url=https://web.archive.org/web/20080511190045/http://testingeducation.org/a/nature.pdf |date=2008-05-11 }}, 2004, p. 14</ref>

The documentation of exploratory testing ranges from documenting all tests performed to just documenting the [[software bug|bugs]]. During [[pair testing]], two persons create test cases together; one performs them, and the other documents.

[[Session-based testtesting]]ing is a method specifically designed to make exploratory testing auditable and measurable on a wider scale.

Exploratory testers often use tools, including screen capture or video tools as a record of the exploratory session, or tools to quickly help generate situations of interest, e.g. James Bach's Perlclip.

==Benefits and drawbacks==

The main advantage of exploratory testing is that less preparation is needed, important bugs are found fast, and is more intellectually stimulating than scripted testing.

== Benefits and drawbacks ==

Another major benefit is that testers can use [[deductive reasoning]] based on the results of previous results to guide their future testing on the fly. They do not have to complete a current series of scripted tests before focusing in on or moving on to exploring a more target rich environment. This also accelerates bug detection when used intelligently.

The main advantage of exploratory testing is that less preparation is needed, important bugs are found fastquickly, and isat execution time, the approach tends to be more intellectually stimulating than execution of scripted testingtests.

Another major benefit is that testers can use [[deductive reasoning]] based on the results of previous results to guide their future testing on the fly. They do not have to complete a current series of scripted tests before focusing in on or moving on to exploring a more target rich environment. This also accelerates bug detection when used intelligently.

Another benefit is that, after initial testing, most bugs are discovered by some sort of exploratory testing. This can be demonstrated logically by stating, "Programs that pass certain tests tend to continue to pass the same tests and are more likely to fail other tests or scenarios that are yet to be explored."

Another benefit is that, after initial testing, most bugs are discovered by some sort of exploratory testing. This can be demonstrated logically by stating, "Programs that pass certain tests tend to continue to pass the same tests and are more likely to fail other tests or scenarios that are yet to be explored."

Disadvantages are that the tests can't be reviewed in advance (and by that prevent errors in code and test cases), and that it can be difficult to show exactly which tests have been run.

WhenDisadvantages repeatingare freestyle exploratorythat tests, theyinvented will not beand performed inon the exact same manner, whichfly can't be anreviewed advantagein ifadvance it(and isby importantthat toprevent finderrors newin errors;code orand atest disadvantagecases), ifand that it iscan morebe importantdifficult to knowshow thatexactly exactwhich thingstests arehave been functionalrun.

Freestyle exploratory test ideas, when revisited, are unlikely to be performed in exactly the same manner, which can be an advantage if it is important to find new errors; or a disadvantage if it is more important to repeat specific details of the earlier tests. This can be controlled with specific instruction to the tester, or by preparing automated tests where feasible, appropriate, and necessary, and ideally as close to the unit level as possible.

==Usage==

Exploratory testing is extra suitable if [[software requirement|requirements]] and [[Program specification|specifications]] are incomplete, or if there is lack of time. The approach can also be used to verify that previous testing has found the most important defects.

It is common to perform a combination of exploratory and scripted testing where the choice is based on [[Risc-based testing|risk]].

== Scientific studies ==

An example of exploratory testing in practice is [[Microsoft]]'s verification of [[Windows]] compatibility.

Replicated experiment has shown that while scripted and exploratory testing result in similar defect detection effectiveness (the total number of defects found) exploratory results in higher efficiency (the number of defects per time unit) as no effort is spent on pre-designing the test cases.<ref>{{Cite journal|last1=Itkonen|first1=Juha|last2=Mäntylä|first2=Mika V.|date=2013-07-11|title=Are test cases needed? Replicated comparison between exploratory and test-case-based software testing|journal=Empirical Software Engineering|language=en|volume=19|issue=2|pages=303–342|doi=10.1007/s10664-013-9266-8|issn=1382-3256|citeseerx=10.1.1.363.6524|s2cid=254472881 }}</ref> Observational study on exploratory testers proposed that the use of knowledge about the ___domain, the system under test, and customers is an important factor explaining the effectiveness of exploratory testing.<ref>{{Cite journal|last1=Itkonen|first1=J.|last2=Mäntylä|first2=M. V.|last3=Lassenius|first3=C.|date=2013-05-01|title=The Role of the Tester's Knowledge in Exploratory Software Testing|journal=IEEE Transactions on Software Engineering|volume=39|issue=5|pages=707–724|doi=10.1109/TSE.2012.55|s2cid=1763558 |issn=0098-5589}}</ref> A case-study of three companies found that ability to provide rapid feedback was a benefit of Exploratory Testing while managing test coverage was pointed as a short-coming.<ref>{{Cite book|last1=Itkonen|first1=J.|last2=Rautiainen|first2=K.|title=2005 International Symposium on Empirical Software Engineering, 2005 |chapter=Exploratory testing: A multiple case study |date=2005-11-01|pages=10 pp.–|doi=10.1109/ISESE.2005.1541817|isbn=978-0-7803-9507-7|s2cid=7974121 }}</ref> A survey found that Exploratory Testing is also used in critical domains and that Exploratory Testing approach places high demands on the person performing the testing.<ref>{{Cite book|last1=Pfahl|first1=Dietmar|last2=Yin|first2=Huishi|last3=Mäntylä|first3=Mika V.|last4=Münch|first4=Jürgen|title=Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement |chapter=How is exploratory testing used? A state-of-the-practice survey |date=2014-01-01|series=ESEM '14|___location=New York, NY, USA|publisher=ACM|pages=5:1–5:10|doi=10.1145/2652524.2652531|isbn=9781450327749|hdl=10138/153363|s2cid=17924562 }}</ref>

==References See also ==

* [[Ad hoc testing]]

* [[Spike (software development)|Spike testing]]

== References ==

<references/>

¹Exploratory & Risk Based Testing www.testingeducation.org 2004, P. 10

== External links ==

²Exploratory & Risk Based Testing www.testingeducation.org 2004, P. 14

* James Bach, ''[http://www.satisfice.com/articles/et-article.pdf Exploratory Testing Explained by James Bach]''

* Cem Kaner, James Bach, ''[http://www.testingeducation.org/a/nature.pdf The Nature of Exploratory Testing] {{Webarchive|url=https://web.archive.org/web/20080511190045/http://testingeducation.org/a/nature.pdf |date=2008-05-11 }}'', 2004

* Cem Kaner, James Bach, ''[http://www.context-driven-testing.com The Seven Basic Principles of the Context-Driven School]''

* Jonathan Kohl, ''[http://www.methodsandtools.com/archive/archive.php?id=65 Exploratory Testing: Finding the Music of Software Investigation]'', byKohl JonathanConcepts Kohl]Inc., 2007

{{portal|Software Testingtesting}}

==External links==

*[http://www.satisfice.com/articles/et-article.pdf Exploratory Testing Explained by James Bach]

*[http://www.context-driven-testing.com The Seven Basic Principles of the Context-Driven School]

*[http://www.checkpointech.com/sqa-software-solutions-4.php Exploratory Testing Theory, Tools and Training]

*[http://www.testingeducation.org/a/nature.pdf Exploratory & Risk Based Testing www.testingeducation.org 2004, by Cem Kaner & James Bach]

*[http://www.methodsandtools.com/archive/archive.php?id=65 Exploratory Testing: Finding the Music of Software Investigation by Jonathan Kohl]

*[http://www.manualtesting.com/why_manual_testing.htm Exploratory Testing Whitepapers, Seminars and more at ManualTesting.com]

==See also==

* [[Ad hoc testing]]

* [[Software testing controversies|Context-Driven School]]

* [[San Francisco depot]]

* [[Session-based test]]ing

* [[Software testing]]

* [[Test case]]

* [[Test script]]

{{DEFAULTSORT:Exploratory Testing}}

[[Category:Software testing]]

[[nl:Exploratory testing]]

[[sv:Utforskande testning]]