Software regression: Difference between revisions

Revision as of 17:59, 30 March 2023

Latest revision as of 10:54, 27 August 2025

(3 intermediate revisions by 3 users not shown)

Line 1:

{{Short description|Type of software bug}}

A '''software regression''' is a type of [[software bug]] where a feature that has worked before stops working. This may happen after changes are applied to the software's [[source code]], including the addition of new [[software feature|features]] and bug fixes.<ref name=wong-issre-97>{{cite book |last1=Wong |first1=W. Eric |last2=Horgan |first2=J.R. |last3=London |first3=Saul |last4=Agrawal |first4=Hira |title=Proceedings of the Eighth International Symposium on Software Reliability Engineering (ISSRE 97) |date=1997 |publisher=IEEE |isbn=0-8186-8120-9 |url=https://ieeexplore.ieee.org/document/630875 |chapter=A Study of Effective Regression Testing in Practice|doi=10.1109/ISSRE.1997.630875 |s2cid=2911517 }}</ref> They may also be introduced by changes to the environment in which the software is running, such as system upgrades, [[Patch (computing)|system patching]] or a change to [[daylight saving time]].<ref name=ibm-locating-bugs>{{cite conference |last1=Yehudai|first1=Amiram |last2=Tyszberowicz|first2=Shmuel |last3=Nir|first3=Dor|title=Locating Regression Bugs|url=https://link.springer.com/chapter/10.1007/978-3-540-77966-7_18#:~:text=Abstract,patch%20causes%20the%20regression%20bug.|conference=Haifa Verification Conference|conference-url=https://www.research.ibm.com/haifa/conferences/hvc2017/previous.shtml|year=2007|doi=10.1007/978-3-540-77966-7_18 |accessdate=10 March 2018|url-access=subscription}}</ref> A '''software performance regression''' is a situation where the software still functions correctly, but performs more slowly or uses more memory or resources than before.<ref>{{Cite journal|last1=Shang|first1=Weiyi|last2=Hassan|first2=Ahmed E.|last3=Nasser|first3=Mohamed|last4=Flora|first4=Parminder|date=11 December 2014|title=Automated Detection of Performance Regressions Using Regression Models on Clustered Performance Counters|url=https://sail.cs.queensu.ca/Downloads/ICPE2015_AutomatedDetectionofPerformanceRegressionsUsingRegressionModelsOnClusteredPerformanceCounters.pdf#page=1|access-date=22 December 2019|archive-date=13 January 2021|archive-url=https://web.archive.org/web/20210113231136/https://sail.cs.queensu.ca/Downloads/ICPE2015_AutomatedDetectionofPerformanceRegressionsUsingRegressionModelsOnClusteredPerformanceCounters.pdf#page=1|url-status=dead}}</ref> Various types of software regressions have been identified in practice, including the following:<ref>{{cite book |last1=Henry |first1=Jean-Jacques Pierre |title=The Testing Network: An Integral Approach to Test Activities in Large Software Projects |date=2008 |publisher=Springer Science & Business Media |isbn=978-3540785040 |page=74}}</ref>

* ''Local'' – a change introduces a new bug in the changed module or component.

* ''Remote'' – a change in one part of the software breaks functionality in another module or component.

* ''Unmasked'' – a change unmasks an already existing bug that had no effect before the change.

Regressions are often caused by [[Hotfix|encompassed bug fixes]] included in [[software patch]]es. One approach to avoiding this kind of problem is [[regression testing]]. A properly designed [[test plan]] aims at preventing this possibility before releasing any software.<ref>{{cite book |last=Richardson |first=Jared |author2=Gwaltney, William Jr |title=Ship It! A Practical Guide to Successful Software Projects |url=https://archive.org/details/shipitpracticalg0000rich/page/32 |year=2006 |publisher=The Pragmatic Bookshelf |___location=Raleigh, NC |pages=[https://archive.org/details/shipitpracticalg0000rich/page/32 32, 193] |isbn=978-0-9745140-4-8 }}</ref> [[Automated testing]] and well-written [[Test case (software)|test case]]s can reduce the likelihood of a regression.

==Prevention and detection==

Techniques have been proposed that try to prevent regressions from being introduced into software at various stages of development, as outlined below.

===Prior to release===

Line 15:

{{main|Regression testing}}

In order to avoid regressions being seen by the [[end-user]] after release, developers regularly run [[regression tests]] after changes are introduced to the software. These tests can include [[unit tests]] to catch local regressions as well as [[integration tests]] to catch remote regressions.<ref>{{cite book |last1=Leung |first1=Hareton K.N. |last2=White |first2=Lee |title=Proceedings of the International Conference on Software Maintenance |date=November 1990 |publisher=IEEE |___location=San Diego, CA, USA |isbn=0-8186-2091-9 |url=https://ieeexplore.ieee.org/document/131377 |chapter=A study of integration testing and software regression at the integration level|doi=10.1109/ICSM.1990.131377 |s2cid=62583582 }}</ref> Regression testing techniques often leverage existing test cases to minimize the effort involved in creating them.<ref>{{cite journal |last1=Rothermel |first1=Gregg |last2=Harrold |first2=Mary Jean |last3=Dedhia |first3=Jeinay |title=Regression test selection for C++ software |journal=Software Testing, Verification and Reliability |date=2000 |volume=10 |issue=2 |pages=77–109 |doi=10.1002/1099-1689(200006)10:2<77::AID-STVR197>3.0.CO;2-E |url=https://onlinelibrary.wiley.com/doi/abs/10.1002/1099-1689(200006)10:2%3C77::AID-STVR197%3E3.0.CO;2-E |language=en |issn=1099-1689|url-access=subscription }}</ref> However, due to the volume of these existing tests, it is often necessary to select a representative subset, using techniques such as [[Regression_testing#Test_case_prioritization|test-case prioritization]].

For detecting performance regressions, [[software performance testing|software performance tests]] are run on a regular basis, to monitor the response time and resource usage metrics of the software after subsequent changes.<ref>{{cite journal |last1=Weyuker |first1=E.J. |last2=Vokolos |first2=F.I. |title=Experience with performance testing of software systems: issues, an approach, and case study |journal=IEEE Transactions on Software Engineering |date=December 2000 |volume=26 |issue=12 |pages=1147–1156 |doi=10.1109/32.888628 |url=https://ieeexplore.ieee.org/document/888628 |issn=1939-3520}}</ref> Unlike functional regression tests, the results of performance tests are subject to [[variance]] - that is, results can differ between tests due to variance in performance measurements; as a result, a decision must be made on whether a change in performance numbers constitutes a regression, based on experience and end-user demands. Approaches such as [[statistical significance test|statistical significance testing]] and [[change point detection]] are sometimes used to aid in this decision.<ref>{{cite book |last1=Daly |first1=David |last2=Brown |first2=William |last3=Ingo |first3=Henrik |last4=O'Leary |first4=Jim |last5=Bradford |first5=David |title=Proceedings of the International Conference on Performance Engineering |date=20 April 2020 |publisher=Association for Computing Machinery |isbn=978-1-4503-6991-6 |pages=67–75 |url=https://dl.acm.org/doi/abs/10.1145/3358960.3375791 |chapter=The Use of Change Point Detection to Identify Software Performance Regressions in a Continuous Integration System|doi=10.1145/3358960.3375791 |s2cid=211677818 }}</ref>

===Prior to commit===

Since [[debugging]] and localizing the root cause of a software regression can be expensive,<ref>{{cite book |last1=Nistor |first1=Adrian |last2=Jiang |first2=Tian |last3=Tan |first3=Lin |title=Proceedings of the Working Conference on Mining Software Repositories (MSR) |date=May 2013 |pages=237–246 |url=https://ieeexplore.ieee.org/document/6624035 |chapter=Discovering, reporting, and fixing performance bugs|doi=10.1109/MSR.2013.6624035 |isbn=978-1-4673-2936-1 |s2cid=12773088 }}</ref><ref>{{cite journal |last1=Agarwal |first1=Pragya |last2=Agrawal |first2=Arun Prakash |title=Fault-localization techniques for software systems: a literature review |journal=ACM SIGSOFT Software Engineering Notes |date=17 September 2014 |volume=39 |issue=5 |pages=1–8 |doi=10.1145/2659118.2659125 |s2cid=12101263 |url=https://dl.acm.org/doi/abs/10.1145/2659118.2659125 |issn=0163-5948|url-access=subscription }}</ref> there also exist some methods that try to prevent regressions from being committed into the [[code repository]] in the first place. For example, [[Git]] Hooks enable developers to run test scripts before code changes are committed or pushed to the code repository.<ref>{{cite web |title=Git - Git Hooks |url=https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks |website=git-scm.com |access-date=7 November 2021}}</ref> In addition, [[change impact analysis]] has been applied to software to predict the impact of a code change on various components of the program, and to supplement test case selection and prioritization.<ref>{{cite journal |last1=Orso |first1=Alessandro |last2=Apiwattanapong |first2=Taweesup |last3=Harrold |first3=Mary Jean |title=Leveraging field data for impact analysis and regression testing |journal=ACM SIGSOFT Software Engineering Notes |date=1 September 2003 |volume=28 |issue=5 |pages=128–137 |doi=10.1145/949952.940089 |url=https://dl.acm.org/doi/abs/10.1145/949952.940089 |issn=0163-5948|url-access=subscription }}</ref><ref>{{cite book |last1=Qu |first1=Xiao |last2=Acharya |first2=Mithun |last3=Robinson |first3=Brian |title=Proceedings of the International Conference on Software Maintenance |date=September 2012 |pages=129–138 |url=https://ieeexplore.ieee.org/document/6405263 |chapter=Configuration selection using code change impact analysis for regression testing|doi=10.1109/ICSM.2012.6405263 |isbn=978-1-4673-2312-3 |s2cid=14928793 }}</ref> [[Lint (software)|Software linters]] are also often added to commit hooks to ensure consistent coding style, thereby minimizing stylistic issues that can make the software prone to regressions.<ref>{{cite book |last1=Tómasdóttir |first1=Kristín Fjóla |last2=Aniche |first2=Mauricio |last3=van Deursen |first3=Arie |title=Proceedings of the International Conference on Automated Software Engineering |date=October 2017 |pages=578–589 |url=https://ieeexplore.ieee.org/document/8115668 |chapter=Why and how JavaScript developers use linters|doi=10.1109/ASE.2017.8115668 |isbn=978-1-5386-2684-9 |s2cid=215750004 }}</ref>

==Localization==

Line 31:

A common technique used to localize functional regressions is [[Bisection (software engineering)|bisection]], which takes both a buggy commit and a previously working commit as input, and tries to find the root cause by doing a binary search on the commits in between.<ref>{{cite book |last1=Gross |first1=Thomas |title=Proceedings of the International Workshop on Automatic Debugging |date=10 September 1997 |publisher=Linkøping University Electronic Press |pages=185–191 |url=https://ep.liu.se/en/conference-article.aspx?series=ecp&issue=1&Article_No=15 |language=en |chapter=Bisection Debugging}}</ref> [[Version control]] systems such as Git and [[Mercurial]] provide built-in ways to perform bisection on a given pair of commits.<ref>{{cite web |title=Git - git-bisect Documentation |url=https://git-scm.com/docs/git-bisect |website=git-scm.com |access-date=7 November 2021}}</ref><ref>{{cite web |title=hg - bisect |url=https://www.selenic.com/mercurial/hg.1.html |website=www.selenic.com |publisher=Mercurial |access-date=7 November 2021}}</ref>

Other options include directly associating the result of a regression test with code changes;<ref>{{cite web |title=Reading 11: Debugging |url=https://web.mit.edu/6.005/www/fa15/classes/11-debugging/ |website=web.mit.edu |publisher=MIT}}</ref> setting divergence breakpoints;<ref>{{cite book |last1=Buhse |first1=Ben |last2=Wei |first2=Thomas |last3=Zang |first3=Zhiqiang |last4=Milicevic |first4=Aleksandar |last5=Gligoric |first5=Milos |title=Proceedings of the International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) |date=May 2019 |pages=15–18 |url=https://ieeexplore.ieee.org/document/8802682 |chapter=VeDebug: Regression Debugging Tool for Java|doi=10.1109/ICSE-Companion.2019.00027 |isbn=978-1-7281-1764-5 |s2cid=174799830 }}</ref> or using incremental [[data-flow analysis]], which identifies test cases - including failing ones - that are relevant to a set of code changes,<ref>{{cite book |last1=Taha |first1=A.-B. |last2=Thebaut |first2=S.M. |last3=Liu |first3=S.-S. |title=Proceedings of the Annual International Computer Software & Applications Conference |date=September 1989 |publisher=IEEE |pages=527–534 |url=https://ieeexplore.ieee.org/document/65142 |chapter=An approach to software fault localization and revalidation based on incremental data flow analysis|doi=10.1109/CMPSAC.1989.65142 |isbn=0-8186-1964-3 |s2cid=41978046 }}</ref> among others.

===Performance regressions===

[[Profiling (computer programming)|Profiling]] measures the performance and resource usage of various components of a program, and is used to generate data useful in debugging performance issues. In the context of software performance regressions, developers often compare the [[call stack|call trees]] (also known as "timelines") generated by profilers for both the buggy version and the previously working version, and mechanisms exist to simplify this comparison.<ref>{{cite journal |last1=Ocariza |first1=Frolin S. |last2=Zhao |first2=Boyang |title=Localizing software performance regressions in web applications by comparing execution timelines |journal=Software Testing, Verification and Reliability |date=2021 |volume=31 |issue=5 |pages=e1750 |doi=10.1002/stvr.1750 |s2cid=225416138 |url=https://onlinelibrary.wiley.com/doi/abs/10.1002/stvr.1750 |language=en |issn=1099-1689|url-access=subscription }}</ref> [[Web development tools]] typically provide developers the ability to record these performance profiles.<ref>{{cite web |title=Analyze runtime performance |url=https://developer.chrome.com/docs/devtools/evaluate-performance/ |website=Chrome Developers |publisher=Google |access-date=7 November 2021 |language=en}}</ref><ref>{{cite web |title=Performance analysis reference - Microsoft Edge Development |url=https://docs.microsoft.com/en-us/microsoft-edge/devtools-guide-chromium/evaluate-performance/reference |website=docs.microsoft.com |publisher=Microsoft |access-date=7 November 2021 |language=en-us}}</ref>

Logging also helps with performance regression localization, and similar to call trees, developers can compare systematically-placed performance logs of multiple versions of the same software.<ref>{{cite book |last1=Yao |first1=Kundi |last2=B. de Pádua |first2=Guilherme |last3=Shang |first3=Weiyi |last4=Sporea |first4=Steve |last5=Toma |first5=Andrei |last6=Sajedi |first6=Sarah |title=Proceedings of the International Conference on Performance Engineering |date=30 March 2018 |publisher=Association for Computing Machinery |isbn=978-1-4503-5095-2 |pages=127–138 |url=https://dl.acm.org/doi/abs/10.1145/3184407.3184416 |chapter=Log4Perf: Suggesting Logging Locations for Web-based Systems' Performance Monitoring|doi=10.1145/3184407.3184416 |s2cid=4557038 }}</ref> A tradeoff exists when adding these performance logs, as adding many logs can help developers pinpoint which portions of the software are regressing at smaller granularities, while adding only a few logs will also reduce overhead when executing the program.<ref>{{cite journal |title=A Qualitative Study of the Benefits and Costs of Logging from Developers' Perspectives |journal=IEEE Transactions on Software Engineering |date=30 January 2020 |doi=10.1109/TSE.2020.2970422 |url=https://ieeexplore.ieee.org/document/8976297|last1=Li |first1=Heng |last2=Shang |first2=Weiyi |last3=Adams |first3=Bram |last4=Sayagh |first4=Mohammed |last5=Hassan |first5=Ahmed E. |volume=47 |issue=12 |pages=2858–2873 |s2cid=213679706 }}</ref>

Additional approaches include writing performance-aware unit tests to help with localization,<ref>{{cite book |last1=Heger |first1=Christoph |last2=Happe |first2=Jens |last3=Farahbod |first3=Roozbeh |title=Proceedings of the International Conference on Performance Engineering |date=21 April 2013 |publisher=Association for Computing Machinery |isbn=978-1-4503-1636-1 |pages=27–38 |url=https://dl.acm.org/doi/abs/10.1145/2479871.2479879 |chapter=Automated root cause isolation of performance regressions during software development|doi=10.1145/2479871.2479879 |s2cid=2593603 }}</ref> and ranking subsystems based on performance counter deviations.<ref>{{cite book |last1=Malik |first1=Haroon |last2=Adams |first2=Bram |last3=Hassan |first3=Ahmed E. |title=Proceedings of the International Symposium on Software Reliability Engineering |date=November 2010 |pages=201–210 |url=https://ieeexplore.ieee.org/document/5635038 |chapter=Pinpointing the Subsystems Responsible for the Performance Deviations in a Load Test|doi=10.1109/ISSRE.2010.43 |isbn=978-1-4244-9056-1 |s2cid=17306870 }}</ref> Bisection can also be repurposed for performance regressions by considering commits that perform below (or above) a certain baseline value as buggy, and taking either the left or the right side of the commits based on the results of this comparison.

==See also==