|
'''**Test and learn'''Learn** is a set of practices followed by retailers, banks, and other consumer-focused companies to test ideas in a small number of locations or with a limited group of customers to predict their impact. The process is often designed to answer three key questions aboutbefore anya tested program beforefull rollout:
#1. What impact will the program have on key performance indicators if executed across the network or customer base?
#2. Will the program have a larger impact on some stores/customers than others?
#3. Which components of the idea are actually working?
==## Historical context ==Context
Test and learnLearn has been systematically applied as far back assince 1988 by [[Capital One]]. CapitalThe Onecompany has been aggressive about testing since theits firmfounding, wasexperimenting founded, testingwith everything from product design to marketing to, customer selection, toand collection policies.<ref name="hbr">{{CiteCapital news|url=https://hbr.org/2009/02/how-to-design-smart-business-experiments|title=HowOne to Design Smart Business Experiments|last=Davenport|first=Thomas H.|date=February 2009|newspaper=[[Harvard Business Review]]|author-link=Thomas H. Davenport}}</ref> In a single year, the company performsconducts tens of thousands of tests annually, allowing it to offer thousands of different types of credit cards to customers, based on the knowledge gained from their tests. Richard Fairbank, [[CEO]] of Capital One, called testTest and learnLearn "a marketing revolution that can be applied to many businesses.".<ref>{{cite web
| last = Fishman
| first = Charles
| url = http://www.fastcompany.com/magazine/24/capone.html
| title = This is a Marketing Revolution
| publisher = [[Fast Company (magazine)|Fast Company]]
| access-date = September 22, 2009
| date = 1999-04-30
}}</ref>
==## Examples ==
* Retail banks running different commercials in different markets.
* Fast food restaurants launching a new menu item or product for a limited time in select locations .
* Big-box retailers evaluating “store of the future” programs.
* Companies evaluating pricing strategies.
* Retailers testing merchandising layouts across their door base.
* The UK government, through research projects such as the College for Teaching and Learning's Close the Gap: Test and learn <ref>{{Cite web | url=https://wwwLearn.gov.uk/government/publications/closing-the-gap-test-and-learn | title=Closing the gap: Test and learn}}</ref>
* [[Amazon (company)|Amazon]] frequently employs the Test and Learn methodology to determine the effectiveness of different homepage layouts, product recommendation algorithms, and pricing strategies.<ref>{{Cite web |date=2020-08-20 |title=Using A/B testing to measure the efficacy of recommendations generated by Amazon Personalize {{!}} AWS Machine Learning Blog |url=https://aws.amazon.com/blogs/machine-learning/using-a-b-testing-to-measure-the-efficacy-of-recommendations-generated-by-amazon-personalize/ |access-date=2023-07-26 |website=aws.amazon.com |language=en-US}}</ref><ref>{{Cite web |title=PFLB Platform - Exploring Test and Learn: A Scientific Examination of Predictive Business Strategies |url=https://platform.pflb.us/ |access-date=2023-07-26 |website=platform.pflb.us |language=en}}</ref>
* [[Netflix]] often employs the Test and Learn methodology for its algorithms, user interface, and even content creation.<ref>{{Cite journal |last=van Es |first=Karin |date=2022-09-26 |title=Netflix & Big Data: The Strategic Ambivalence of an Entertainment Company |url=http://journals.sagepub.com/doi/10.1177/15274764221125745 |journal=Television & New Media |language=en |pages=152747642211257 |doi=10.1177/15274764221125745 |issn=1527-4764|doi-access=free }}</ref>
==## Test and learnLearn in practice ==Practice
* [[Woolworths Group (Australia)|Woolworths]] uses a test-Test and-learn Learn approach to identify which stores should undergo renovations. <ref>{{cite web
* [[Wawa Food Markets ]], a Mid-Atlantic convenience store chain, uses Test and Learn across several aspects of its business: <ref name="storesmag">{{cite news▼ | url = https://www.9news.com.au/technology/woolworths-refurbished-stores-decided-by-computer-software-coles-stores/6d512adf-baa4-4320-b817-457a2f8d5897
* * Measuring the effect of introducing a new product .▼ | title = High-tech secret behind the future of your local Woolworths
* * Determining the impact of adding additional staff .▼ | publisher = 9News
* * Understanding how the opening a new ___location would impact existing nearby locations .▼ | date = January 23, 2020
* [[Subway (restaurant)|Subway]] ran a promotion that lowered the price of its foot-long sandwiches to $5. Before deciding to launch the promotion nationally, Subway ran a test that comparedcomparing locations offering the $5 foot-long sandwiches to a control group of similar [[Scientific control|control]] restaurants that were not offering the promotion. <ref name="hbr" />▼ | access-date = July 6, 2020
* [[Kraft Foods ]], a manufacturer and distributor of packaged foods, uses testing to optimize the placement of its products on grocery store shelves: <ref>{{cite web▼}}</ref>
* * Determining the effect of a coffee aisle re-design on sales of premium coffees .▼▲* [[Wawa Food Markets]], a Mid-Atlantic convenience store chain, uses Test and Learn across several aspects of its business:<ref name="storesmag">{{cite news
* * Measuring the impact on product sales of a change to in-store sales representatives' responsibilities .▼ |last = Fleenor
* [[Pier 1 Imports ]], a recently bankrupt retail chain specializing in home decor and furnishings, uses testing to measure the effectiveness of [[search advertising ]] campaigns. <ref>{{cite web▼ |first = D. Gail
* Test and learnLearn is also methodically used online to test the impact of various web page layouts on user behavior. Tools such as [[Google ]]'s Website Analyzer provide every site with the ability to test various elements and their effect on user engagement. ▼ |title = These Tests Are Positive
|date = June 2009
|newspaper = STORES Magazine
|url = http://www.stores.org/InformationTechnology/2009/06/Edit03.asp
|access-date = 18 September 2009
|url-status = dead
|archive-url = https://web.archive.org/web/20091012122924/http://www.stores.org/InformationTechnology/2009/06/Edit03.asp
|archive-date = 12 October 2009
}}</ref>
▲** Measuring the effect of introducing a new product
▲** Determining the impact of adding additional staff
▲** Understanding how the opening a new ___location would impact existing nearby locations
▲* [[Subway (restaurant)|Subway]] ran a promotion that lowered the price of its foot-long sandwiches to $5. Before deciding to launch the promotion nationally, Subway ran a test that compared locations offering the $5 foot-long sandwiches to a group of similar [[Scientific control|control]] restaurants that were not offering the promotion.<ref name="hbr" />
▲* [[Kraft Foods]], a manufacturer and distributor of packaged foods, uses testing to optimize the placement of its products on grocery store shelves:<ref>{{cite web
| last = Angrisani
| first = Carol
| url = http://subscribers.supermarketnews.com/Grocery_Center_Store_Brands/coffee_aisle_revamp_1222/
| title = Coffee Aisle Revamp Lifts Sales
| publisher = Supermarket News
| date = December 22, 2008
| access-date = September 22, 2009
}}</ref>
▲** Determining the effect of a coffee aisle re-design on sales of premium coffees
▲** Measuring the impact on product sales of a change to in-store sales representatives' responsibilities
▲* [[Pier 1 Imports]], a recently bankrupt retail chain specializing in home decor and furnishings, uses testing to measure the effectiveness of [[search advertising]] campaigns.<ref>{{cite web
| last = Wong
| first = Elaine
| url = http://www.brandweek.com/bw/content_display/news-and-features/direct/e3i048f01beefa084a38c84fb0602a1b651
| title = Case Study: How Search Ads Helped Pier 1 Stay Afloat
| publisher = [[Brandweek]]
| date = June 24, 2009
| access-date = September 22, 2009
}}</ref>
▲* Test and learn is also methodically used online to test impact of various web page layouts on user behavior. Tools such as [[Google]] Website Analyzer provide every site with the ability to test various elements and their effect on user engagement.
==## Test and learnLearn software==Software
There are a variety of [[Computer software|software]] tools available today to support systematic testing within an organization. However, there is currently no one software package that covers all types of tests, and in some cases significant knowledge of statistics is still required for effective analysis.<ref name="hbr" />
Various software tools support systematic testing within organizations. However, there is no single software package that covers all types of tests, and significant knowledge of statistics is often required for effective analysis. Many companies, including [[Capital One]] and [[eBay]], have developed internal experiment management software within the company. Alternatives to internal solutions include broad [[statistical analysis]] software tools such as [[SAS Institute|SAS]] or test and learn -focused software from [[Applied Predictive Technologies]], Trial Run, and MarketDial. There are also a variety of more narrowly tailored products, focused on particularspecific problems or fields., Insuch as [[web analytics]], for example,with firms such aslike [[Omniture]], [[WebTrends]], and [[Google]] have createdoffering specialized software.<ref name="hbr" />
==## Criticism ==
Companies wanting to conductConducting Test and Learn mustrequires mobilizemobilizing, tracktracking, and analyzeanalyzing site and customer data at very granular levels., Thiswhich can often be a massive and expensive undertaking. Increasingly howeverHowever, anthe exponential increase in computing power, and the corresponding decrease in its cost, have made testing more accessible. InThe theFebruary 2009 [[Harvard Business Review]] February 2009 article "How to Design Smart Business Experiments",<ref name="hbr" />by Thomas Davenport discusses the Test and Learn Process,process and describes how several companies have overcome these and other barriers to testing.
==## See also ==Also
* [[Clinical trials]]
* [[Scientific method]]
* [[Design of experiments]]
* [[Evidence-based management]] ▼* [[Evidence-based practices]]management
▲* [[Evidence-based management]]practices
* [[Business process reengineering]]
* [[A/B testing]]
{{Reflist}}
{{DEFAULTSORT:Test And Learn}}
| 