Content deleted Content added
Rewrite, what was a description of "Divide and conquer", not of map-reduce. |
Rescuing 2 sources and tagging 0 as dead.) #IABot (v2.0.9.5 |
||
| (357 intermediate revisions by more than 100 users not shown) | |||
Line 1:
{{Short description|Parallel programming model}}
'''MapReduce''' is a [[programming model]] and an associated implementation for processing and generating
A MapReduce program is composed of a
The model is a specialization of the ''split-apply-combine'' strategy for data analysis.<ref>{{Cite journal | doi = 10.18637/jss.v040.i01| title = The split-apply-combine strategy for data analysis| journal = Journal of Statistical Software| volume = 40| pages = 1–29| year = 2011| last1 = Wickham| first1 = Hadley | doi-access = free}}</ref>
It is inspired by the [[map (higher-order function)|map]] and [[reduce (higher-order function)|reduce]] functions commonly used in [[functional programming]],<ref name="map">"Our abstraction is inspired by the map and reduce primitives present in Lisp and many other functional languages." -[http://research.google.com/archive/mapreduce.html "MapReduce: Simplified Data Processing on Large Clusters"], by Jeffrey Dean and Sanjay Ghemawat; from Google Research</ref> although their purpose in the MapReduce framework is not the same as in their original forms.<ref>{{Cite journal | doi = 10.1016/j.scico.2007.07.001| title = Google's Map ''Reduce'' programming model — Revisited| journal = Science of Computer Programming| volume = 70| pages = 1–30| year = 2008| last1 = Lämmel | first1 = R. | doi-access = }}</ref> The key contributions of the MapReduce framework are not the actual map and reduce functions (which, for example, resemble the 1995 [[Message Passing Interface]] standard's<ref>http://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-2.0/mpi2-report.htm MPI 2 standard</ref> ''reduce''<ref>{{cite web|url=http://mpitutorial.com/tutorials/mpi-reduce-and-allreduce/|title=MPI Reduce and Allreduce · MPI Tutorial|website=mpitutorial.com}}</ref> and ''scatter''<ref>{{cite web|url=http://mpitutorial.com/tutorials/performing-parallel-rank-with-mpi/|title=Performing Parallel Rank with MPI · MPI Tutorial|website=mpitutorial.com}}</ref> operations), but the scalability and fault-tolerance achieved for a variety of applications due to parallelization. As such, a [[single-threaded]] implementation of MapReduce is usually not faster than a traditional (non-MapReduce) implementation; any gains are usually only seen with [[multi-threaded]] implementations on multi-processor hardware.<ref name=stackoverflow>{{cite web
| url = https://stackoverflow.com/questions/3947889/mongodb-terrible-mapreduce-performance
| title = MongoDB: Terrible MapReduce Performance
Line 10 ⟶ 11:
| date = October 16, 2010
| quote = The MapReduce implementation in MongoDB has little to do with map reduce apparently. Because for all I read, it is single-threaded, while map-reduce is meant to be used highly parallel on a cluster. ... MongoDB MapReduce is single threaded on a single server...
}}</ref>
MapReduce [[library (software)|libraries]] have been written in many programming languages, with different levels of optimization. A popular [[open-source software|open-source]] implementation that has support for distributed shuffles is part of [[Apache Hadoop]]. The name MapReduce originally referred to the proprietary [[Google]] technology, but has since
==Overview==
MapReduce is a framework for processing [[Parallel computing|parallelizable]] problems across large datasets using a large number of computers (nodes), collectively referred to as a [[Computer cluster|cluster]] (if all nodes are on the same local network and use similar hardware) or a [[Grid Computing|grid]] (if the nodes are shared across geographically and administratively distributed systems, and use more heterogeneous hardware). Processing can occur on data stored either in a [[filesystem]] (unstructured) or in a [[database]] (structured). MapReduce can take advantage of the locality of data, processing it near the place it is stored in order to minimize communication overhead.
A MapReduce framework (or system) is usually composed of three operations (or steps):
# '''
# '''Shuffle:''' worker nodes redistribute data based on the output keys (produced by the <code>map</code> function), such that all data belonging to one key is located on the same worker node.
# '''Reduce:''' worker nodes now process each group of output data, per key, in parallel.
MapReduce allows for the distributed processing of the map and reduction operations. Maps can be performed in parallel, provided that each mapping operation is independent of the others; in practice, this is limited by the number of independent data sources and/or the number of CPUs near each source. Similarly, a set of 'reducers' can perform the reduction phase, provided that all outputs of the map operation that share the same key are presented to the same reducer at the same time, or that the reduction function is [[Associative property|associative]]. While this process often appears inefficient compared to algorithms that are more sequential (because multiple instances of the reduction process must be run), MapReduce can be applied to significantly larger datasets than a single [[Commodity computing|"commodity" server]] can handle – a large [[server farm]] can use MapReduce to sort a [[petabyte]] of data in only a few hours.<ref>{{cite web|last=Czajkowski|first=Grzegorz|title=Sorting Petabytes with MapReduce – The Next Episode|url=https://googleresearch.blogspot.com/2011/09/sorting-petabytes-with-mapreduce-next.html|access-date=7 April 2014|author2=Marián Dvorský |author3=Jerry Zhao |author4=Michael Conley |date=7 September 2011 }}</ref> The parallelism also offers some possibility of recovering from partial failure of servers or storage during the operation: if one mapper or reducer fails, the work can be rescheduled – assuming the input data are still available.
Another way to look at MapReduce is as a 5-step parallel and distributed computation:
# '''Prepare the Map() input''' – the "MapReduce system" designates Map processors, assigns the input key
# '''Run the user-provided Map() code''' – Map() is run exactly once for each ''K1'' key
# '''"Shuffle" the Map output to the Reduce processors''' – the MapReduce system designates Reduce processors, assigns the ''K2'' key
# '''Run the user-provided Reduce() code''' – Reduce() is run exactly once for each ''K2'' key
# '''Produce the final output''' – the MapReduce system collects all the Reduce output, and sorts it by ''K2'' to produce the final outcome.
In many situations, the input data might have already
==Logical view==
Line 43:
<code>Map(k1,v1)</code> → <code>list(k2,v2)</code>
The ''Map'' function is applied in parallel to every pair (keyed by <code>k1</code>) in the input dataset. This produces a list of pairs (keyed by <code>k2</code>) for each call.
After that, the MapReduce framework collects all pairs with the same key (<code>k2</code>) from all lists and groups them together, creating one group for each key.
The ''Reduce'' function is then applied in parallel to each group, which in turn produces a collection of values in the same ___domain:
<code>Reduce(k2, list (v2))</code> → <code>list((k3, v3))</code><ref>{{Cite web|url=https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Inputs+and+Outputs|title = MapReduce Tutorial}}</ref>
Each ''Reduce'' call typically produces either one key value
Thus the MapReduce framework transforms a list of (key, value) pairs into
It is [[Necessity and sufficiency|necessary but not sufficient]] to have implementations of the map and reduce abstractions in order to implement MapReduce. Distributed implementations of MapReduce require a means of connecting the processes performing the Map and Reduce phases. This may be a [[distributed file system]]. Other options are possible, such as direct streaming from mappers to reducers, or for the mapping processors to serve up their results to reducers that query them.
===Examples===
The
'''function''' <u>map</u>(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)
'''function''' <u>reduce</u>(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum +=
emit (word, sum)
Here, each document is split into words, and each word is counted by the ''map'' function, using the word as the result key. The framework puts together all the pairs with the same key and feeds them to the same call to ''reduce''. Thus, this function just needs to sum all of its input values to find the total appearances of that word.
Line 77:
As another example, imagine that for a database of 1.1 billion people, one would like to compute the average number of social contacts a person has according to age. In [[SQL]], such a query could be expressed as:
<
SELECT age, AVG(contacts)
FROM social.person
GROUP BY age
ORDER BY age
</syntaxhighlight>
Using MapReduce, the
'''function''' Map '''is'''
Line 105:
'''end function'''
Note that in the {{mono|Reduce}} function, {{mono|C}} is the count of people having in total N contacts, so in the {{mono|Map}} function it is natural to write {{mono|1=C=1}}, since every output pair is referring to the contacts of one single person.
The MapReduce system would line up the 1100 Map processors, and would provide each with its corresponding 1 million input records. The Map step would produce 1.1 billion {{mono|(Y,(N,1))}} records, with {{mono|Y}} values ranging between, say, 8 and 103. The MapReduce System would then line up the 96 Reduce processors by performing shuffling operation of the key/value pairs due to the fact that we need average per age, and provide each with its millions of corresponding input records. The Reduce step would result in the much reduced set of only 96 output records {{mono|(Y,A)}}, which would be put in the final result file, sorted by {{mono|Y}}.
The count info in the record is important if the processing is reduced more than one time. If we did not add the count of the records, the computed average would be wrong, for example:
Line 121 ⟶ 123:
10, 10
If we reduce files
''-- reduce step #1: age, average of contacts''
10, 9
If we reduce it with file
==Dataflow==
[[Software framework#Architecture|Software framework architecture]] adheres to [[open-closed principle]] where code is effectively divided into unmodifiable ''frozen spots'' and [[extensibility|extensible]] ''hot spots''. The frozen
* an ''input reader''
* a ''Map'' function
Line 138 ⟶ 140:
===Input reader===
The ''input reader'' divides the input into appropriate size 'splits' (in practice, typically, 64 MB to 128 MB) and the framework assigns one split to each ''Map'' function. The ''input reader'' reads data from stable storage (typically, a [[distributed file system]]) and generates key/value pairs.
A common example will read a directory full of text files and return each line as a record.
Line 150 ⟶ 152:
Each ''Map'' function output is allocated to a particular ''reducer'' by the application's ''partition'' function for [[sharding]] purposes. The ''partition'' function is given the key and the number of reducers and returns the index of the desired ''reducer''.
A typical default is to [[Hash function|hash]] the key and use the hash value [[Modulo operation|modulo]] the number of ''reducers''.
(i.e. the reducers assigned
Between the map and reduce stages, the data
===Comparison function===
Line 164 ⟶ 166:
===Output writer===
The ''Output Writer'' writes the output of the ''Reduce'' to the stable storage
==Theoretical background==
Properties of [[Monoid|monoids]] are the basis for ensuring the validity of MapReduce operations.<ref>{{Cite journal
| doi = 10.1017/S0956796817000193
| title = An algebra for distributed Big Data analytics
| journal = Journal of Functional Programming
| volume = 28
| year = 2017
| last = Fegaras
| first = Leonidas
| s2cid = 44629767
| doi-access =
}}</ref><ref>{{cite arXiv
|last=Lin
|first=Jimmy
|title=Monoidify! Monoids as a Design Principle for Efficient MapReduce Algorithms
|eprint=1304.7544
|date=29 Apr 2013|class=cs.DC
}}</ref>
In the Algebird package<ref>{{Cite web
|title= Abstract Algebra for Scala
|url=https://twitter.github.io/algebird/}}</ref> a Scala implementation of Map/Reduce explicitly requires a monoid class type
.<ref>{{Cite web
|title= Encoding Map-Reduce As A Monoid With Left Folding
|date= 5 September 2016|url= http://erikerlandson.github.io/blog/2016/09/05/expressing-map-reduce-as-a-left-folding-monoid/}}</ref>
The operations of MapReduce deal with two types: the type ''A'' of input data being mapped, and the type ''B'' of output data being reduced.
The ''Map'' operation takes individual values of type ''A'' and produces, for each ''a:A'' a value ''b:B''; The ''Reduce'' operation requires a binary operation • defined on values of type ''B''; it consists of folding all available ''b:B'' to a single value.
From a basic requirements point of view, any MapReduce operation must involve the ability to arbitrarily regroup data being reduced. Such a requirement amounts to two properties of the operation •:
* associativity: (''x'' • ''y'') • ''z'' = ''x'' • (''y'' • ''z'')
* existence of neutral element ''e'' such that ''e'' • ''x'' = ''x'' • ''e'' = ''x'' for every ''x:B''.
The second property guarantees that, when parallelized over multiple nodes, the nodes that don't have any data to process would have no impact on the result.
These two properties amount to having a [[monoid]] (''B'', •, ''e'') on values of type ''B'' with operation • and with neutral element ''e''.
There's no requirements on the values of type ''A''; an arbitrary function ''A'' → ''B'' can be used for the ''Map'' operation. This means that we have a [[catamorphism]] ''A*'' → (''B'', •, ''e''). Here ''A*'' denotes a [[Kleene star]], also known as the type of lists over ''A''.
The ''Shuffle'' operation per se is not related to the essence of MapReduce; it's needed to distribute calculations over the cloud.
It follows from the above that not every binary ''Reduce'' operation will work in MapReduce. Here are the counter-examples:
* building a tree from subtrees: this operation is not associative, and the result will depend on grouping;
* direct calculation of averages: ''avg'' is also not associative (and it has no neutral element); to calculate an average, one needs to calculate [[Moment (mathematics)|moments]].
==Performance considerations==
MapReduce programs are not guaranteed to be fast. The main benefit of this programming model is to exploit the optimized shuffle operation of the platform, and only having to write the ''Map'' and ''Reduce'' parts of the program.
In practice, the author of a MapReduce program however has to take the shuffle step into consideration; in particular the partition function and the amount of data written by the ''Map'' function can have a large impact on the performance and scalability. Additional modules such as the ''Combiner'' function can help to reduce the amount of data written to disk, and transmitted over the network. MapReduce applications can achieve sub-linear speedups under specific circumstances.<ref name=":0">{{Cite journal|title = BSP cost and scalability analysis for MapReduce operations|journal = Concurrency and Computation: Practice and Experience|date = 2015-01-01|issn = 1532-0634|pages = 2503–2527|doi = 10.1002/cpe.3628|first1 = Hermes|last1 = Senger|first2 = Veronica|last2 = Gil-Costa|first3 = Luciana|last3 = Arantes|first4 = Cesar A. C.|last4 = Marcondes|first5 = Mauricio|last5 = Marín|first6 = Liria M.|last6 = Sato|first7 = Fabrício A.B.|last7 = da Silva|volume=28|issue = 8|hdl = 10533/147670|s2cid = 33645927|hdl-access = free}}</ref>
When designing a MapReduce algorithm, the author needs to choose a good tradeoff<ref name="ullman">{{
In tuning performance of MapReduce, the complexity of mapping, shuffle, sorting (grouping by the key), and reducing has to be taken into account. The amount of data produced by the mappers is a key parameter that shifts the bulk of the computation cost between mapping and reducing. Reducing includes sorting (grouping of the keys) which has nonlinear complexity. Hence, small partition sizes reduce sorting time, but there is a trade-off because having a large number of reducers may be impractical. The influence of split unit size is marginal (unless chosen particularly badly, say <1MB). The gains from some mappers reading load from local disks, on average, is minor.<ref>{{Cite journal|title = Scheduling divisible MapReduce computations|last1 = Berlińska|first1 = Joanna|date = 2010-12-01|journal = Journal of Parallel and Distributed Computing|doi = 10.1016/j.jpdc.2010.12.004|last2 = Drozdowski|first2 = Maciej|volume=71|issue = 3|pages=450–459}}</ref>
For processes that complete quickly, and where the data fits into main memory of a single machine or a small cluster, using a MapReduce framework usually is not effective. Since these frameworks are designed to recover from the loss of whole nodes during the computation, they write interim results to distributed storage. This crash recovery is expensive, and only pays off when the computation involves many computers and a long runtime of the computation. A task that completes in seconds can just be restarted in the case of an error, and the likelihood of at least one machine failing grows quickly with the cluster size. On such problems, implementations keeping all data in memory and simply restarting a computation on node failures or —when the data is small enough— non-distributed solutions will often be faster than a MapReduce system.
==Distribution and reliability==
Line 182 ⟶ 234:
==Uses==
MapReduce is useful in a wide range of applications, including distributed pattern-based searching, distributed sorting, web link-graph reversal, Singular Value Decomposition,<ref>{{cite
multi-cluster,<ref name="HMR">{{Cite book | doi = 10.1145/1996023.1996026| chapter = A Hierarchical Framework for Cross-Domain MapReduce Execution|chapter-url = http://yuanluo.net/publications/LUO_ECMLS2011.pdf| title = Proceedings of the second international workshop on Emerging computational methods for the life sciences (ECMLS '11)| year = 2011| last1 = Luo | first1 = Y. | last2 = Guo | first2 = Z. | last3 = Sun | first3 = Y.| last4 = Plale | first4 = B. |author4-link=Beth Plale| last5 = Qiu | first5 = J. | last6=Li|
first6=W. |isbn = 978-1-4503-0702-4| citeseerx = 10.1.1.364.9898| s2cid = 15179363}}</ref> volunteer computing environments,<ref name="volunteerMR">{{Cite book | doi = 10.1145/1851476.1851489| chapter = MOON: MapReduce On Opportunistic eNvironments| title = Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing – HPDC '10| pages = 95| year = 2010| last1 = Lin | first1 = H. | last2 = Ma | first2 = X. | last3 = Archuleta | first3 = J. | last4 = Feng | first4 = W. C. | last5 = Gardner | first5 = M. | last6 = Zhang | first6 = Z. | chapter-url = http://eprints.cs.vt.edu/archive/00001089/01/moon.pdf| isbn = 9781605589428| s2cid = 2351790}}</ref> dynamic cloud environments,<ref name="dynCloudMR">{{Cite journal| doi = 10.1016/j.jcss.2011.12.021| title = P2P-MapReduce: Parallel data processing in dynamic Cloud environments| journal = [[Journal of Computer and System Sciences]]| volume = 78| issue = 5| pages = 1382–1402| year = 2012| last1 = Marozzo| first1 = F.| last2 = Talia| first2 = D.| last3 = Trunfio| first3 = P.| doi-access = free}}</ref> mobile environments,<ref name="mobileMR">{{Cite book | doi = 10.1145/1839294.1839332| chapter = Misco: a MapReduce framework for mobile systems| title = Proceedings of the 3rd International Conference on PErvasive Technologies Related to Assistive Environments – PETRA '10| pages = 1| year = 2010| last1 = Dou | first1 = A. | last2 = Kalogeraki | first2 = V. | last3 = Gunopulos | first3 = D. | last4 = Mielikainen | first4 = T. | last5 = Tuulos | first5 = V. H. | isbn = 9781450300711| s2cid = 14517696}}</ref> and high-performance computing environments.<ref>{{cite book|chapter=Characterization and Optimization of Memory-Resident MapReduce on HPC Systems|publisher=IEEE|date=May 2014|doi=10.1109/IPDPS.2014.87|isbn=978-1-4799-3800-1|title=2014 IEEE 28th International Parallel and Distributed Processing Symposium|last1=Wang|first1=Yandong|last2=Goldstone|first2=Robin|last3=Yu|first3=Weikuan|last4=Wang|first4=Teng|pages=799–808|s2cid=11157612}}</ref>
At Google, MapReduce was used to completely regenerate Google's index of the [[World Wide Web]]. It replaced the old ''ad hoc'' programs that updated the index and ran the various analyses.<ref name="usage">{{cite web| quote=As of October, Google was running about 3,000 computing jobs per day through MapReduce, representing thousands of machine-days, according to a presentation by Dean. Among other things, these batch routines analyze the latest Web pages and update Google's indexes.| url=http://www.baselinemag.com/
MapReduce's stable inputs and outputs are usually stored in a [[distributed file system]]. The transient data
==Criticism==
===Lack of novelty===
[[David DeWitt]] and [[Michael Stonebraker]], computer scientists specializing in [[parallel database]]s and [[shared-nothing architecture]]s, have been critical of the breadth of problems that MapReduce can be used for.<ref name="shark">{{cite web| url=http://typicalprogrammer.com/
Greg Jorgensen wrote an article rejecting these views.<ref name="gj1">{{cite web| url=http://typicalprogrammer.com/
DeWitt and Stonebraker have subsequently published a detailed benchmark study in 2009 comparing performance of [[Hadoop|Hadoop's]] MapReduce and [[RDBMS]] approaches on several specific problems.<ref name="sigmod">{{cite web| url=
The MapReduce programming paradigm was also described in [[Danny Hillis]]'s 1985 thesis <ref name="WDHmit86">{{cite book |author-first=W. Danny |author-last=Hillis |date=1986 |title=The Connection Machine |publisher=[[MIT Press]] |isbn=0262081571 |url-access=registration |url=https://archive.org/details/connectionmachin00hill }}</ref> intended for use on the [[Connection Machine]], where it was called "xapping/reduction"<ref>{{cite web |url=http://bitsavers.trailing-edge.com/pdf/thinkingMachines/CM2/HA87-4_Connection_Machine_Model_CM-2_Technical_Summary_Apr1987.pdf |title=Connection Machine Model CM-2 Technical Summary |author=<!--Not stated--> |date=1987-04-01 |publisher=[[Thinking Machines Corporation]] |access-date=2022-11-21}}</ref> and relied upon that machine's special hardware to accelerate both map and reduce. The dialect ultimately used for the Connection Machine, the 1986 [[StarLisp]], had parallel <code>*map</code> and <code>reduce!!</code>,<ref>{{cite web |url=https://www.softwarepreservation.org/projects/LISP/starlisp/supplement-to-the-starlisp-reference-manual-version-5-0.pdf |title=Supplement to the *Lisp Reference Manual |author=<!--Not stated--> |date=1988-09-01 |publisher=[[Thinking Machines Corporation]] |access-date=2022-11-21}}</ref> which in turn was based on the 1984 [[Common Lisp]], which had non-parallel <code>map</code> and <code>reduce</code> built in.<ref>{{cite web |url=https://collections.lib.utah.edu/dl_files/20/2e/202ebf04b52d043c78297444bc9bc4fbc17b6b5e.pdf |title=Rediflow Architecture Prospectus |author=<!--Not stated--> |date=1986-04-05 |publisher=[[University of Utah School of Computing|University of Utah Department of Computer Science]] |access-date=2022-11-21}}</ref> The [[Fold (higher-order function)#Linear vs. tree-like folds|tree-like]] approach that the Connection Machine's [[Hypercube internetwork topology|hypercube architecture]] uses to execute <code>reduce</code> in <math>O(\log n)</math> time<ref>{{cite book |url=https://www.cise.ufl.edu/~sahni/papers/imagemono.pdf#page=20 |title=Hypercube Algorithms for Image Processing and Pattern Recognition |last=Ranka |first=Sanjay |date=1989 |access-date=2022-12-08 |section=2.6 Data Sum |publisher=University of Florida}}</ref> is effectively the same as the approach referred to within the Google paper as prior work.{{r|GoogleMapReduce|p=11|q=an associative function can be computed over all prefixes of an N element array in log N time on N processors using parallel prefix computations. MapReduce can be considered a simplification and distillation of some of these models}}
In 2010 Google was granted what is described as a patent on MapReduce. The patent, filed in 2004, may cover use of MapReduce by open source software such as [[Hadoop]], [[CouchDB]], and others. In ''[[Ars Technica]]'', an editor acknowledged Google's role in popularizing the MapReduce concept, but questioned whether the patent was valid or novel.<ref>{{cite news |last1=Paul |first1=Ryan |title=Google's MapReduce patent: what does it mean for Hadoop? |url=https://arstechnica.com/information-technology/2010/01/googles-mapreduce-patent-what-does-it-mean-for-hadoop/ |access-date=21 March 2021 |work=Ars Technica |date=20 January 2010 |language=en-us}}</ref><ref name="patent">{{cite web|url=http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL&p=1&u=/netahtml/PTO/srchnum.htm&r=1&f=G&l=50&s1=7,650,331.PN.&OS=PN/7,650,331&RS=PN/7,650,331|title=United States Patent: 7650331 - System and method for efficient large-scale data processing|website=uspto.gov|access-date=2010-01-19|archive-date=2013-09-21|archive-url=https://web.archive.org/web/20130921164908/http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL&p=1&u=/netahtml/PTO/srchnum.htm&r=1&f=G&l=50&s1=7,650,331.PN.&OS=PN/7,650,331&RS=PN/7,650,331|url-status=dead}}</ref> In 2013, as part of its "Open Patent Non-Assertion (OPN) Pledge", Google pledged to only use the patent defensively.<ref>{{cite news |last1=Nazer |first1=Daniel |title=Google Makes Open Patent Non-assertion Pledge and Proposes New Licensing Models |url=https://www.eff.org/deeplinks/2013/03/google-makes-open-patent-non-assertion-pledge |access-date=21 March 2021 |work=Electronic Frontier Foundation |date=28 March 2013 |language=en}}</ref><ref>{{cite news |last1=King |first1=Rachel |title=Google expands open patent pledge to 79 more about data center management |url=https://www.zdnet.com/article/google-expands-open-patent-pledge-to-79-more-about-data-center-management/ |access-date=21 March 2021 |work=ZDNet |date=2013 |language=en}}</ref> The patent is expected to expire on 23 December 2026.<ref>{{cite web |title=System and method for efficient large-scale data processing |url=https://patents.google.com/patent/US7650331B1/en |publisher=Google Patents Search |access-date=21 March 2021 |language=en |date=18 June 2004}}</ref>
===Restricted programming framework===
MapReduce tasks must be written as acyclic dataflow programs, i.e. a stateless mapper followed by a stateless reducer, that are executed by a batch job scheduler. This paradigm makes repeated querying of datasets difficult and imposes limitations that are felt in fields such as [[
==See also==
* [[Bird–Meertens formalism]]
* [[Parallelization contract]]
===Implementations of MapReduce===
* [[
* [[Apache Hadoop]]
* [[Infinispan]]
* [[Riak]]
==References==
{{reflist|30em}}
{{Commons category|MapReduce}}
{{Google
{{Authority control}}
{{DEFAULTSORT:Mapreduce}}
Line 295 ⟶ 281:
[[Category:Parallel computing]]
[[Category:Distributed computing architecture]]
[[Category:Articles with example code]]
| |||