Parallelization contract: Difference between revisions

Revision as of 15:29, 2 March 2020

Revision as of 16:37, 9 January 2021

Line 23:

Optionally, the user code can be annotated with additional information. These annotations disclose some information on the behavior of the black-box user function. The [[PactCompiler|PACT Compiler]] can utilize the information to obtain more efficient execution plans. However, while a missing annotation will not change the result of the execution, an incorrect Output Contract produces wrong results.

{{:wiki:pact.png?nolink&600|}}

The currently supported Input Contracts and annotation are presented and discussed in the following.

Line 38 ⟶ 37:

== MAP ==

The Map Input Contract works in the same way as in MapReduce. It has a single input and assigns each input record to its own subset. Hence, all records are processed independently from each other (see figure below). \\

{{:wiki:map.png?nolink&100|}}

== REDUCE ==

Line 46 ⟶ 45:

The Reduce Input Contract has the same semantics as the reduce function in MapReduce. It has a single input and groups together all records that have identical key fields. Each of these groups is handed as a whole to the user code and processed by it (see figure below). The PACT Programming Model does also support optional Combiners, e.g. for partial aggregations. \\

{{:wiki:reduce.png?nolink&100|}}

== CROSS ==

Line 52 ⟶ 50:

The Cross Input Contract works on two inputs. It builds the Cartesian product of the records of both inputs. Each element of the Cartesian product (pair of records) is handed to the user code. \\

{{:wiki:cross.png?nolink&160|}}

== MATCH ==

Line 58 ⟶ 56:

The Match Input Contract works on two inputs. From both inputs it matches those records that are identical on their key fields come from different inputs. Hence, it resembles an equality join where the keys of both inputs are the attributes to join on. Each matched pair of records is handed to the user code. \\

{{:wiki:match.png?nolink&160|}}

== COGROUP ==

The CoGroup Input Contract works on two inputs as well. It can be seen as a Reduce on two inputs. On each input, the records are grouped by key (such as Reduce does) and handed to the user code. In contrast to Match, the user code is also called for a key if only one input has a pair with it (see blue key in example below).

{{:wiki:cogroup.png?nolink&200|}}

=== Pact Record Data Model ===