Hypercube (communication pattern): Difference between revisions

Revision as of 19:16, 31 March 2019

Revision as of 19:21, 31 March 2019

Line 58:

With each iteration, the transferred message doubles in length. This leads to a runtime of <math>T(n,p) \approx \sum_{j=0}^{d-1}(T_\text{start} + n \cdot 2^jT_\text{byte})= \log(p) T_\text{start} + (p-1)nT_\text{byte}</math>.

The same principle can be applied to the '''All-Reduce''' operations, but instead of concatenating the messages, it performs ana reduction operation on the two messages. So it is a '''Reduce''' operation, where all processing units know the result. InCompared Hypercubesto a modifiednormal reduce operation followed by a broadcast, '''All-Gather'''Reduce in hypercubes reduces the number of communications compared to Reduce andcommunication Broadcaststeps.

=== All-to-All ===