Content deleted Content added
Prime suraj (talk | contribs) m By editor Tags: Reverted Visual edit Mobile edit Mobile web edit Newcomer task Newcomer task: references |
m →Python: HTTP to HTTPS for SourceForge |
||
| (37 intermediate revisions by 25 users not shown) | |||
Line 1:
{{short description|Message-passing system for parallel computers}}
{{multiple issues|
{{Update|reason=MPI-4.0 was approved by the MPI Forum in June 2021|date=October 2021}}
{{Update|reason=MPI-4.1 was approved by the MPI Forum in November 2023 (https://www.mpi-forum.org/docs/)|date=May 2024}}
'''Message Passing Interface''' ('''MPI''') is a standardized and portable [[message-passing]] standard designed to function on [[parallel computing]] [[computer architecture|architectures]].<ref>{{Cite web |title=Message Passing Interface :: High Performance Computing |url=https://hpc.nmsu.edu/discovery/mpi/introduction/ |access-date=2022-08-06 |website=hpc.nmsu.edu}}</ref> The MPI standard defines the [[syntax (programming languages)|syntax]] and [[semantics]] of [[library routine]]s that are useful to a wide range of users writing [[software portability|portable]] message-passing programs in [[C (programming language)|C]], [[C++]], and [[Fortran]]. There are several [[open-source]] MPI [[programming language implementation|implementations]], which fostered the development of a parallel [[software industry]], and encouraged development of portable and scalable large-scale parallel applications.▼
}}
▲The '''Message Passing Interface''' ('''MPI''') is a
==History==
The message passing interface effort began in the summer of 1991 when a small group of researchers started discussions at a mountain retreat in Austria. Out of that discussion came a Workshop on Standards for Message Passing in a Distributed Memory Environment, held on April 29–30, 1992 in [[Williamsburg, Virginia]].<ref>{{cite report |id= ORNL/TM-12147 |osti= 10170156 |author= Walker DW |date= August 1992 |title= Standards for message-passing in a distributed memory environment |url= https://technicalreports.ornl.gov/1992/3445603661204.pdf |institution= Oak Ridge National Lab., TN (United States), Center for Research on Parallel Computing (CRPC) |pages= 25 |access-date= 2019-08-18 |archive-date= 2023-11-15 |archive-url= https://web.archive.org/web/20231115183232/https://technicalreports.ornl.gov/1992/3445603661204.pdf |url-status= dead }}</ref> Attendees at Williamsburg discussed the basic features essential to a standard message-passing interface and established a working group to continue the standardization process. [[Jack Dongarra]], [[Tony Hey]], and David W. Walker put forward a preliminary draft proposal, "MPI1", in November 1992. In November 1992 a meeting of the MPI working group took place in Minneapolis and decided to place the standardization process on a more formal footing. The MPI working group met every 6 weeks throughout the first 9 months of 1993. The draft MPI standard was presented at the Supercomputing '93 conference in November 1993.<ref>{{cite conference |title= MPI: A Message Passing Interface |author= The MPI Forum, CORPORATE |date= November 15–19, 1993 |conference= Supercomputing '93 |conference-url= http://supercomputing.org/ |book-title= Proceedings of the 1993 ACM/IEEE conference on Supercomputing |publisher= ACM |___location= Portland, Oregon, USA |pages= 878–883 |isbn= 0-8186-4340-4 |doi= 10.1145/169627.169855 }}</ref> After a period of public comments, which resulted in some changes in MPI, version 1.0 of MPI was released in June 1994. These meetings and the email discussion together constituted the MPI Forum, membership of which has been open to all members of the [[High-performance computing
The MPI effort involved about 80 people from 40 organizations, mainly in the United States and Europe. Most of the major vendors of [[concurrent computer]]s were involved in the MPI effort, collaborating with researchers from universities, government laboratories, and [[Private industry|industry]].
Line 12 ⟶ 15:
MPI provides parallel hardware vendors with a clearly defined base set of routines that can be efficiently implemented. As a result, hardware vendors can build upon this collection of standard [[Low-level programming language|low-level]] routines to create [[High-level programming language|higher-level]] routines for the distributed-memory communication environment supplied with their [[parallel machine]]s. MPI provides a simple-to-use portable interface for the basic user, yet one powerful enough to allow programmers to use the high-performance message passing operations available on advanced machines.
In an effort to create a universal standard for message passing, researchers did not base it off of a single system but it incorporated the most useful features of several systems, including those designed by IBM, [[Intel]], [[nCUBE]], [[Parallel Virtual Machine|PVM]], Express, P4 and PARMACS. The message-passing paradigm is attractive because of wide portability and can be used in communication for distributed-memory and shared-memory multiprocessors, networks of workstations, and a combination of these elements. The paradigm can apply in multiple settings, independent of network speed or memory architecture.
Support for MPI meetings came in part from [[DARPA]] and from the U.S. [[National Science Foundation]] (NSF) under grant ASC-9310330, NSF Science and Technology Center Cooperative agreement number CCR-8809615, and from the [[European Commission]] through Esprit Project P6643. The [[University of Tennessee]] also made financial contributions to the MPI Forum.
Line 21 ⟶ 24:
MPI is a [[communication protocol]] for programming<ref>{{cite book |first=Frank |last=Nielsen | title=Introduction to HPC with MPI for Data Science | year=2016 | publisher=Springer |isbn=978-3-319-21903-5 |pages=195–211
|chapter=2. Introduction to MPI: The MessagePassing Interface | url=https://franknielsen.github.io/HPC4DS/index.html
|chapter-url=https://www.researchgate.net/publication/314626214 }}</ref> [[parallel computers]]. Both point-to-point and collective communication are supported. MPI "is a message-passing application programmer interface, together with protocol and semantic specifications for how its features must behave in any implementation."<ref>{{harvnb |Gropp |Lusk |Skjellum |1996 |p=3 }}</ref> MPI's goals are high performance, scalability, and portability. MPI remains the dominant model used in [[high-performance computing]]
MPI is not sanctioned by any major standards body; nevertheless, it has become a [[de facto standard|''de facto'' standard]] for [[communication]] among processes that model a [[parallel programming|parallel program]] running on a [[distributed memory]] system. Actual distributed memory supercomputers such as computer clusters often run such programs.
Line 34 ⟶ 37:
{{Anchor|VERSIONS}}
At present, the standard has several versions: version 1.3 (commonly abbreviated ''MPI-1''), which emphasizes message passing and has a static runtime environment, MPI-2.2 (MPI-2), which includes new features such as parallel I/O, dynamic process management and remote memory operations,<ref name="Gropp99adv-pp4-5">{{harvnb|Gropp|Lusk|Skjellum|1999b|pp=4–5}}</ref> and MPI-3.1 (MPI-3), which includes extensions to the collective operations with non-blocking versions and extensions to the one-sided operations.<ref name="MPI_3.1">[http://www.mpi-forum.org/docs/mpi-3.1/mpi31-report.pdf MPI: A Message-Passing Interface Standard<br />Version 3.1, Message Passing Interface Forum, June 4, 2015]. http://www.mpi-forum.org. Retrieved on 2015-06-16.</ref>
MPI-2's LIS specifies over 500 functions and provides language bindings for ISO [[C (programming language)|C]], ISO [[C++]], and [[Fortran 90]]. Object interoperability was also added to allow easier mixed-language message passing programming. A side-effect of standardizing MPI-2, completed in 1996, was clarifying the MPI-1 standard, creating the MPI-1.2.
''MPI-2'' is mostly a superset of MPI-1, although some functions have been deprecated. MPI-1.3 programs still work under MPI implementations compliant with the MPI-2 standard.
''MPI-3.0'' introduces significant updates to the MPI standard, including nonblocking versions of collective operations, enhancements to one-sided operations, and a Fortran 2008 binding. It removes deprecated C++ bindings and various obsolete routines and objects. Importantly, any valid MPI-2.2 program that avoids the removed elements is also valid in MPI-3.0.
''MPI-3.1'' is a minor update focused on corrections and clarifications, particularly for Fortran bindings. It introduces new functions for manipulating MPI_Aint values, nonblocking collective I/O routines, and methods for retrieving index values by name for MPI_T performance variables. Additionally, a general index was added. All valid MPI-3.0 programs are also valid in MPI-3.1.
''MPI-4.0'' is a major update that introduces large-count versions of many routines, persistent collective operations, partitioned communications, and a new MPI initialization method. It also adds application info assertions and improves error handling definitions, along with various smaller enhancements. Any valid MPI-3.1 program is compatible with MPI-4.0.
MPI-4.1 is a minor update focused on corrections and clarifications to the MPI-4.0 standard. It deprecates several routines, the MPI_HOST attribute key, and the mpif.h Fortran include file. A new routine has been added to inquire about the hardware running the MPI program. Any valid MPI-4.0 program remains valid in MPI-4.1.
MPI is often compared with [[Parallel Virtual Machine]] (PVM), which is a popular distributed environment and message passing system developed in 1989, and which was one of the systems that motivated the need for standard parallel message passing. Threaded shared memory programming models (such as [[Pthreads]] and [[OpenMP]]) and message passing programming (MPI/PVM) can be considered complementary and have been used together on occasion in, for example, servers with multiple large shared-memory nodes.
Line 47 ⟶ 56:
The MPI interface is meant to provide essential virtual topology, [[synchronization]], and communication functionality between a set of processes (that have been mapped to nodes/servers/computer instances) in a language-independent way, with language-specific syntax (bindings), plus a few language-specific features. MPI programs always work with processes, but programmers commonly refer to the processes as processors. Typically, for maximum performance, each [[CPU]] (or [[multi-core (computing)|core]] in a multi-core machine) will be assigned just a single process. This assignment happens at runtime through the agent that starts the MPI program, normally called mpirun or mpiexec.
MPI library functions include, but are not limited to, point-to-point rendezvous-type send/receive operations, choosing between a [[Cartesian tree|Cartesian]] or [[Graph (data structure)|graph]]-like logical process topology, exchanging data between process pairs (send/receive operations), combining partial results of computations (gather and reduce operations), synchronizing nodes (barrier operation) as well as obtaining network-related information such as the number of processes in the computing session, current processor identity that a process is mapped to, neighboring processes accessible in a logical topology, and so on. Point-to-point operations come in [[synchronization (computer science)|synchronous]], [[asynchronous i/o|asynchronous]], buffered, and ''ready'' forms, to allow both relatively stronger and weaker [[semantics]] for the synchronization aspects of a rendezvous-send. Many
MPI-1 and MPI-2 both enable implementations that overlap communication and computation, but practice and theory differ. MPI also specifies ''[[thread safe]]'' interfaces, which have [[cohesion (computer science)|cohesion]] and [[coupling (computer science)|coupling]] strategies that help avoid hidden state within the interface. It is relatively easy to write multithreaded point-to-point MPI code, and some implementations support such code. [[Multithreading (computer architecture)|Multithreaded]] collective communication is best accomplished with multiple copies of Communicators, as described below.
Line 70 ⟶ 79:
===Derived data types===
Many MPI functions require
Here is an example in C that passes arrays of <code>int</code>s from all processes to one. The one receiving process is called the "root" process, and it can be any designated process but normally it will be process 0. All the processes ask to send their arrays to the root with <code>MPI_Gather</code>, which is equivalent to having each process (including the root itself) call <code>MPI_Send</code> and the root make the corresponding number of ordered <code>MPI_Recv</code> calls to assemble all of these arrays into a larger one:<ref>{{cite web|url=https://www.open-mpi.org/doc/v1.8/man3/MPI_Gather.3.php|title=MPI_Gather(3) man page (version 1.8.8)|website=www.open-mpi.org}}</ref>
Line 84 ⟶ 93:
</syntaxhighlight>
However,
<syntaxhighlight lang="c">
MPI_Datatype newtype;
Line 105 ⟶ 114:
* <code>disp</code> contains byte displacements of each block,
* <code>type</code> contains types of element in each block.
* <code>newtype</code> (an output) contains the new derived type created by this function
The <code>disp</code> (displacements) array is needed for [[data structure alignment]], since the compiler may pad the variables in a class or data structure. The safest way to find the distance between different fields is by obtaining their addresses in memory. This is done with <code>MPI_Get_address</code>, which is normally the same as C's <code>&</code> operator but that might not be true when dealing with [[memory segmentation]].<ref>{{cite web|url=http://www.mpich.org/static/docs/v3.1/www3/MPI_Get_address.html|title=MPI_Get_address|website=www.mpich.org}}</ref>
Line 152 ⟶ 161:
===I/O===
The parallel I/O feature is sometimes called MPI-IO,<ref name="Gropp99adv-pp5-6">{{harvnb |Gropp |Lusk |Skjelling |1999b |pp=5–6 }}</ref> and refers to a set of functions designed to abstract I/O management on distributed systems to MPI, and allow files to be easily accessed in a patterned way using the existing derived datatype functionality.
The little research that has been done on this feature indicates that it may not be trivial to get high performance gains by using MPI-IO. For example, an implementation of sparse [[Matrix multiplication|matrix-vector multiplications]] using the MPI I/O library shows a general behavior of minor performance gain, but these results are inconclusive.<ref>{{
until the idea of collective I/O<ref>{{cite web|title=Data Sieving and Collective I/O in ROMIO|url=http://www.mcs.anl.gov/~thakur/papers/romio-coll.pdf|publisher=IEEE|date=Feb 1999}}</ref> implemented into MPI-IO that MPI-IO started to reach widespread adoption. Collective I/O substantially boosts applications' I/O bandwidth by having processes collectively transform the small and noncontiguous I/O operations into large and contiguous ones, thereby reducing the [[Record locking|locking]] and disk seek overhead. Due to its vast performance benefits, MPI-IO also became the underlying I/O layer for many state-of-the-art I/O libraries, such as [[HDF5]] and [[NetCDF|Parallel NetCDF]]. Its popularity also triggered research on collective I/O optimizations, such as layout-aware I/O<ref>{{cite book|chapter=LACIO: A New Collective I/O Strategy for Parallel I/O Systems|publisher=IEEE|date=Sep 2011|doi=10.1109/IPDPS.2011.79|isbn=978-1-61284-372-8|citeseerx=10.1.1.699.8972|title=2011 IEEE International Parallel & Distributed Processing Symposium|last1=Chen|first1=Yong|last2=Sun|first2=Xian-He|last3=Thakur|first3=Rajeev|last4=Roth|first4=Philip C.|last5=Gropp|first5=William D.|pages=794–804|s2cid=7110094}}</ref> and cross-file aggregation.<ref>{{cite journal|author1=Teng Wang|author2=Kevin Vasko|author3=Zhuo Liu|author4=Hui Chen|author5=Weikuan Yu|title=Enhance parallel input/output with cross-bundle aggregation|journal=The International Journal of High Performance Computing Applications|volume=30|issue=2|pages=241–256|date=2016|doi=10.1177/1094342015618017|s2cid=12067366}}</ref><ref>{{cite book|chapter=BPAR: A Bundle-Based Parallel Aggregation Framework for Decoupled I/O Execution|publisher=IEEE|date=Nov 2014|doi=10.1109/DISCS.2014.6|isbn=978-1-4673-6750-9|title=2014 International Workshop on Data Intensive Scalable Computing Systems|last1=Wang|first1=Teng|last2=Vasko|first2=Kevin|last3=Liu|first3=Zhuo|last4=Chen|first4=Hui|last5=Yu|first5=Weikuan|pages=25–32|s2cid=2402391}}</ref>
==Official implementations==
* The initial implementation of the MPI 1.x standard was [[MPICH]], from [[Argonne National Laboratory]] (ANL) and [[Mississippi State University]]. [[IBM]] also was an early implementor, and most early 90s [[supercomputer]] companies either commercialized MPICH, or built their own implementation. [[LAM/MPI]] from [[Ohio Supercomputer Center]] was another early open implementation. ANL has continued developing MPICH for over a decade, and now offers MPICH-4.3.
* [[Open MPI]] (not to be confused with [[OpenMP]]) was formed by the merging FT-MPI, LA-MPI, [[LAM/MPI]], and PACX-MPI, and is found in many [[TOP-500]] [[supercomputer]]s.
Many other efforts are derivatives of MPICH, LAM, and other works, including, but not limited to, commercial implementations from [[Hewlett
While the specifications mandate a C and Fortran interface, the language used to implement MPI is not constrained to match the language or languages it seeks to support at runtime. Most implementations combine C, C++ and assembly language, and target C, C++, and Fortran programmers. Bindings are available for many other languages, including Perl, Python, R, Ruby, Java, and [[Control Language|CL]] (see [[#Language bindings]]).
Line 179 ⟶ 186:
==Language bindings==
[[Language binding|Bindings]] are libraries that extend MPI support to other languages by wrapping an existing MPI implementation such as MPICH or Open MPI.
===Common Language Infrastructure===
The two managed [[Common Language Infrastructure]] [[.NET Framework|.NET]] implementations are Pure Mpi.NET<ref>{{Cite web|url=http://www.purempi.net/|title=移住の際は空き家バンクと自治体の支援制度を利用しよう
▲[http://www.purempi.net Pure Mpi.NET]</ref> and MPI.NET,<ref>{{cite web|url=http://www.osl.iu.edu/research/mpi.net/|title=MPI.NET: High-Performance C# Library for Message Passing|website=www.osl.iu.edu}}</ref> a research effort at [[Indiana University]] licensed under a [[BSD]]-style license. It is compatible with [[Mono (software)|Mono]], and can make full use of underlying low-latency MPI network fabrics.
===Java===
Line 208 ⟶ 214:
===Python===
Actively maintained MPI
Discontinued developments include: pyMPI, pypar,<ref>{{cite web|url=https://code.google.com/p/pypar/|title=Google Code Archive - Long-term storage for Google Code Project Hosting.|website=code.google.com}}</ref> MYMPI<ref>Now part of [https://sourceforge.net/projects/pydusa/ Pydusa]</ref> and the MPI submodule in [[ScientificPython]].
===R===
Line 303 ⟶ 308:
# MPI-2 implementations include I/O and dynamic process management, and the size of the middleware is substantially larger. Most sites that use batch scheduling systems cannot support dynamic process management. MPI-2's parallel I/O is well accepted.{{Citation needed|date=January 2011}}
# Many MPI-1.2 programs were developed before MPI-2. Portability concerns initially slowed adoption, although wider support has lessened this.
# Many MPI-1.2 applications use only a subset of that standard (
==Future==
Some aspects of the MPI's future appear solid; others less so. The
Architectures are changing, with greater internal concurrency ([[Multi-core processor|multi-core]]), better fine-grained concurrency control (threading, affinity), and more levels of [[memory hierarchy]]. [[Multithreading (computer architecture)|Multithreaded]] programs can take advantage of these developments more easily than single-threaded applications. This has already yielded separate, complementary standards for [[symmetric multiprocessing]], namely [[OpenMP]]. MPI-2 defines how standard-conforming implementations should deal with multithreaded issues, but does not require that implementations be multithreaded, or even thread-safe. MPI-3 adds the ability to use shared-memory parallelism within a node. Implementations of MPI such as Adaptive MPI, Hybrid MPI, Fine-Grained MPI, MPC and others offer extensions to the MPI standard that address different challenges in MPI.
Line 364 ⟶ 369:
|title=A High-Performance, Portable Implementation of the MPI Message Passing Interface
|journal=Parallel Computing |volume=22 |issue=6 |pages=789–828 |doi=10.1016/0167-8191(96)00024-5 }}
* Pacheco, Peter S. (1997) ''[https://books.google.com/books?&id=tCVkM1z2aOoC Parallel Programming with MPI]''.[http://www.cs.usfca.edu/mpi/ Parallel Programming with MPI] 500 pp. Morgan Kaufmann {{ISBN|1-55860-339-5}}.
* ''MPI—The Complete Reference'' series:
** Snir, Marc; Otto, Steve W.; Huss-Lederman, Steven; Walker, David W.; Dongarra, Jack J. (1995) ''[http://www.netlib.org/utk/papers/mpi-book/mpi-book.html MPI: The Complete Reference]''. MIT Press Cambridge, MA, USA. {{ISBN|0-262-69215-5}}
Line 371 ⟶ 376:
* Firuziaan, Mohammad; Nommensen, O. (2002) ''Parallel Processing via MPI & OpenMP'', Linux Enterprise, 10/2002
* Vanneschi, Marco (1999) ''Parallel paradigms for scientific computing'' In Proceedings of the European School on Computational Chemistry (1999, Perugia, Italy), number 75 in ''[https://books.google.com/books?&id=zMqVdFgVnrgC Lecture Notes in Chemistry]'', pages 170–183. Springer, 2000
* Bala, Bruck, Cypher, Elustondo, A Ho, CT Ho, Kipnis, Snir (1995) ″[https://ieeexplore.ieee.org/abstract/document/342126/ A portable and tunable collective communication library for scalable parallel computers]" in IEEE Transactions on Parallel and Distributed Systems,″ vol. 6, no. 2, pp.
{{div col end}}
Line 378 ⟶ 383:
* {{Official website|https://www.mpi-forum.org/}}
*[https://www.mpi-forum.org/docs/mpi-3.1/mpi31-report.pdf Official MPI-3.1 standard] ([https://www.mpi-forum.org/docs/mpi-3.1/mpi31-report/mpi31-report.htm unofficial HTML version])
* [http://polaris.cs.uiuc.edu/~padua/cs320/mpi/tutorial.pdf Tutorial on MPI: The Message-Passing Interface]
* [http://moss.csc.ncsu.edu/~mueller/cluster/mpi.guide.pdf A User's Guide to MPI]
| |||