Content deleted Content added
Fvillanustre (talk | contribs) m Fixed another broken link (for key-value pairs) |
Fvillanustre (talk | contribs) m Fixed broken link to LexisNexis Risk Solutions |
||
Line 36:
== System Architectures ==
A variety of [[system]] architectures have been implemented for data-intensive computing and large-scale data analysis applications including parallel and distributed [[relational database management systems]] which have been available to run on shared nothing clusters of processing nodes for more than two decades <ref>[http://www.cse.nd.edu/~dthain/courses/cse40771/spring2010/benchmarks-sigmod09.pdf A Comparison of Approaches to Large-Scale Data Analysis] by A. Pavlo, E. Paulson, A. Rasin, D.J. Abadi, D.J. Dewitt, S. Madden, and M. Stonebraker. Proceedings of the 35th SIGMOD International conference on Management of Data, 2009.</ref>.
. However most data growth is with data in unstructured form and new processing paradigms with more flexible data models were needed. Several solutions have emerged including the [[MapReduce]] architecture pioneered by Google and now available in an open-source implementation called [[Hadoop]] used by [[Yahoo]], [[Facebook]], and others. [[LexisNexis|LexisNexis Risk Solutions]] also developed and implemented a scalable platform for data-intensive computing which is used by [[LexisNexis]].
===MapReduce===
Line 52:
===HPCC===
[[LexisNexis|LexisNexis Risk Solutions]], independently developed and implemented a solution for data-intensive computing called the [[HPCC]] (High-Performance Computing Cluster). The development of this computing platform began in 1999 and applications were in production by late 2000. The [[LexisNexis]] approach also utilizes commodity clusters of hardware running the [[Linux]] operating system as shown in Figure 1. Custom system software and middleware components were developed and layered on the base Linux operating system to provide the execution environment and distributed filesystem support required for data-intensive computing. LexisNexis also implemented a new high-level language for data-intensive computing called ECL.
The [[ECL, data-centric programming language for Big Data|ECL programming language]] is the primary distinguishing factor between HPCC and other data-intensive computing solutions. It is a high-level, declarative, data-centric, [[Implicit parallelism|implicitly parallel]] language that allows the programmer to define what the data processing result should be and the dataflows and transformations that are necessary to achieve the result. The [[ECL]] language includes extensive capabilities for data definition, filtering, data management, and data transformation, and provides an extensive set of built-in functions to operate on records in datasets which can include user-defined transformation functions. [[ECL, data-centric programming language for Big Data|ECL]] programs are compiled into optimized [[C++]] source code, which is subsequently compiled into executable code and distributed to the nodes of a processing cluster.
| |||