Programming with Big Data in R: Difference between revisions

Revision as of 23:09, 26 June 2013

Revision as of 23:28, 26 June 2013

Line 14:

| website = [http://www.r-pbd.org r-pbd.org]

}}

'''Programming with Big Data in R''' (pbdR)<ref>{{cite web|author=Ostrouchov, G., Chen, W.-C., Schmidt, D., Patel, P.|title=Programming with Big Data in R|year=2012|url=http://r-pbd.org/}}</ref><ref>{{cite web|title=XSEDE|url=https://portal.xsede.org/knowledge-base/-/kb/document/bcrw}}</ref> is a [[freeseries software]]of [[R (programming language)|R]] packages and an environment for [[statistical computing]] with [[Big Data]] by utilizing high-performance statistical computation.<ref>{{cite web|author=Chen, W.-C. and Ostrouchov, G.|url=http://thirteen-01.stat.iastate.edu/snoweye/hpsc/|year=2011|title=HPSC -- High Performance Statistical Computing for Data Intensive Research}}</ref> The pbdR languageuses isthe carriedsame fromprogramming language as [[R (programming language)|R]]<ref name=R>{{cite book|author=R Core Team|title=R: A Language and Environment for Statistical Computing|year=2012|isbn=3-900051-07-0|url=http://www.r-project.org/}}</ref> which is used among [[statistician]]s and [[Data mining|data miners]] for developing [[statistical software]]. The significant difference of pbdR and [[R (programming language)|R]]<ref name=R/> codes is pbdR mainly focuses on [[distributed memory]] system where data are distributed across several nodes, while communications between nodes are based on [[Message Passing Interface|MPI]] which is easily utilized in large [[High-performance computing|high-performance computing (HPC)]] systems. [[R (programming language)|R]] system<ref name=R/> mainly focuses on interactive data analysis on single [[Multi-core processor| multi-core]] machines. Two main implementations in [[R (programming language)|R]] using [[Message Passing Interface|MPI]] are [http://cran.r-project.org/package=Rmpi Rmpi]<ref name=rmpi/> and [http://cran.r-project.org/package=pbdMPI pbdMPI] of pbdR.

Two main implementations in [[R (programming language)|R]] using [[Message Passing Interface|MPI]] are [http://cran.r-project.org/package=Rmpi Rmpi]<ref name=rmpi/> and [http://cran.r-project.org/package=pbdMPI pbdMPI] of pbdR.

* The pbdR built on [http://cran.r-project.org/package=pbdMPI pbdMPI] uses [[SPMD|SPMD Parallelism]] where every processors are considered as workers and own parts of data. This parallelism is particularly for large data, for example, performing [[Singular value decomposition|singular value decomposition]] on a large matrix, or performing [[Mixture model|clustering analysis]] on high-dimensional large data. On the other hand, there is no restriction to use [[Master/slave (technology)|Manager/Workers Parallelism]] in [[SPMD|SPMD Parallelism]] environment.

* The [http://cran.r-project.org/package=Rmpi Rmpi]<ref name=rmpi/> uses [[Master/slave (technology)|Manager/Workers Parallelism]] where one main processor (manager) servers as the control of all other processors (workers). This parallelism is particularly efficient for large tasks in small [[Computer cluster|clusters]], for example, [[Bootstrapping (statistics)|bootstrap method]] and [[Monte Carlo method|Monte Carlo simulation]] in applied statistics since [[Independent and identically distributed random variables|i.i.d.]] assumption is commonly used in most [[Statistics|statistical analysis]].

Line 133 ⟶ 132:

* [http://cran.r-project.org/web/views/HighPerformanceComputing.html High-Performance and Parallel Computing with R].<ref>{{cite web|title=High-Performance and Parallel Computing with R|author=Dirk Eddelbuettel|url=http://cran.r-project.org/web/views/HighPerformanceComputing.html}}</ref>

* [http://userpages.umbc.edu/~gobbert/papers/pbdRtara2013.pdf UMBC HPCF Technique Report by Raim, A.M. (2013)].<ref>{{cite journal|author=Raim, A.M.|year=2013|title= Introduction to distributed computing with pbdR at the UMBC High Performance Computing Facility|journal= Technical Report HPCF-2013-2|url=http://userpages.umbc.edu/~gobbert/papers/pbdRtara2013.pdf}}</ref>

* [http://www.r-bloggers.com/r-at-12000-cores/ R at 12,000 Cores].<ref>{{cite news|title=R at 12,000 Cores|url=http://www.r-bloggers.com/r-at-12000-cores/}}</ref> This article was read 22,584 times in 2012 since it posted on October 16, 2012 and ranked number 3 according to [http://www.r-bloggers.com/100-most-read-r-posts-for-2012-stats-from-r-bloggers-big-data-visualization-data-manipulation-and-other-languages/|Top 100 R posts of 2012]<ref>{{cite webnews|url=http://www.r-bloggers.com/100-most-read-r-posts-for-2012-stats-from-r-bloggers-big-data-visualization-data-manipulation-and-other-languages/|title=100 most read R posts in 2012 (stats from R-bloggers) – big data, visualization, data manipulation, and other languages}}</ref>

== External links ==