In-memory processing: Difference between revisions

Revision as of 05:21, 4 September 2013

Revision as of 05:36, 4 September 2013

Line 5:

{{Ad|date=June 2013}}

}}

{{merge|In-memory database|discuss=Talk:In-MemoryTHIS ProcessingPAGE#Merger proposal|date=January 2012}}

== Definition ==

With businesses demanding faster and easy access to information in order to make reliable and smart decisions, In-memory processing is an emerging technology that is gaining attention. It enables users to have immediate access to right information which results in more informed decisions. Traditional [[Business Intelligence]] (BI) technology loads data onto the computer's hard disk in the form of tables and [[OLAP cube|multi-dimensional cubes]] against which queries are run. Using In-memory data is loaded into memory ([[Random Access Memory|RAM]] (RAM) or [[flash memory|flash]]) instead of hard disks and therefore information technology (IT) staff spendspends less development time on [[data modeling]], query analysis, cube building and table design.<ref>{{cite book|last=Earls|first=A|title=Tips on evaluating, deploying and managing in-memory analytics tools|year=2011|publisher=Tableau|url=http://www.analyticsearches.com/site/files/776/66977/259607/579091/In-Memory_Analytics_11.10.11.pdf}}</ref>

== Traditional Business Intelligence technologyBI ==

Historically, every computer has two types of data storage mechanisms: the– disk (hard disk) and RAM (Random Access Memory). Modern computers have more available disk storage than RAM, but reading data from the disk is much slower (possibly hundreds of times) when compared to reading the same data from RAM. WhenEspecially usingwhen traditionalanalyzing disklarge technologyvolumes ([[Relationalof Databasedata, Managementperformance Systems]]is suchseverely asdegraded. [[MySQL]]Using andtraditional [[Oracledisk database|Oracle]]),based atechnology the query accesses information from multiple tables stored on thea server’s hard disk. Traditional disk based technologies means [[Relational Database Management Systems]] such as SQL Server, MySQL, [[Oracle database|Oracle]] and many others. RDMS are designed around usingkeeping transactional processing in mind. Having a database that supports insertionsboth andinsertions, updates as well as performsperforming aggregations and, joins (typical in BI solutions) is not possible. forAlso thesethe traditionalstructured systems.query Also,language ([[SQL]]) is designed to more efficiently fetch rows of data thanwhile BI queries, which usually involve fetching of partial rows of data involving heavy calculations.

Though SQL is a very powerful tool for running complex queries, ittook takes avery long time to execute and often resultsresulted in bringing down transactional processing. To improve query performance, multidimensional databases or cubes (also called [[MOLAP|multidimensional online analytical processing]] (MOLAP) were formed. Designing a cube design involved an elaborate and lengthy process andwhich ittook wasa cumbersomesignificant toamount changeof itstime from IT staff. Changing the cubes structure to adapt to dynamically changing business needs was cumbersome. Cubes are pre- populated with data to answer specific queries and although theyit increased performance, theyit still failed to answer ad hoc queries.<ref>{{cite journal|last=Gill|first=John|title=Shifting the BI Paradigm with In-Memory Database Technologies|journal=Business Intelligence Journal|year=2007|month=Second Quarter|volume=12|issue=2|pages=58–62|url=http://www.highbeam.com/doc/1P3-1636785121.html}}</ref>

== Disadvantages of traditional Business IntelligenceBI ==

To avoid performance issues and provide faster query processing when dealing with large volumes of data, organizations needed optimized database methods like creating [[index (database)|index]]es, and usinguse specialized data structures and [[Aggregate (data warehouse)|aggregate tables]].

The point of having a data warehouse is to be able to get results for any queries asked at any time. However,But in order to achieve better response times,time for users many [[data mart]]smarts are designed to precalculatepre calculate summaries and answer specific queries, which defeatsdefeating the purpose of a [[data warehouse]]. Optimized aggregation algorithms needed to be used to increase performance.
Traditional BI tools were neither able tocouldn’t keep up with the ever growing BI requirements, norand were they ableunable to deliver real- time data to thefor end userusers.<ref>{{cite web|title=In_memory Analytics|url=http://www.yellowfinbi.com/Document.i4?DocumentId=104879|publisher=yellowfin|pages=6}}</ref>

== DetailedHow descriptiondoes of inIn-memory processing Work? ==

The arrival of [[Column-oriented_DBMS|column centric databases]], which storestored similar information together, allowed storing data to be stored more efficiently and compressedwith to a highergreater degreecompression. This allowsin theturn storageallowed ofto store huge amounts of data in the same physical space, which in turn reducesreduced the amount of memory needed to perform a query and increasesincreased the processing speed. With [[Inin-memory database|in-memory databases]], all information is initially loaded into memory,. therebyIt eliminatingeliminates the need for database-optimizing techniquesdatabase like creating indexes, [[Aggregateaggregates (dataand warehouse)|aggregates]],designing of cubes and [[star schema]]sschemas.

Most in-memory tools use [[Data compression|compression algorithms]] towhich reduce the size of in-memory data than what would be needed for hard disks. Users query the data loaded into the system’s memory, thereby avoiding slower database access, which is slower and has performance bottlenecks. This is different from caching, a very widely used method to speed up query performance, in that [[Cache_(computing)|caches]] are subsets of very specific pre-defined organized data. With in-memory tools, data available for analysis can be as large as data mart or small data warehouse which is stored entirely in memory. This can be accessed within seconds by multiple concurrent users at a detailed level and offers the potential for excellent analytics. Theoretically, the improvement in data access is 10,000 to 1,000,000 times faster than if the data was stored on a hardfrom disk. It also minimizes the need for performance tuning by IT staff and provides faster service tofor end users.

== Factors driving inIn-memory products ==

Cheaper and higher performing hardware: According to [[Moore’s law]] the computing power doubles every two to three years while decreasing in costs. CPU processing, memory and disk storage are all subject to some variation of this law. Also hardware innovations like multi-core architecture, NAND flash memory, parallel servers, increased memory processing capability, etc. and software innovations like column centric databases, compression techniques and handling aggregate tables, etc. have all contributed to the demand of In-memory products.<ref>{{cite web|last=Kote|first=Sparjan|title=In-memory computing in Business Intelligence|url=http://www.infosysblogs.com/oracle/2011/03/in-memory_computing_in_busines.html}}</ref>

[[64 bit|64-bit]]bits operating systemssystem: DueThough tothe affordableidea 64of In-bitmemory processorstechnology andis thenot decliningnew, pricesit ofis memoryonly chips,recently inemerging thanks to the widely popular and affordable 64-memorybit technologyprocessors isand beingdeclining usedmemory morechips widelyprices. The[[64 bit]] operating systems allowallows access to far more RAM (up to 100GB or more) than 32-bitthe systems (2 or 4 GB) accessible on 32-bit systems. By providing [[terabyte]]sTerabytes (1 TB = 1,024 GB) of space available for storage and analysis, 64-bit operating systems make in-memory processing scalable. FlashThe memoryuse alsoof flash memory enables systems to scale to many terabytesTerabytes more economically.

Data Volumes: As the data used by organizations grew traditional data warehouses just couldn’t deliver a timely, accurate and real time data. The extract, transform, load ([[Extract, transform, load|ETL]]) process that periodically updates data warehouses with operational data can take anywhere from a few hours to weeks to complete. So at any given point of time data is at least a day old. In-memory processing makes easy to have instant access to terabytes of data for real time reporting.

Reduced Costs: In-memory processing comes at a lower cost and can be easily deployed and maintained when compared to traditional BI tools. According to Gartner survey deploying traditional BI tools can take as long as 17 months. Many data warehouse vendors are choosing In-memory technology over traditional BI to speed up implementation times.