In-memory processing: Difference between revisions

Revision as of 13:29, 8 June 2013

Revision as of 05:21, 4 September 2013

Line 5:

{{Ad|date=June 2013}}

}}

{{merge|In-memory database|discuss=Talk:THISIn-Memory PAGEProcessing#Merger proposal|date=January 2012}}

== Definition ==

With businesses demanding faster and easy access to information in order to make reliable and smart decisions, In-memory processing is an emerging technology that is gaining attention. It enables users to have immediate access to right information which results in more informed decisions. Traditional [[Business Intelligence]] (BI) technology loads data onto the computer's hard disk in the form of tables and [[OLAP cube|multi-dimensional cubes]] against which queries are run. Using In-memory data is loaded into memory ([[Random Access Memory|RAM]] (RAM) or [[flash memory|flash]]) instead of hard disks and therefore information technology (IT) staff spendsspend less development time on [[data modeling]], query analysis, cube building and table design.<ref>{{cite book|last=Earls|first=A|title=Tips on evaluating, deploying and managing in-memory analytics tools|year=2011|publisher=Tableau|url=http://www.analyticsearches.com/site/files/776/66977/259607/579091/In-Memory_Analytics_11.10.11.pdf}}</ref>

== Traditional BIBusiness Intelligence technology ==

Historically, every computer has two types of data storage mechanisms: –the disk (hard disk) and RAM (Random Access Memory). Modern computers have more available disk storage than RAM, but reading data from the disk is much slower (possibly hundreds of times) when compared to reading the same data from RAM. EspeciallyWhen whenusing analyzingtraditional largedisk volumestechnology of([[Relational data,Database performanceManagement isSystems]] severelysuch degraded.as Using[[MySQL]] traditionaland disk[[Oracle baseddatabase|Oracle]]), technology thea query accesses information from multiple tables stored on athe server’s hard disk. Traditional disk based technologies means [[Relational Database Management Systems]] such as SQL Server, MySQL, [[Oracle database|Oracle]] and many others. RDMS are designed keepingaround using transactional processing in mind. Having a database that supports both insertions, and updates as well as performingperforms aggregations, and joins (typical in BI solutions) is not possible. Alsofor thethese structuredtraditional querysystems. languageAlso, ([[SQL]]) is designed to more efficiently fetch rows of data whilethan BI queries, which usually involve fetching of partial rows of data involving heavy calculations.

Though SQL is a very powerful tool for running complex queries, tookit verytakes a long time to execute and often resultedresults in bringing down transactional processing. To improve query performance, multidimensional databases or cubes (also called [[MOLAP|multidimensional online analytical processing (MOLAP]]) were formed. Designing a cube design involved an elaborate and lengthy process whichand tookit awas significantcumbersome amountto ofchange time from IT staff. Changing the cubesits structure to adapt to dynamically changing business needs was cumbersome. Cubes are pre -populated with data to answer specific queries and although itthey increased performance, itthey still failed to answer ad hoc queries.<ref>{{cite journal|last=Gill|first=John|title=Shifting the BI Paradigm with In-Memory Database Technologies|journal=Business Intelligence Journal|year=2007|month=Second Quarter|volume=12|issue=2|pages=58–62|url=http://www.highbeam.com/doc/1P3-1636785121.html}}</ref>

== Disadvantages of traditional BIBusiness Intelligence ==

To avoid performance issues and provide faster query processing when dealing with large volumes of data, organizations needed optimized database methods like creating [[index (database)|index]]es, useand using specialized data structures and [[Aggregate (data warehouse)|aggregate tables]].

The point of having a data warehouse is to be able to get results for any queries asked at any time. ButHowever, in order to achieve better response time for userstimes, many [[data martsmart]]s are designed to pre calculateprecalculate summaries and answer specific queries, defeatingwhich defeats the purpose of a [[data warehouse]]. Optimized aggregation algorithms needed to be used to increase performance. Traditional BI tools were neither able to keep up with BI requirements, nor were they able to deliver real-time data to the end user.<ref>{{cite web|title=In_memory Analytics|url=http://www.yellowfinbi.com/Document.i4?DocumentId=104879|publisher=yellowfin|pages=6}}</ref>

Traditional BI tools couldn’t keep up with the ever growing BI requirements and were unable to deliver real time data for end users.<ref>{{cite web|title=In_memory Analytics|url=http://www.yellowfinbi.com/Document.i4?DocumentId=104879|publisher=yellowfin|pages=6}}</ref>

== HowDetailed doesdescription Inof in-memory processing Work? ==

The arrival of [[Column-oriented_DBMS|column centric databases]], which storedstore similar information together, allowed storing data to be stored more efficiently and withcompressed greaterto a higher compressiondegree. This inallows turnthe allowedstorage to storeof huge amounts of data in the same physical space, which in turn reducedreduces the amount of memory needed to perform a query and increasedincreases the processing speed. With in[[In-memory database|in-memory databases]], all information is initially loaded into memory., Itthereby eliminateseliminating the need for database-optimizing databasetechniques like creating indexes, aggregates[[Aggregate and(data designing ofwarehouse)|aggregates]], cubes and [[star schemasschema]]s.

Most in-memory tools use [[Data compression|compression algorithms]] whichto reduce the size of in-memory data than what would be needed for hard disks. Users query the data loaded into the system’s memory, thereby avoiding slower database access, which is slower and has performance bottlenecks. This is different from caching, a very widely used method to speed up query performance, in that [[Cache_(computing)|caches]] are subsets of very specific pre-defined organized data. With in-memory tools, data available for analysis can be as large as data mart or small data warehouse which is stored entirely in memory. This can be accessed within seconds by multiple concurrent users at a detailed level and offers the potential for excellent analytics. Theoretically, the improvement in data access is 10,000 to 1,000,000 times faster than fromif the data was stored on a hard disk. It also minimizes the need for performance tuning by IT staff and provides faster service forto end users.

== Factors driving Inin-memory products ==

Cheaper and higher performing hardware: According to [[Moore’s law]] the computing power doubles every two to three years while decreasing in costs. CPU processing, memory and disk storage are all subject to some variation of this law. Also hardware innovations like multi-core architecture, NAND flash memory, parallel servers, increased memory processing capability, etc. and software innovations like column centric databases, compression techniques and handling aggregate tables, etc. have all contributed to the demand of In-memory products.<ref>{{cite web|last=Kote|first=Sparjan|title=In-memory computing in Business Intelligence|url=http://www.infosysblogs.com/oracle/2011/03/in-memory_computing_in_busines.html}}</ref>

[[64 bit|64-bitsbit]] operating systemsystems: ThoughDue theto ideaaffordable of In64-memorybit technologyprocessors isand notthe new,declining itprices isof onlymemory recentlychips, emerging thanks to the widely popular and affordable 64in-bitmemory processorstechnology andis decliningbeing memoryused chipsmore priceswidely. [[64 bit]]The operating systems allowsallow access to far more RAM (up to 100GB or more) than the32-bit systems (2 or 4 GB accessible on 32-bit systems). By providing Terabytes (1 TB = 1,024 GB)[[terabyte]]s of space available for storage and analysis, 64-bit operating systems make in-memory processing scalable. TheFlash usememory of flash memoryalso enables systems to scale to many Terabytesterabytes more economically.

Data Volumes: As the data used by organizations grew traditional data warehouses just couldn’t deliver a timely, accurate and real time data. The extract, transform, load ([[Extract, transform, load|ETL]]) process that periodically updates data warehouses with operational data can take anywhere from a few hours to weeks to complete. So at any given point of time data is at least a day old. In-memory processing makes easy to have instant access to terabytes of data for real time reporting.

Reduced Costs: In-memory processing comes at a lower cost and can be easily deployed and maintained when compared to traditional BI tools. According to Gartner survey deploying traditional BI tools can take as long as 17 months. Many data warehouse vendors are choosing In-memory technology over traditional BI to speed up implementation times.