Array DBMS: Difference between revisions

Revision as of 09:25, 30 June 2014

Revision as of 15:38, 20 August 2014

Line 7:

== Overview ==

In the same style as standard [[Database Management System|database systems]] do on sets, Array DBMSs offer scalable, flexible storage and flexible retrieval/manipulation on arrays of (conceptually) unlimited size. As in practice arrays never appear standalone, such an array model normally is embedded into some overall data model, such as the relational model.

In the same style as standard [[Database Management System|database systems]] do on sets, Array DBMSs offer scalable, flexible storage and flexible retrieval/manipulation on arrays of (conceptually) unlimited size.

As in practice arrays never appear standalone, such an array model normally is embedded into some overall data model, such as the relational model.

Some systems implement arrays as an analogy to tables, some introduce arrays as an additional attribute type.

Management of arrays requires novel techniques, particularly due to the fact that traditional database tuples and objects tend to fit well into a single database page - a unit of disk access on server, typically 4 kB [[kilobyte|KB]] - while array objects easily can span several media. The prime task of the array storage manager is to give fast access to large arrays and sub-arrays. To this end, arrays get partitioned, during insertion, into so-called ''tiles'' or ''chunks'' of convenient size which then act as units of access during query evaluation.

The prime task of the array storage manager is to give fast access to large arrays and sub-arrays.

To this end, arrays get partitioned, during insertion, into so-called ''tiles'' or ''chunks'' of convenient size which then act as units of access during query evaluation.

Array DBMSs offer [[Data Manipulation Language|query languages]] giving [[Declarative programming|declarative]] access to such arrays, allowing to create, manipulate, search, and delete them.

Like with, e.g., [[SQL]], expressions of arbitrary complexity can be built on top of a set of core array operations. Due to the extensions made in the data and query model, Array DBMSs sometimes are subsumed under the [[NoSQL]] category, in the sense of "not only SQL". Query [[Query optimization|optimization]] and [[Parallel computing|parallelization]] are important for achieving [[scalability]]; actually, many array operators lend themselves well towards parallel evaluation, by processing each tile on separate nodes or cores.

Due to the extensions made in the data and query model, Array DBMSs sometimes are subsumed under the [[NoSQL]] category, in the sense of "not only SQL".

Query [[Query optimization|optimization]] and [[Parallel computing|parallelization]] are important for achieving [[scalability]]; actually, many array operators lend themselves well towards parallel evaluation, by processing each tile on separate nodes or cores.

Important application domains of Array DBMSs include Earth, Space, Life, and Social sciences, as well as the related commercial applications (such as [[Oil exploration|hydrocarbon exploration]] in industry and [[OLAP]] in business). The variety occurring can be observed, e.g., in geo data where 1-D environmental sensor time series, 2-D satellite images, 3-D x/y/t image time series and x/y/z geophysics data, as well as 4-D x/y/z/t climate and ocean data can be found.

The variety occurring can be observed, e.g., in geo data where 1-D environmental sensor time series, 2-D satellite images, 3-D x/y/t image time series and x/y/z geophysics data, as well as 4-D x/y/z/t climate and ocean data can be found.

== History and Statusstatus ==

The [[Relational database|relational data model]], which is prevailing today, does not directly support the array paradigm to the same extent as sets and tuples. [[International Organization for Standards|ISO]] [[SQL]] lists an array-valued attribute type, but this is only one-dimensional, with almost no operational support, and not usable for the [[#Application_Domains|application domains]] of Array DBMSs. Another option is to resort to [[Binary large object|BLOB]]s ("binary large objects") which are the equivalent to files: byte strings of (conceptually) unlimited length, but again without any query language functionality, such as multi-dimensional subsetting.

The [[Relational database|relational data model]], which is prevailing today, does not directly support the array paradigm to the same extent as sets and tuples. [[International Organization for Standards|ISO]] [[SQL]] lists an array-valued attribute type, but this is only one-dimensional, with almost no operational support, and not usable for the [[#Application_Domains|application domains]] of Array DBMSs.

Another option is to resort to [[Binary large object|BLOB]]s ("binary large objects") which are the equivalent to files: byte strings of (conceptually) unlimited length, but again without any query language functionality, such as multi-dimensional subsetting.

First significant work in going beyond BLOBs has been established with PICDMS.<ref>Chock, M., Cardenas, A., Klinger, A.: Database structure and manipulation capabilities of a picture database management system (PICDMS). IEEE ToPAMI, 6(4):484-492, 1984</ref> This system offers the precursor of a 2-D array query language, albeit still procedural and without suitable storage support.

A first declarative query language suitable for multiple dimensions and with an algebra-based semantics has been published by [[Peter Baumann (computer scientist)|Baumann]], together with a scalable architecture.<ref>Baumann, P.: [http://www.informatik.uni-trier.de/~ley/db/journals/vldb/vldb3.html#Baumann94 On the Management of Multidimensional Discrete Data]. VLDB Journal 4(3)1994, Special Issue on Spatial Database Systems, pp. 401 - 444</ref><ref>Baumann, P.: [http://www.informatik.uni-trier.de/~ley/db/conf/ngits/ngits99.html#Baumann99 A Database Array Algebra for Spatio-Temporal Data and Beyond]. Proc. NGITS’99, LNCS 1649, Springer 1999, pp.76-93</ref> Another array database language, constrained to 2-D, has been presented by Marathe and Salem.<ref>Marathe, A., Salem, K.: A language for manipulating arrays. Proc. VLDB’97, Athens, Greece, August 1997, pages 46 - 55</ref> Seminal theoretical work has been accomplished by Libkin et al.;<ref>Libkin, L., Machlin, R., Wong, L.: A query language for multidimensional arrays: design, implementation and optimization techniques. Proc. ACM SIGMOD’96, Montreal, Canada, pp. 228 - 239</ref> in their model, called NCRA, they extend a nested relational calculus with multidimensional arrays; among the results are important contributions on array query complexity analysis. A map algebra, suitable for 2-D and 3-D spatial raster data, has been published by Mennis et al.<ref>Mennis, J., Viger, R., Tomlin, C.D.: Cubic Map Algebra Functions for Spatio-Temporal Analysis. Cartography and Geographic Information Science 32(1)2005, pp. 17 - 32</ref>

Another array database language, constrained to 2-D, has been presented by Marathe and Salem.<ref>Marathe, A., Salem, K.: A language for manipulating arrays. Proc. VLDB’97, Athens, Greece, August 1997, pages 46 - 55</ref>

Seminal theoretical work has been accomplished by Libkin et al.;<ref>Libkin, L., Machlin, R., Wong, L.: A query language for multidimensional arrays: design, implementation and optimization techniques. Proc. ACM SIGMOD’96, Montreal, Canada, pp. 228 - 239</ref> in their model, called NCRA, they extend a nested relational calculus with multidimensional arrays; among the results are important contributions on array query complexity analysis.

A map algebra, suitable for 2-D and 3-D spatial raster data, has been published by Mennis et al.<ref>Mennis, J., Viger, R., Tomlin, C.D.: Cubic Map Algebra Functions for Spatio-Temporal Analysis. Cartography and Geographic Information Science 32(1)2005, pp. 17 - 32</ref>

In terms of Array DBMS implementations, the [[rasdaman]] system has the longest implementation track record of n-D arrays with full query support. [[Oracle Spatial|Oracle GeoRaster]] offers chunked storage of 2-D raster maps, albeit without SQL integration. [[TerraLib]] is an open-source GIS software that extends object-relational DBMS technology to handle spatio-temporal data types; while main focus is on vector data, there is also some support for rasters. Starting with version 2.0, [[Postgis|PostGIS]] embeds raster support for 2-D rasters; a special function offers declarative raster query functionality. [[SciQL]] is an array query language being added to the [[MonetDB]] DBMS. [[Michael_Stonebraker#SciDB|SciDB]] is a more recent initiative to establish array database support. Like SciQL, arrays are seen as an equivalent to tables, rather than a new attribute type as in rasdaman and PostGIS.

In terms of Array DBMS implementations, the [[rasdaman]] system has the longest implementation track record of n-D arrays with full query support.

[[Oracle Spatial|Oracle GeoRaster]] offers chunked storage of 2-D raster maps, albeit without SQL integration.

[[TerraLib]] is an open-source GIS software that extends object-relational DBMS technology to handle spatio-temporal data types; while main focus is on vector data, there is also some support for rasters.

Starting with version 2.0, [[Postgis|PostGIS]] embeds raster support for 2-D rasters; a special function offers declarative raster query functionality.

[[SciQL]] is an array query language being added to the [[MonetDB]] DBMS. [[Michael_Stonebraker#SciDB|SciDB]] is a more recent initiative to establish array database support. Like SciQL, arrays are seen as an equivalent to tables, rather than a new attribute type as in rasdaman and PostGIS.

For the special case of [[Sparse matrix|sparse data]], [[OLAP]] data cubes are well established; they store cell values together with their ___location - an adequate compression technique in face of the few locations carrying valid information at all - and operate with SQL on them. As this technique does not scale in density, standard databases are not used today for dense data, like satellite images, where most cells carry meaningful information; rather, proprietary ad-hoc implementations prevail in scientific data management and similar situations. Hence, this is where Array DBMSs can make a particular contribution.

As this technique does not scale in density, standard databases are not used today for dense data, like satellite images, where most cells carry meaningful information; rather, proprietary ad-hoc implementations prevail in scientific data management and similar situations. Hence, this is where Array DBMSs can make a particular contribution.

Generally, Array DBMSs are an emerging technology. While operationally deployed systems exist, like [[Oracle Spatial|Oracle GeoRaster]], [[Postgis|PostGIS 2.0]] and [[rasdaman]], there are still many open research questions, including query language design and formalization, query optimization, parallelization and distributed processing, and scalability issues in general. Besides, scientific communities still appear reluctant in taking up array database technology and tend to favor specialized, proprietary technology.

== Concepts ==

When adding arrays to databases, all facets of database design need to be reconsidered - ranging from conceptual modeling (such as suitable operators) over storage management (such as management of arrays spanning multiple media) to query processing (such as efficient processing strategies).

=== Conceptual Modelingmodeling ===

Formally, an array ''A'' is given by a (total or partial) function ''A'': ''X'' → ''V'' where ''X'', the ''___domain'' is a ''d''-dimensional integer interval for some ''d''>0 and ''V'', called ''range'', is some (non-empty) value set; in set notation, this can be rewritten as { (''p'',''v'') | ''p'' in ''X'', ''v'' in ''V'' }.

Each (''p'',''v'') in ''A'' denotes an array element or ''cell'', and following common notation we write ''A''[''p''] = ''v''.

Line 62 ⟶ 43:

At the same time, avoiding explicit loop sequences opens up manifold optimization opportunities.

=== Array Queryingquerying ===

As an example for array query operators the [[rasdaman]] algebra and query language can serve, which establish an expression language over a minimal set of array primitives.

We begin with the generic core operators and then present common special cases and shorthands.

Line 72 ⟶ 52:

values cell-value-expression

</source>

where ''index-range-specification'' defines the result ___domain and binds an iteration variable to it, without specifying iteration sequence. The ''cell-value-expression'' is evaluated at each ___location

of the ___domain.

Line 80 ⟶ 61:

values A[p]

</source>

This special case, pure subsetting, can be abbreviated as

<source lang="sql">

Line 90 ⟶ 72:

A[*:*,*:*,100]

</source>

The wildcard operator ''*'' indicates that the current boundary of the array is to be used; note that arrays where dimension boundaries are left open at definition time may change size in that dimensions over the array's lifetime.

Line 99 ⟶ 82:

</source>

This can be abbreviated as:

<source lang="sql">

log( A )

</source>

Through a principle called ''induced operations'',<ref>Ritter, G. and Wilson, J. and Davidson, J.: Image Algebra: An Overview. Computer Vision, Graphics, and Image Processing, 49(1)1994, 297-336</ref> the query language offers all operations the cell type offers on array level, too. Hence, on numeric values all the usual unary and binary arithmetic, exponential, and trigonometric operations are available in a straightforward manner, plus the standard set of Boolean operators.

Hence, on numeric values all the usual unary and binary arithmetic, exponential, and trigonometric operations are available in a straightforward manner, plus the standard set of Boolean operators.

The '''condense''' operator aggregates cell values into one scalar result, similar to SQL aggregates. Its application has the general form:

<source lang="sql">

condense condense-op

Line 113 ⟶ 95:

using cell-value-expression

</source>

As with ''marray'' before, the ''index-range-specification'' specifies the ___domain to be iterated over and binds an iteration variable to it - again, without specifying iteration sequence. Likewise, ''cell-value-expression'' is evaluated at each ___domain ___location.

As with ''marray'' before, the ''index-range-specification'' specifies the ___domain to be iterated over and binds an iteration variable to it - again, without specifying iteration sequence. Likewise, ''cell-value-expression'' is evaluated at each ___domain ___location. The ''condense-op'' clause specifies the aggregating operation used to combine the cell value expressions into one single value.

'''Example:''' "The sum over all values in A."

Line 122 ⟶ 104:

using A[p]

</source>

A shorthand for this operation is:

<source lang="sql">

add_cells( A )

</source>

In the same manner and in analogy to SQL aggregates, a number of further shorthands are provided, including counting, average, minimum, maximum, and Boolean quantifiers.

Line 134 ⟶ 118:

values count_cells( A = bucket )

</source>

The induced comparison, ''A=bucket'', establishes a Boolean array of the same extent as ''A''.

The aggregation operator counts the occurrences of ''true'' for each value of ''bucket'', which subsequently is put into the proper array cell of the 1-D histogram array.

The induced comparison, ''A=bucket'', establishes a Boolean array of the same extent as ''A''. The aggregation operator counts the occurrences of ''true'' for each value of ''bucket'', which subsequently is put into the proper array cell of the 1-D histogram array.

This concludes the brief presentation.

Such languages allow formulating statistical and imaging operations which can be expressed analytically without using loops. It has been proven<ref>Machlin, R.: Index-Based Multidimensional Array Queries: Safety and Equivalence. Proc. ACM PODS'07, Beijing, China, June 2007, pp. 175 - 184</ref> that the expressive power of such array languages in principle is equivalent to relational query languages with ranking.

=== Array Storagestorage ===

Array storage has to accommodate arrays of different dimensions and typically large sizes. A core task is to maintain spatial proximity on disk so as to reduce the number of disk accesses during subsetting. Note that an emulation of multi-dimensional arrays as nested lists (or 1-D arrays) will not per se accomplish this and, therefore, in general will not lead to scalable architectures.

Commonly arrays are partitioned into sub-arrays which form the unit of access. Regular partitioning where all partitions have the same size (except possibly for boundaries) is referred to as ''chunking''.<ref>Sarawagi, S., Stonebraker, M.: Efficient Organization of Large Multidimensional Arrays. Proc. ICDE'94, Houston, USA, 1994, pp. 328-336</ref> A generalization which removes the restriction to equally sized partitions by supporting any kind of partitioning is ''tiling''.<ref>Furtado, P., Baumann, P.: [http://www.informatik.uni-trier.de/~ley/db/conf/icde/icde99.html#FurtadoB99 Storage of Multidimensional Arrays based on Arbitrary Tiling]. Proc. ICDE'99, March 23–26, 1999, Sydney, Australia, pp. 328 - 336</ref> Array partitioning can improve access to array subsets significantly: by adjusting tiling to the access pattern, the server ideally can fetch all required data with only one disk access.

Array storage has to accommodate arrays of different dimensions and typically large sizes.

A core task is to maintain spatial proximity on disk so as to reduce the number of disk accesses during subsetting.

Note that an emulation of multi-dimensional arrays as nested lists (or 1-D arrays) will not per se accomplish this and, therefore, in general will not lead to scalable architectures.

Commonly arrays are partitioned into sub-arrays which form the unit of access.

Regular partitioning where all partitions have the same size (except possibly for boundaries) is referred to as ''chunking''.<ref>Sarawagi, S., Stonebraker, M.: Efficient Organization of Large Multidimensional Arrays. Proc. ICDE'94, Houston, USA, 1994, pp. 328-336</ref> A generalization which removes the restriction to equally sized partitions by supporting any kind of partitioning is ''tiling''.<ref>Furtado, P., Baumann, P.: [http://www.informatik.uni-trier.de/~ley/db/conf/icde/icde99.html#FurtadoB99 Storage of Multidimensional Arrays based on Arbitrary Tiling]. Proc. ICDE'99, March 23–26, 1999, Sydney, Australia, pp. 328 - 336</ref> Array partitioning can improve access to array subsets significantly: by adjusting tiling to the access pattern, the server ideally can fetch all required data with only one disk access.

Compression of tiles can sometimes reduce substantially the amount of storage needed. Also for transmission of results compression is useful, as for the large amounts of data under consideration networks bandwidth often constitutes a limiting factor.

=== Query Processingprocessing ===

A tile-based storage structure suggests a tile-by-tile processing strategy (in [[rasdaman]] called ''tile streaming''). A large class of practically relevant queries can be evaluated by loading tile after tile, thereby allowing servers to process arrays orders of magnitude beyoned their main memory.

Line 159 ⟶ 137:

Due to the massive sizes of arrays in scientific/technical applications in combination with often complex queries, optimization plays a central role in making array queries efficient. Both hardware and software parallelization can be applied. An example for heuristic optimization is the rule "averaging over an array resulting from the cell-wise addition of two input images is equivalent to adding the averages of each input array". By replacing the left-hand variant by the right-hand expression, costs shrink from three (costly) array traversals to two array traversals plus one (cheap) scalar operation (see Figure, which uses the [[rasdaman]] query language introduced before).

== Application Domainsdomains ==

In many -&ndash; if not most -&ndash; cases where some phenomenon is sampled or simulated the result is a rasterized data set which can conveniently be stored, retrieved, and forwarded as an array. Typically, the array data are ornamented with metadata describing them further; for example, geographically referenced imagery will carry its geographic position and the coordinate reference system in which it is expressed.

The following are representative domains in which large-scale multi-dimensional array data are handled:

Line 173 ⟶ 151:

== Standardization ==

Many communities have established data exchange formats, such as [[Hierarchical Data Format|HDF]], [[Netcdf|NetCDF]], and [[Tagged Image File Format|TIFF]].

A de facto standard in the Earth Science communities is [[Opendap|OPeNDAP]], a data transport architecture and protocol. While this is not a database specification, it offers important components that characterize a database system, such as a conceptual model and client/server implementations.

Line 180 ⟶ 157:

== List of Array DBMS ==

* [[GeoRaster|Oracle GeoRaster]]

* [[MonetDB]]

Line 192 ⟶ 168:

== See also ==

[[Data Intensive Computing]]

{{Database models}}