Content deleted Content added
citation needed for the Concurrency section |
|||
| (44 intermediate revisions by 37 users not shown) | |||
Line 1:
{{Short description|Software library providing an embedded transactional key-value database}}
{{Infobox software
| name = OpenLDAP Lightning Memory-Mapped Database
Line 16:
}}
{{Portal|Free and open-source software}}
'''Lightning Memory-Mapped Database''' ('''LMDB''') is
▲'''Lightning Memory-Mapped Database''' (LMDB) is a [[software library]] that provides a high-performance embedded transactional database in the form of a [[key-value store]]. LMDB is written in [[C (programming language)|C]] with [[#API and uses|API bindings]] for several [[programming language]]s. LMDB stores arbitrary key/data pairs as byte arrays, has a range-based search capability, supports multiple data items for a single key and has a special mode for appending records at the end of the database (MDB_APPEND) which gives a dramatic write performance increase over other similar stores.<ref name="auto">[http://symas.com/mdb/doc/group__internal.html LMDB Reference Guide] {{Webarchive|url=https://web.archive.org/web/20141020182433/http://symas.com/mdb/doc/group__internal.html |date=2014-10-20 }}. Retrieved on 2014-10-19</ref> LMDB is not a [[relational database]], it is strictly a key-value store like [[Berkeley DB]] and [[DBM (computing)|dbm]].
LMDB may also be used [[#Concurrency|concurrently]] in a multi-threaded or multi-processing environment, with read performance scaling linearly by design. LMDB databases may have only one writer at a time, however unlike many similar key-value databases, write transactions do ''not'' block readers, nor do readers block writers. LMDB is also unusual in that multiple applications on the same system may simultaneously open and use the same LMDB store, as a means to scale up performance. Also, LMDB does not require a transaction log (thereby increasing write performance by not needing to write data twice) because it maintains data integrity inherently by design.
Line 30 ⟶ 28:
== Technical description ==
Internally LMDB uses [[B+ tree]] data structures.
Specific noteworthy technical features of LMDB are:
* Its use of [[B+ tree]].
* New data is written without overwriting or moving existing data.
* The provision of a unique append-write mode (MDB_APPEND)<ref name="auto"/>
* [[Copy-on-write]] semantics help ensure [[#Reliability|data integrity]] as well as providing transactional guarantees and simultaneous access by readers without requiring any locking, even by the current writer.
* As LMDB is memory-mapped, it can return ''direct'' pointers to memory addresses of keys and values through its API, thereby avoiding unnecessary and expensive copying of memory.
* LMDB also tracks unused memory pages, using a [[B+ tree]] to keep track of pages freed (no longer needed) during transactions.
The file format of LMDB is, unlike that of [[Berkeley DB]], architecture-dependent. This means that a conversion must be done before moving a database from a 32-bit machine to a 64-bit machine,<ref>{{cite web |title=The LMDB file format |url=https://blog.separateconcerns.com/2016-04-03-lmdb-format.html |website=Separate Concern |accessdate=27 February 2020}}</ref> or between computers of differing [[endianness]].<ref>{{cite web |last1=Chu |first1=Howard |title=lmdb - Is the Monero blockchain database portable between 32 and 64 bit architectures, and little/big endian architectures? |url=https://monero.stackexchange.com/a/4526 |website=Monero Stack Exchange}}</ref>
=== Concurrency ===
LMDB employs [[multiversion concurrency control]] (MVCC) and allows multiple threads within multiple processes to coordinate simultaneous access to a database. Readers scale linearly by design.<ref>[http://www.lmdb.tech/bench/inmem/scaling.html scaling benchmarks for LMDB]</ref><ref>[http://www.lmdb.tech/bench/inmem/scale2/ in-memory benchmark scaling for LMDB]</ref> While write transactions are globally serialized via a [[mutex]], read-only transactions operate in parallel, including in the presence of a write transaction
== Performance ==
In 2011, Google published software
The benchmarks showed the underlying filesystem as having a big influence on performance. [[JFS (file system)|JFS]] with an external journal performs well, especially compared to other modern systems like [[Btrfs]] and [[ZFS]].<ref>{{cite web | url=http://lmdb.tech/bench/microbench/|title=MDB Microbenchmarks|publisher=Symas Corp.}}, 2012-09</ref><ref>[http://lmdb.tech/bench/microbench/july/ Database Microbenchmarks], Symas Corp., 2012-07.</ref>
Zimbra has tested back-mdb vs back-hdb performance in OpenLDAP, with LMDB clearly outperforming the BDB based back-hdb.<ref>{{cite web | url=http://wiki.zimbra.com/wiki/OpenLDAP_MDB_vs_HDB_performance|title=OpenLDAP MDB vs HDB performance|publisher=Zimbra, Inc.}}</ref> Many other OpenLDAP users have observed similar benefits.<ref>{{cite web|url=https://mishikal.wordpress.com/2013/05/16/openldap-a-comparison-of-back-mdb-and-back-hdb-performance/|title=OpenLDAP: A comparison of back-mdb and back-hdb performance|date=16 May 2013|accessdate=8 May 2017}}</ref>
Since the initial benchmarking work done in 2012, multiple follow-on tests have been conducted with additional database engines for both in-memory
== Reliability ==
LMDB was designed
Two academic papers from the
When provided with a corrupt database, such as one produced by [[fuzzing]], LMDB may crash. LMDB's author considers the case unlikely to be concerning
== Open source license ==
In June 2013, [[Oracle Corporation|Oracle]] changed the license of [[Berkeley DB]] (a related project) from the [[Sleepycat license]] to the [[Affero General Public License]],<ref>{{cite web | url=https://oss.oracle.com/pipermail/bdb/2013-June/000056.html|title=Berkeley DB Release Announcement|date=11 June 2013 |publisher=[[Oracle Corporation]] |quote=Starting with the 6.0 / 12c releases, all Berkeley DB products are licensed under the GNU AFFERO GENERAL PUBLIC LICENSE (AGPL), version 3. This license is published by the Free Software Foundation (FSF) (1) and approved by the Open Source Initiative (2).
The Berkeley DB license issue has caused major
== API and uses ==
There are wrappers for several programming languages, such as C++,<ref>[https://github.com/bendiken/lmdbxx LMDB C++11 wrapper], 2015-04</ref><ref>[https://code.google.com/p/libglim/source/browse/trunk/mdb.hpp LMDB C++ wrapper], 2012-11.</ref> Java,<ref>[https://github.com/lmdbjava/lmdbjava LmdbJava], 2019-04</ref> Python,<ref>[https://github.com/dw/py-lmdb/ LMDB Python wrapper], 2013-02</ref><ref>[http://lmdb.readthedocs.org py-lmdb]. Retrieved on 2014-10-20.</ref> Lua,<ref>[https://github.com/shmul/lightningdbm LMDB Lua wrapper], 2013-04.</ref> Rust,<ref>[https://github.com/meilisearch/heed typed LMDB Rust wrapper], 2023-01</ref><ref>[https://github.com/mozilla/rkv high-level Rust wrapper], 2022-12</ref> Go,<ref>[https://github.com/
Howard Chu ported [[SQLite]] 3.7.7.1 to use LMDB instead of its original [[B-tree]] code, calling the
An up-to-date list of applications using LMDB is maintained on the main web site.<ref>{{cite web|url=
▲An up-to-date list of applications using LMDB is maintained on the main web site.<ref>{{cite web|url=http://symas.com/mdb/#projects|title=List of projects using LMDB}}</ref>
== Application support ==
Many popular [[free software]] projects distribute or include support for LMDB, often as the primary or sole storage mechanism.
* The [[Debian]],<ref>[https://packages.debian.org/liblmdb0 liblmdb0 in Debian]. Retrieved 2014-10-20.</ref> [[Ubuntu (operating system)|Ubuntu]],<ref>{{cite web|url=https://packages.ubuntu.com/lmdb-utils|title=Ubuntu – Package Search Results -- lmdb-utils|first=Rhonda
* [[OpenLDAP]] for which LMDB was originally developed via {{mono|back-mdb}}.<ref>[http://www.openldap.org/doc/admin24/backends.html#LMDB OpenLDAP back-mdb]. Retrieved 2014-10-20</ref>
* [[Postfix (software)|Postfix]] via the {{mono|lmdb_table}} adapter.<ref>[http://www.postfix.org/lmdb_table.5.html Postfix lmdb_table(5)]. Retrieved 2014-10-20</ref>
Line 87 ⟶ 82:
* [[Knot DNS]] a high performance DNS server.
* [[Monero (cryptocurrency)|Monero]] an open source cryptocurrency created in April 2014 that focuses on privacy, decentralisation and scalability.
* [[Enduro/X]] middleware uses LMDB for optional XATMI Microservices (SOA) cache.
* [[Samba (software)|Samba]] Active Directory Domain Controller
* [[Nano (cryptocurrency)|Nano]] a peer-to-peer, open source cryptocurrency created in 2015 that prioritizes fast and fee-less transactions.
* Meilisearch an open source, lightning-fast, easy-to-use, and hyper-relevant search engine.<ref>{{cite web|url=https://docs.meilisearch.com/learn/advanced/storage.html|title=Storage | Meilisearch Documentation v1.0|access-date=21 Mar 2023}}</ref>
* LMDB-IndexedDB is a JavaScript wrapper around [[IndexedDB]] to provide support for LMDB in web browsers.<ref>{{cite web|url=https://github.com/anywhichway/lmdb-indexeddb|title=LMDB-IndexedDB on GitHub|website=[[GitHub]] |access-date=2 Apr 2023}}</ref>
== Technical reviews of LMDB ==
LMDB makes novel use of well-known computer science techniques such as [[copy-on-write]] semantics and [[B+ tree]]s to provide atomicity and reliability guarantees as well as performance that can be hard to accept, given the library's relative simplicity and that no other similar [[key-value store]] database offers the same guarantees or overall performance, even though the authors ''explicitly state'' in presentations that LMDB is read-optimised not write-optimised.
The presentation did spark other database developers
Multiple other reviews cover LMDB<ref>{{cite web|url=https://mozilla.github.io/firefox-browser-architecture/text/0015-rkv.html|title=Design Review: Key-Value Storage|website=mozilla.github.io|quote=We propose the standardization of a simple key-value storage capability, based on LMDB, that is fast, compact, multi-process-capable, and equally usable from JS, Java, Rust, Swift, and C++.}}</ref><ref>{{cite web|url=http://sampathherga.in/lmdb/|title=LMDB|publisher=Sampath Herga|access-date=2013-08-30|archive-url=https://web.archive.org/web/20130829025217/http://sampathherga.in/lmdb/|archive-date=2013-08-29|url-status=dead}}</ref> in various languages including Chinese.<ref>{{cite web | url=http://jianshu.io/p/yzFf8j|title=lmdb简介 - 简书}}</ref>
▲The presentation did spark other database developers dissecting the code in-depth to understand how and why it works. Reviews run from brief <ref>{{cite web | url=http://kellabyte.com/2013/07/09/lightning-memory-mapped-database/|title=Lightning Memory-Mapped Database}}</ref> to in-depth. Database developer Oren Eini wrote a 12-part series of articles on his analysis of LMDB, beginning July 9, 2013. The conclusion was in the lines of "impressive codebase ... dearly needs some love", mainly because of too long methods and code duplication.<ref>{{cite web | url=http://ayende.com/blog/162754/reviewing-lightning-memory-mapped-database-library-partial|title=Reviewing Lightning memory-mapped database library: Partial}}</ref> This review, conducted by a .NET developer with no former experience of [[C (programming language)|C]], concluded on August 22, 2013 with "beyond my issues with the code, the implementation is really quite brilliant. The way LMDB manages to pack so much functionality by not doing things is quite impressive... I learned quite a lot from the project, and it has been frustrating, annoying and fascinating experience"<ref>{{cite web | url=http://ayende.com/blog/162917/some-final-notes-about-lmdb-review|title=Some final notes about LMDB review}}</ref>
==See also==
* [[Ordered Key-Value Store]]
* [[libmdbx]] (aka MDBX) is a deeply revised and extended descendant of LMDB, which according to the developers, "is superior to legendary LMDB in terms of features and reliability, not inferior in performance". It is noteworthy MDBX is used in [[Ethereum]] (Erigon, Reth), in products of [[StarkWare Industries]], [[Positive Technologies]], as well as in many other software projects.
== References ==
| |||