Content deleted Content added
m clean up, typo(s) fixed: August 22, 2013 → August 22, 2013, |
|||
| (8 intermediate revisions by 8 users not shown) | |||
Line 1:
{{Short description|Software library providing an embedded transactional key-value database}}
{{Infobox software
| name = OpenLDAP Lightning Memory-Mapped Database
Line 17 ⟶ 16:
}}
{{Portal|Free and open-source software}}
'''Lightning Memory-Mapped Database''' ('''LMDB''') is an embedded transactional [[database]] in the form of a [[key-value store]]. LMDB is written in [[C (programming language)|C]] with [[#API and uses|API bindings]] for several [[programming language]]s. LMDB stores arbitrary key/data pairs as byte arrays, has a range-based search capability, supports multiple data items for a single key and has a special mode for appending records (MDB_APPEND) without checking for consistency.<ref name="auto">[http://www.lmdb.tech/doc/ LMDB Reference Guide] Retrieved on 2023-03-21</ref> LMDB is not a [[relational database]], it is strictly a key-value store like [[Berkeley DB]] and [[DBM (computing)|DBM]].
LMDB may also be used [[#Concurrency|concurrently]] in a multi-threaded or multi-processing environment, with read performance scaling linearly by design. LMDB databases may have only one writer at a time, however unlike many similar key-value databases, write transactions do ''not'' block readers, nor do readers block writers. LMDB is also unusual in that multiple applications on the same system may simultaneously open and use the same LMDB store, as a means to scale up performance. Also, LMDB does not require a transaction log (thereby increasing write performance by not needing to write data twice) because it maintains data integrity inherently by design.
Line 33 ⟶ 32:
Specific noteworthy technical features of LMDB are:
* Its use of [[B+ tree]]. With an LMDB instance being in shared memory and the [[B+ tree]] block size being set to the OS page size, access to an LMDB store is extremely memory efficient.<ref>[[B+ tree#Implementation]]</ref>
* New data is written without overwriting or moving existing data. This guarantees data integrity and [[#Reliability|reliability]] without requiring transaction logs or cleanup services.
* The provision of a unique append-write mode (MDB_APPEND)<ref name="auto"/> is implemented by allowing the new record to be added directly to the end of the [[B+ tree]]. This reduces the number of reads and writes page operations, resulting in greatly-increased performance but requiring the programmer to ensure keys are already in sorted order when storing in the DB.
Line 56 ⟶ 55:
LMDB was designed to resist data loss in the face of system and application crashes. Its [[copy-on-write]] approach never overwrites currently-in-use data. Avoiding overwrites means the structure on disk/storage is always valid, so application or system crashes can never leave the database in a corrupted state. In its default mode, at worst, a crash can lose data from the last not-yet-committed write transaction. Even with all asynchronous modes enabled, it is only an OS catastrophic failure or hardware power-loss<ref>{{cite web | url=https://bugs.openldap.org/show_bug.cgi?id=7668|title=LMDB Corruption detection}}</ref> event rather than merely an application crash that could potentially result in any data corruption.
Two academic papers from the USENIX OSDI Symposium<ref>{{cite web | url=https://www.usenix.org/conference/osdi14|title=OSDI 2014|date=2013-02-08}}</ref> covered failure modes of DB engines (including LMDB) under a sudden power loss or system crash.<ref>{{cite book | url=https://www.usenix.org/conference/osdi14/technical-sessions/presentation/pillai|title=OSDI 2014, All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications|pages=433–448|isbn=9781931971164|year=2014|last1=Langston|first1=Mark C.|last2=Skelly|first2=Hal}}</ref><ref>{{cite book | url=https://www.usenix.org/conference/osdi14/technical-sessions/presentation/zheng_mai|title=OSDI 2014, Torturing Databases for Fun and Profit|pages=449–464|isbn=9781931971164|year=2014|last1=Langston|first1=Mark C.|last2=Skelly|first2=Hal}}</ref> The paper from Pillai et al., did not find any failure in LMDB that would occur in the real-world file systems considered; the single failure identified by the study in LMDB only relates to hypothetical file systems.<ref>{{cite web | url=http://www.openldap.org/lists/openldap-devel/201410/msg00004.html|title=Archive of discussion regarding the Usenix 2014 pillai paper}}</ref> The Mai Zheng et al. paper claims to point out failures in LMDB, but the conclusion depends on whether fsync or fdatasync is utilised. Using fsync ameliorates the problem. The selection of fsync
When provided with a corrupt database, such as one produced by [[fuzzing]], LMDB may crash. LMDB's author considers the case unlikely to be concerning but has produced a partial fix in a separate branch.<ref name=fuzz>{{cite web |last1=Debroux |first1=Lionel |title=oss-security - Fun with DBM-type databases... |url=https://www.openwall.com/lists/oss-security/2018/06/17/1 |website=openwall.com |date=16 Jun 2018}}</ref>
Line 66 ⟶ 65:
== API and uses ==
There are wrappers for several programming languages, such as C++,<ref>[https://github.com/bendiken/lmdbxx LMDB C++11 wrapper], 2015-04</ref><ref>[https://code.google.com/p/libglim/source/browse/trunk/mdb.hpp LMDB C++ wrapper], 2012-11.</ref> Java,<ref>[https://github.com/lmdbjava/lmdbjava LmdbJava], 2019-04</ref> Python,<ref>[https://github.com/dw/py-lmdb/ LMDB Python wrapper], 2013-02</ref><ref>[http://lmdb.readthedocs.org py-lmdb]. Retrieved on 2014-10-20.</ref> Lua,<ref>[https://github.com/shmul/lightningdbm LMDB Lua wrapper], 2013-04.</ref> Rust,<ref>[https://github.com/meilisearch/heed typed LMDB Rust wrapper], 2023-01</ref><ref>[https://github.com/mozilla/rkv high-level Rust wrapper], 2022-12</ref> Go,<ref>[https://github.com/bmatsuo/lmdb-go LMDB Go wrapper], 2013-03</ref> Ruby,<ref>[https://github.com/minad/lmdb LMDB Ruby wrapper], 2013-02</ref> Objective C,<ref>[https://github.com/rbartolome/LMDBKit LMDB Objective-C wrapper], 2013-04</ref> Objective CAML,<ref>[https://github.com/Drup/ocaml-lmdb OCaml lmdb bindings], 2025-08</ref> Javascript,<ref>[https://github.com/Venemo/node-lmdb LMDB Node.js wrapper], 2013-05</ref> C#,<ref>[https://github.com/ilyalukyanov/Lightning.NET LMDB .Net wrapper], 2013-06</ref> Perl,<ref>[http://search.cpan.org/dist/LMDB_File/ LMDB Perl wrapper], 2013-08</ref> PHP,<ref>[https://github.com/mpremus/lmdb-php LMDB PHP wrapper], 2015-04</ref> Tcl<ref>[https://core.tcl.tk/jenglish/gutter/packages/lmdb.html tcl-lmdb], 2015-11</ref> and Common Lisp.<ref>[http://eudoxia.me/article/lmdb-from-common-lisp Using LMDB from Common Lisp], 2016-04</ref> A complete list of wrappers may be found on the main web site.<ref>{{cite web|url=https://www.symas.com/symas-lmdb-tech-info|title=Symas LMDB Tech Info}}</ref>
Howard Chu ported [[SQLite]] 3.7.7.1 to use LMDB instead of its original [[B-tree]] code, calling the result SQLightning.<ref>{{cite web|url=http://gitorious.org/mdb/sqlightning|title=gitorious.org Git - mdb:sqlightning.git/summary|website=gitorious.org|accessdate=8 May 2017|archive-url=https://web.archive.org/web/20130809145553/http://gitorious.org/mdb/sqlightning|archive-date=9 August 2013|url-status=live}}</ref> One cited insert test of 1000 records was 20 times faster (than the original SQLite with its B-Tree implementation).<ref>[http://pastebin.com/B5SfEieL SQLightning tests].</ref> LMDB is available as a backing store for other open source projects including Cyrus SASL,<ref>{{cite web|url=http://cyrusimap.web.cmu.edu/|title=Cyrus IMAP — Cyrus IMAP 3.0.1 (stable) documentation|website=cyrusimap.web.cmu.edu|accessdate=8 May 2017|url-status=dead|archive-url=https://web.archive.org/web/20170430023937/http://cyrusimap.web.cmu.edu/|archive-date=30 April 2017}}</ref> Heimdal Kerberos,<ref>{{cite web|url=http://h5l.org/|title=Heimdal|website=h5l.org|accessdate=8 May 2017}}</ref> and OpenDKIM.<ref>{{cite web|url=http://www.opendkim.org/|title=OpenDKIM|website=www.opendkim.org|accessdate=8 May 2017}}</ref> It is also available in some other NoSQL projects like MemcacheDB <ref>{{cite web|url=https://gitorious.org/mdb/memcachedb|title=gitorious.org Git - mdb:memcachedb.git/summary|website=gitorious.org|accessdate=8 May 2017}}</ref> and Mapkeeper.<ref>{{cite web|url=https://github.com/m1ch1/mapkeeper/|title=GitHub - m1ch1/mapkeeper: Thrift based key-value store with various storage backends, including MySQL, Berkeley DB, and LevelDB.|website=github.com|url-status=dead|archive-url=https://web.archive.org/web/20160209024233/https://github.com/m1ch1/mapkeeper/|archive-date=9 February 2016}}</ref> LMDB was used to make the in-memory store [[Redis]] persist data on disk. The existing back-end in [[Redis]] showed pathological behaviour in rare cases, and a replacement was sought. The baroque API of LMDB was criticized though, forcing a lot of coding to get simple things done. However, its performance and reliability during testing was considerably better than the alternative back-end stores that were tried.<ref>{{cite web|url=http://www.anchor.com.au/blog/2013/05/second-strike-with-lightning/|title=Second Strike With Lightning|publisher=Anchor|date=2013-05-09}}</ref>
An up-to-date list of applications using LMDB is maintained on the main web site.<ref>{{cite web|url=https://www.symas.com/symas-lmdb-tech-info|title=List of projects using LMDB}}</ref>
Line 85 ⟶ 82:
* [[Knot DNS]] a high performance DNS server.
* [[Monero (cryptocurrency)|Monero]] an open source cryptocurrency created in April 2014 that focuses on privacy, decentralisation and scalability.
* [[Enduro/X]] middleware uses LMDB for optional XATMI Microservices (SOA) cache.
* [[Samba (software)|Samba]] Active Directory Domain Controller
* [[Nano (cryptocurrency)|Nano]] a peer-to-peer, open source cryptocurrency created in 2015 that prioritizes fast and fee-less transactions.
Line 96 ⟶ 93:
The presentation did spark other database developers to dissect the code in-depth to understand how and why it works. Reviews run from brief <ref>{{cite web | url=http://kellabyte.com/2013/07/09/lightning-memory-mapped-database/|title=Lightning Memory-Mapped Database|archive-url=https://web.archive.org/web/20160314133119/http://kellabyte.com/2013/07/09/lightning-memory-mapped-database/|archive-date=14 March 2016}}</ref> to in-depth. Database developer Oren Eini wrote a 12-part series of articles on his analysis of LMDB, beginning July 9, 2013. The conclusion was in the lines of "impressive codebase ... dearly needs some love", mainly because of too long methods and code duplication.<ref>{{cite web | url=http://ayende.com/blog/162754/reviewing-lightning-memory-mapped-database-library-partial|title=Reviewing Lightning memory-mapped database library: Partial}}</ref> This review, conducted by a .NET developer with no former experience of [[C (programming language)|C]], concluded on August 22, 2013, with "beyond my issues with the code, the implementation is really quite brilliant. The way LMDB manages to pack so much functionality by not doing things is quite impressive... I learned quite a lot from the project, and it has been frustrating, annoying and fascinating experience".<ref>{{cite web | url=http://ayende.com/blog/162917/some-final-notes-about-lmdb-review|title=Some final notes about LMDB review}}</ref>
Multiple other reviews cover LMDB<ref>{{cite web|url=https://mozilla.github.io/firefox-browser-architecture/text/0015-rkv.html|title=Design Review: Key-Value Storage|website=mozilla.github.io|quote=We propose the standardization of a simple key-value storage capability, based on LMDB, that is fast, compact, multi-process-capable, and equally usable from JS, Java, Rust, Swift, and C++.}}</ref><ref>{{cite web|url=http://sampathherga.in/lmdb/|title=LMDB|publisher=Sampath Herga|access-date=2013-08-30|archive-url=https://web.archive.org/web/20130829025217/http://sampathherga.in/lmdb/|archive-date=2013-08-29|url-status=dead}}</ref> in various languages including Chinese.<ref>{{cite web | url=http://jianshu.io/p/yzFf8j|title=lmdb简介 - 简书}}</ref>
= * [[Ordered Key-Value Store]]
* [[libmdbx]] (aka MDBX) is a deeply revised and extended descendant of LMDB, which according to the developers, "is superior to legendary LMDB in terms of features and reliability, not inferior in performance". It is noteworthy MDBX is used in [[Ethereum]] (Erigon, Reth), in products of [[StarkWare Industries]], [[Positive Technologies]], as well as in many other software projects.
== References ==
| |||