Content deleted Content added
m →Performance: Added comma |
|||
| (20 intermediate revisions by 19 users not shown) | |||
Line 1:
{{Short description|Software library providing an embedded transactional key-value database}}
{{Infobox software
| name = OpenLDAP Lightning Memory-Mapped Database
Line 17 ⟶ 16:
}}
{{Portal|Free and open-source software}}
'''Lightning Memory-Mapped Database''' ('''LMDB''') is
LMDB may also be used [[#Concurrency|concurrently]] in a multi-threaded or multi-processing environment, with read performance scaling linearly by design. LMDB databases may have only one writer at a time, however unlike many similar key-value databases, write transactions do ''not'' block readers, nor do readers block writers. LMDB is also unusual in that multiple applications on the same system may simultaneously open and use the same LMDB store, as a means to scale up performance. Also, LMDB does not require a transaction log (thereby increasing write performance by not needing to write data twice) because it maintains data integrity inherently by design.
Line 29 ⟶ 28:
== Technical description ==
Internally LMDB uses [[B+ tree]] data structures.
Specific noteworthy technical features of LMDB are:
* Its use of [[B+ tree]].
* New data is written without overwriting or moving existing data. This guarantees data integrity and [[#Reliability|reliability]] without requiring transaction logs or cleanup services.
* The provision of a unique append-write mode (MDB_APPEND)<ref name="auto"/> is implemented by allowing the new record to be added directly to the end of the [[B+ tree]]. This reduces the number of reads and writes page operations, resulting in greatly-increased performance but requiring the programmer to ensure keys are already in sorted order when storing in the DB.
* [[Copy-on-write]] semantics help ensure [[#Reliability|data integrity]] as well as providing transactional guarantees and simultaneous access by readers without requiring any locking, even by the current writer.
* As LMDB is memory-mapped, it can return ''direct'' pointers to memory addresses of keys and values through its API, thereby avoiding unnecessary and expensive copying of memory. This results in greatly-increased performance (especially when the values stored are extremely large), and expands the potential use cases for LMDB.
* LMDB also tracks unused memory pages, using a [[B+ tree]] to keep track of pages freed (no longer needed) during transactions.
The file format of LMDB is, unlike that of [[Berkeley DB]], architecture-dependent. This means that a conversion must be done before moving a database from a 32-bit machine to a 64-bit machine,<ref>{{cite web |title=The LMDB file format |url=https://blog.separateconcerns.com/2016-04-03-lmdb-format.html |website=Separate Concern |accessdate=27 February 2020}}</ref> or between computers of differing [[endianness]].<ref>{{cite web |last1=Chu |first1=Howard |title=lmdb - Is the Monero blockchain database portable between 32 and 64 bit architectures, and little/big endian architectures? |url=https://monero.stackexchange.com/a/4526 |website=Monero Stack Exchange}}</ref>
Line 56 ⟶ 55:
LMDB was designed to resist data loss in the face of system and application crashes. Its [[copy-on-write]] approach never overwrites currently-in-use data. Avoiding overwrites means the structure on disk/storage is always valid, so application or system crashes can never leave the database in a corrupted state. In its default mode, at worst, a crash can lose data from the last not-yet-committed write transaction. Even with all asynchronous modes enabled, it is only an OS catastrophic failure or hardware power-loss<ref>{{cite web | url=https://bugs.openldap.org/show_bug.cgi?id=7668|title=LMDB Corruption detection}}</ref> event rather than merely an application crash that could potentially result in any data corruption.
Two academic papers from the
When provided with a corrupt database, such as one produced by [[fuzzing]], LMDB may crash. LMDB's author considers the case unlikely to be concerning but has produced a partial fix in a separate branch.<ref name=fuzz>{{cite web |last1=Debroux |first1=Lionel |title=oss-security - Fun with DBM-type databases... |url=https://www.openwall.com/lists/oss-security/2018/06/17/1 |website=openwall.com |date=16 Jun 2018}}</ref>
== Open source license ==
In June 2013, [[Oracle Corporation|Oracle]] changed the license of [[Berkeley DB]] (a related project) from the [[Sleepycat license]] to the [[Affero General Public License]],<ref>{{cite web | url=https://oss.oracle.com/pipermail/bdb/2013-June/000056.html|title=Berkeley DB Release Announcement|date=11 June 2013 |publisher=[[Oracle Corporation]] |quote=Starting with the 6.0 / 12c releases, all Berkeley DB products are licensed under the GNU AFFERO GENERAL PUBLIC LICENSE (AGPL), version 3. This license is published by the Free Software Foundation (FSF) (1) and approved by the Open Source Initiative (2).
The Berkeley DB license issue has caused major Linux distributions such as [[Debian]] to completely phase out their use of Berkeley DB, with a preference for LMDB.<ref>{{cite mailing list | url=https://lists.debian.org/debian-devel/2014/06/msg00338.html | title=New project goal: Get rid of Berkeley DB (post jessie) | mailing-list=debian-devel | date=June 19, 2014 | author=Ondřej Surý |publisher=[[Debian]]}}</ref>
== API and uses ==
There are wrappers for several programming languages, such as C++,<ref>[https://github.com/bendiken/lmdbxx LMDB C++11 wrapper], 2015-04</ref><ref>[https://code.google.com/p/libglim/source/browse/trunk/mdb.hpp LMDB C++ wrapper], 2012-11.</ref> Java,<ref>[https://github.com/lmdbjava/lmdbjava LmdbJava], 2019-04</ref> Python,<ref>[https://github.com/dw/py-lmdb/ LMDB Python wrapper], 2013-02</ref><ref>[http://lmdb.readthedocs.org py-lmdb]. Retrieved on 2014-10-20.</ref> Lua,<ref>[https://github.com/shmul/lightningdbm LMDB Lua wrapper], 2013-04.</ref> Rust,<ref>[https://github.com/meilisearch/heed typed LMDB Rust wrapper], 2023-01</ref><ref>[https://github.com/mozilla/rkv high-level Rust wrapper], 2022-12</ref> Go,<ref>[https://github.com/bmatsuo/lmdb-go LMDB Go wrapper], 2013-03</ref> Ruby,<ref>[https://github.com/minad/lmdb LMDB Ruby wrapper], 2013-02</ref> Objective C,<ref>[https://github.com/rbartolome/LMDBKit LMDB Objective-C wrapper], 2013-04</ref> Objective CAML,<ref>[https://github.com/Drup/ocaml-lmdb OCaml lmdb bindings], 2025-08</ref> Javascript,<ref>[https://github.com/Venemo/node-lmdb LMDB Node.js wrapper], 2013-05</ref> C#,<ref>[https://github.com/ilyalukyanov/Lightning.NET LMDB .Net wrapper], 2013-06</ref> Perl,<ref>[http://search.cpan.org/dist/LMDB_File/ LMDB Perl wrapper], 2013-08</ref> PHP,<ref>[https://github.com/mpremus/lmdb-php LMDB PHP wrapper], 2015-04</ref> Tcl<ref>[https://core.tcl.tk/jenglish/gutter/packages/lmdb.html tcl-lmdb], 2015-11</ref> and Common Lisp.<ref>[http://eudoxia.me/article/lmdb-from-common-lisp Using LMDB from Common Lisp], 2016-04</ref> A complete list of wrappers may be found on the main web site.<ref>{{cite web|url=https://www.symas.com/symas-lmdb-tech-info|title=Symas LMDB Tech Info}}</ref>
Howard Chu ported [[SQLite]] 3.7.7.1 to use LMDB instead of its original [[B-tree]] code, calling the
An up-to-date list of applications using LMDB is maintained on the main web site.<ref>{{cite web|url=https://www.symas.com/symas-lmdb-tech-info|title=List of projects using LMDB}}</ref>
Line 77 ⟶ 74:
Many popular [[free software]] projects distribute or include support for LMDB, often as the primary or sole storage mechanism.
* The [[Debian]],<ref>[https://packages.debian.org/liblmdb0 liblmdb0 in Debian]. Retrieved 2014-10-20.</ref> [[Ubuntu (operating system)|Ubuntu]],<ref>{{cite web|url=https://packages.ubuntu.com/lmdb-utils|title=Ubuntu – Package Search Results -- lmdb-utils|first=Rhonda
* [[OpenLDAP]] for which LMDB was originally developed via {{mono|back-mdb}}.<ref>[http://www.openldap.org/doc/admin24/backends.html#LMDB OpenLDAP back-mdb]. Retrieved 2014-10-20</ref>
* [[Postfix (software)|Postfix]] via the {{mono|lmdb_table}} adapter.<ref>[http://www.postfix.org/lmdb_table.5.html Postfix lmdb_table(5)]. Retrieved 2014-10-20</ref>
Line 85 ⟶ 82:
* [[Knot DNS]] a high performance DNS server.
* [[Monero (cryptocurrency)|Monero]] an open source cryptocurrency created in April 2014 that focuses on privacy, decentralisation and scalability.
* [[Enduro/X]] middleware uses LMDB for optional XATMI Microservices (SOA) cache.
* [[
* [[Nano (cryptocurrency)|Nano]] a peer-to-peer, open source cryptocurrency created in 2015 that prioritizes fast and fee-less transactions.
*
* LMDB-IndexedDB is a JavaScript wrapper around [[IndexedDB]] to provide support for LMDB in web browsers.<ref>{{cite web|url=https://github.com/anywhichway/lmdb-indexeddb|title=LMDB-IndexedDB on GitHub|website=[[GitHub]] |access-date=2 Apr 2023}}</ref>
== Technical reviews of LMDB ==
LMDB makes novel use of well-known computer science techniques such as [[copy-on-write]] semantics and [[B+ tree]]s to provide atomicity and reliability guarantees as well as performance that can be hard to accept, given the library's relative simplicity and that no other similar [[key-value store]] database offers the same guarantees or overall performance, even though the authors ''explicitly state'' in presentations that LMDB is read-optimised not write-optimised.
The presentation did spark other database developers to dissect the code in-depth to understand how and why it works. Reviews run from brief <ref>{{cite web | url=http://kellabyte.com/2013/07/09/lightning-memory-mapped-database/|title=Lightning Memory-Mapped Database|archive-url=https://web.archive.org/web/20160314133119/http://kellabyte.com/2013/07/09/lightning-memory-mapped-database/|archive-date=
Multiple other reviews cover LMDB<ref>{{cite web|url=https://mozilla.github.io/firefox-browser-architecture/text/0015-rkv.html|title=Design Review: Key-Value Storage|website=mozilla.github.io|quote=We propose the standardization of a simple key-value storage capability, based on LMDB, that is fast, compact, multi-process-capable, and equally usable from JS, Java, Rust, Swift, and C++.}}</ref>
▲The presentation did spark other database developers to dissect the code in-depth to understand how and why it works. Reviews run from brief <ref>{{cite web | url=http://kellabyte.com/2013/07/09/lightning-memory-mapped-database/|title=Lightning Memory-Mapped Database|archive-url=https://web.archive.org/web/20160314133119/http://kellabyte.com/2013/07/09/lightning-memory-mapped-database/|archive-date=9 Aug 2013}}</ref> to in-depth. Database developer Oren Eini wrote a 12-part series of articles on his analysis of LMDB, beginning July 9, 2013. The conclusion was in the lines of "impressive codebase ... dearly needs some love", mainly because of too long methods and code duplication.<ref>{{cite web | url=http://ayende.com/blog/162754/reviewing-lightning-memory-mapped-database-library-partial|title=Reviewing Lightning memory-mapped database library: Partial}}</ref> This review, conducted by a .NET developer with no former experience of [[C (programming language)|C]], concluded on August 22, 2013 with "beyond my issues with the code, the implementation is really quite brilliant. The way LMDB manages to pack so much functionality by not doing things is quite impressive... I learned quite a lot from the project, and it has been frustrating, annoying and fascinating experience".<ref>{{cite web | url=http://ayende.com/blog/162917/some-final-notes-about-lmdb-review|title=Some final notes about LMDB review}}</ref>
==See also==
▲Multiple other reviews cover LMDB<ref>{{cite web|url=https://mozilla.github.io/firefox-browser-architecture/text/0015-rkv.html|title=Design Review: Key-Value Storage|website=mozilla.github.io|quote=We propose the standardization of a simple key-value storage capability, based on LMDB, that is fast, compact, multi-process-capable, and equally usable from JS, Java, Rust, Swift, and C++.}}</ref> <ref>{{cite web|url=http://sampathherga.in/lmdb/|title=LMDB|publisher=Sampath Herga|access-date=2013-08-30|archive-url=https://web.archive.org/web/20130829025217/http://sampathherga.in/lmdb/|archive-date=2013-08-29|url-status=dead}}</ref> in various languages including Chinese.<ref>{{cite web | url=http://jianshu.io/p/yzFf8j|title=lmdb简介 - 简书}}</ref><ref>{{cite web|url=http://matao.writings.io/articles/1-lmdb|title=lmdb|accessdate=8 May 2017|archive-url=https://web.archive.org/web/20160305120551/http://matao.writings.io/articles/1-lmdb|archive-date=5 March 2016|url-status=dead}}</ref>
* [[Ordered Key-Value Store]]
* [[libmdbx]] (aka MDBX) is a deeply revised and extended descendant of LMDB, which according to the developers, "is superior to legendary LMDB in terms of features and reliability, not inferior in performance". It is noteworthy MDBX is used in [[Ethereum]] (Erigon, Reth), in products of [[StarkWare Industries]], [[Positive Technologies]], as well as in many other software projects.
== References ==
| |||