Lightning Memory-Mapped Database: Difference between revisions

Revision as of 17:20, 14 January 2025

Latest revision as of 21:22, 27 August 2025

(4 intermediate revisions by 4 users not shown)

Line 16:

}}

{{Portal|Free and open-source software}}

'''Lightning Memory-Mapped Database''' ('''LMDB''') is an embedded transactional [[database]] in the form of a [[key-value store]]. LMDB is written in [[C (programming language)|C]] with [[#API and uses|API bindings]] for several [[programming language]]s. LMDB stores arbitrary key/data pairs as byte arrays, has a range-based search capability, supports multiple data items for a single key and has a special mode for appending records (MDB_APPEND) without checking for consistency.<ref name="auto">[http://www.lmdb.tech/doc/ LMDB Reference Guide] Retrieved on 2023-03-21</ref> LMDB is not a [[relational database]], it is strictly a key-value store like [[Berkeley DB]] and [[DBM (computing)|DBM]].

LMDB may also be used [[#Concurrency|concurrently]] in a multi-threaded or multi-processing environment, with read performance scaling linearly by design. LMDB databases may have only one writer at a time, however unlike many similar key-value databases, write transactions do ''not'' block readers, nor do readers block writers. LMDB is also unusual in that multiple applications on the same system may simultaneously open and use the same LMDB store, as a means to scale up performance. Also, LMDB does not require a transaction log (thereby increasing write performance by not needing to write data twice) because it maintains data integrity inherently by design.

Line 65:

== API and uses ==

There are wrappers for several programming languages, such as C++,<ref>[https://github.com/bendiken/lmdbxx LMDB C++11 wrapper], 2015-04</ref><ref>[https://code.google.com/p/libglim/source/browse/trunk/mdb.hpp LMDB C++ wrapper], 2012-11.</ref> Java,<ref>[https://github.com/lmdbjava/lmdbjava LmdbJava], 2019-04</ref> Python,<ref>[https://github.com/dw/py-lmdb/ LMDB Python wrapper], 2013-02</ref><ref>[http://lmdb.readthedocs.org py-lmdb]. Retrieved on 2014-10-20.</ref> Lua,<ref>[https://github.com/shmul/lightningdbm LMDB Lua wrapper], 2013-04.</ref> Rust,<ref>[https://github.com/meilisearch/heed typed LMDB Rust wrapper], 2023-01</ref><ref>[https://github.com/mozilla/rkv high-level Rust wrapper], 2022-12</ref> Go,<ref>[https://github.com/bmatsuo/lmdb-go LMDB Go wrapper], 2013-03</ref> Ruby,<ref>[https://github.com/minad/lmdb LMDB Ruby wrapper], 2013-02</ref> Objective C,<ref>[https://github.com/rbartolome/LMDBKit LMDB Objective-C wrapper], 2013-04</ref> Objective CAML,<ref>[https://github.com/Drup/ocaml-lmdb OCaml lmdb bindings], 2025-08</ref> Javascript,<ref>[https://github.com/Venemo/node-lmdb LMDB Node.js wrapper], 2013-05</ref> C#,<ref>[https://github.com/ilyalukyanov/Lightning.NET LMDB .Net wrapper], 2013-06</ref> Perl,<ref>[http://search.cpan.org/dist/LMDB_File/ LMDB Perl wrapper], 2013-08</ref> PHP,<ref>[https://github.com/mpremus/lmdb-php LMDB PHP wrapper], 2015-04</ref> Tcl<ref>[https://core.tcl.tk/jenglish/gutter/packages/lmdb.html tcl-lmdb], 2015-11</ref> and Common Lisp.<ref>[http://eudoxia.me/article/lmdb-from-common-lisp Using LMDB from Common Lisp], 2016-04</ref> A complete list of wrappers may be found on the main web site.<ref>{{cite web|url=https://www.symas.com/symas-lmdb-tech-info|title=Symas LMDB Tech Info}}</ref>

Howard Chu ported [[SQLite]] 3.7.7.1 to use LMDB instead of its original [[B-tree]] code, calling the result SQLightning.<ref>{{cite web|url=http://gitorious.org/mdb/sqlightning|title=gitorious.org Git - mdb:sqlightning.git/summary|website=gitorious.org|accessdate=8 May 2017|archive-url=https://web.archive.org/web/20130809145553/http://gitorious.org/mdb/sqlightning|archive-date=9 August 2013|url-status=live}}</ref> One cited insert test of 1000 records was 20 times faster (than the original SQLite with its B-Tree implementation).<ref>[http://pastebin.com/B5SfEieL SQLightning tests].</ref> LMDB is available as a backing store for other open source projects including Cyrus SASL,<ref>{{cite web|url=http://cyrusimap.web.cmu.edu/|title=Cyrus IMAP — Cyrus IMAP 3.0.1 (stable) documentation|website=cyrusimap.web.cmu.edu|accessdate=8 May 2017|url-status=dead|archive-url=https://web.archive.org/web/20170430023937/http://cyrusimap.web.cmu.edu/|archive-date=30 April 2017}}</ref> Heimdal Kerberos,<ref>{{cite web|url=http://h5l.org/|title=Heimdal|website=h5l.org|accessdate=8 May 2017}}</ref> and OpenDKIM.<ref>{{cite web|url=http://www.opendkim.org/|title=OpenDKIM|website=www.opendkim.org|accessdate=8 May 2017}}</ref> It is also available in some other NoSQL projects like MemcacheDB <ref>{{cite web|url=https://gitorious.org/mdb/memcachedb|title=gitorious.org Git - mdb:memcachedb.git/summary|website=gitorious.org|accessdate=8 May 2017}}</ref> and Mapkeeper.<ref>{{cite web|url=https://github.com/m1ch1/mapkeeper/|title=GitHub - m1ch1/mapkeeper: Thrift based key-value store with various storage backends, including MySQL, Berkeley DB, and LevelDB.|website=github.com|url-status=dead|archive-url=https://web.archive.org/web/20160209024233/https://github.com/m1ch1/mapkeeper/|archive-date=9 February 2016}}</ref> LMDB was used to make the in-memory store [[Redis]] persist data on disk. The existing back-end in [[Redis]] showed pathological behaviour in rare cases, and a replacement was sought. The baroque API of LMDB was criticized though, forcing a lot of coding to get simple things done. However, its performance and reliability during testing was considerably better than the alternative back-end stores that were tried.<ref>{{cite web|url=http://www.anchor.com.au/blog/2013/05/second-strike-with-lightning/|title=Second Strike With Lightning|publisher=Anchor|date=2013-05-09}}</ref>

An independent third-party software developer utilised the [[Python (programming language)|Python]] bindings to LMDB<ref>{{cite web | url=http://lmdb.readthedocs.org|title=Python bindings to LMDB}}</ref> in a high-performance environment and published, on the technology news site [[Slashdot]], how the system managed to successfully sustain 200,000 simultaneous read, write and delete operations per second (a total of 600,000 database operations per second).<ref>{{cite web | url=http://developers.slashdot.org/story/14/10/17/1547222/python-lmdb-in-a-high-performance-environment|title=Python-LMDB in a high-performance environment on Slashdot|date=17 October 2014 }}</ref><ref>{{cite web | url=http://lkcl.net/reports/python.lmdb.html|title=Open letter to Howard Chu and David Wilson regarding Python-LMDB}}</ref>

An up-to-date list of applications using LMDB is maintained on the main web site.<ref>{{cite web|url=https://www.symas.com/symas-lmdb-tech-info|title=List of projects using LMDB}}</ref>

Line 96 ⟶ 94:

Multiple other reviews cover LMDB<ref>{{cite web|url=https://mozilla.github.io/firefox-browser-architecture/text/0015-rkv.html|title=Design Review: Key-Value Storage|website=mozilla.github.io|quote=We propose the standardization of a simple key-value storage capability, based on LMDB, that is fast, compact, multi-process-capable, and equally usable from JS, Java, Rust, Swift, and C++.}}</ref><ref>{{cite web|url=http://sampathherga.in/lmdb/|title=LMDB|publisher=Sampath Herga|access-date=2013-08-30|archive-url=https://web.archive.org/web/20130829025217/http://sampathherga.in/lmdb/|archive-date=2013-08-29|url-status=dead}}</ref> in various languages including Chinese.<ref>{{cite web | url=http://jianshu.io/p/yzFf8j|title=lmdb简介 - 简书}}</ref>

==See also==

* [[Ordered Key-Value Store]]

* [[libmdbx]] (aka MDBX) is a deeply revised and extended descendant of LMDB, which according to the developers, "is superior to legendary LMDB in terms of features and reliability, not inferior in performance". It is noteworthy MDBX is used in [[Ethereum]] (Erigon, Reth), in products of [[StarkWare Industries]], [[Positive Technologies]], as well as in many other software projects.

== References ==