Lightning Memory-Mapped Database: Difference between revisions

Revision as of 20:45, 22 March 2023

Revision as of 05:26, 28 March 2023

Line 43:

=== Concurrency ===

LMDB employs [[multiversion concurrency control]] (MVCC) and allows multiple threads within multiple processes to coordinate simultaneous access to a database. Readers scale linearly by design.<ref>[http://www.lmdb.tech/bench/inmem/scaling.html scaling benchmarks for LMDB]</ref><ref>[http://www.lmdb.tech/bench/inmem/scale2/ in-memory benchmark scaling for LMDB]</ref>. While write transactions are globally serialized via a [[mutex]], read-only transactions operate in parallel, including in the presence of a write transaction. They are entirely [[wait free]] except for the first read-only transaction on a thread. Each thread reading from a database gains ownership of an element in a shared memory array, which it may update to indicate when it is within a transaction. Writers scan the array to determine the oldest database version the transaction must preserve without requiring direct synchronization with active readers.

== Performance ==

Line 68:

There are wrappers for several programming languages, such as C++,<ref>[https://github.com/bendiken/lmdbxx LMDB C++11 wrapper], 2015-04</ref><ref>[https://code.google.com/p/libglim/source/browse/trunk/mdb.hpp LMDB C++ wrapper], 2012-11.</ref> Java,<ref>[https://github.com/lmdbjava/lmdbjava LmdbJava], 2019-04</ref> Python,<ref>[https://github.com/dw/py-lmdb/ LMDB Python wrapper], 2013-02</ref><ref>[http://lmdb.readthedocs.org py-lmdb]. Retrieved on 2014-10-20.</ref> Lua,<ref>[https://github.com/shmul/lightningdbm LMDB Lua wrapper], 2013-04.</ref> Rust,<ref>[https://github.com/meilisearch/heed typed LMDB Rust wrapper], 2023-01</ref><ref>[https://github.com/mozilla/rkv high-level Rust wrapper], 2022-12</ref> Go,<ref>[https://github.com/bmatsuo/lmdb-go LMDB Go wrapper], 2013-03</ref> Ruby,<ref>[https://github.com/minad/lmdb LMDB Ruby wrapper], 2013-02</ref> Objective C,<ref>[https://github.com/rbartolome/LMDBKit LMDB Objective-C wrapper], 2013-04</ref> Javascript,<ref>[https://github.com/Venemo/node-lmdb LMDB Node.js wrapper], 2013-05</ref> C#,<ref>[https://github.com/ilyalukyanov/Lightning.NET LMDB .Net wrapper], 2013-06</ref> Perl,<ref>[http://search.cpan.org/dist/LMDB_File/ LMDB Perl wrapper], 2013-08</ref> PHP,<ref>[https://github.com/mpremus/lmdb-php LMDB PHP wrapper], 2015-04</ref> Tcl<ref>[https://core.tcl.tk/jenglish/gutter/packages/lmdb.html tcl-lmdb], 2015-11</ref> and Common Lisp.<ref>[http://eudoxia.me/article/lmdb-from-common-lisp Using LMDB from Common Lisp], 2016-04</ref> A complete list of wrappers may be found on the main web site.<ref>{{cite web|url=https://www.symas.com/symas-lmdb-tech-info|title=Symas LMDB Tech Info}}</ref>

Howard Chu ported [[SQLite]] 3.7.7.1 to use LMDB instead of its original [[B-tree]] code, calling the end result SQLightning.<ref>{{cite web|url=http://gitorious.org/mdb/sqlightning|title=gitorious.org Git - mdb:<nowiki />sqlightning.git/summary|website=gitorious.org|accessdate=8 May 2017|archive-url=https://web.archive.org/web/20130809145553/http://gitorious.org/mdb/sqlightning|archive-date=9 August 2013|url-status=live}}</ref> One cited insert test of 1000 records was 20 times faster (than the original SQLite with its B-Tree implementation).<ref>[http://pastebin.com/B5SfEieL SQLightning tests].</ref> LMDB is available as a backing store for other open source projects including Cyrus SASL,<ref>{{cite web|url=http://cyrusimap.web.cmu.edu/|title=Cyrus IMAP — Cyrus IMAP 3.0.1 (stable) documentation|website=cyrusimap.web.cmu.edu|accessdate=8 May 2017|url-status=dead|archive-url=https://web.archive.org/web/20170430023937/http://cyrusimap.web.cmu.edu/|archive-date=30 April 2017}}</ref> Heimdal Kerberos,<ref>{{cite web|url=http://h5l.org/|title=Heimdal|website=h5l.org|accessdate=8 May 2017}}</ref> and OpenDKIM.<ref>{{cite web|url=http://www.opendkim.org/|title=OpenDKIM|website=www.opendkim.org|accessdate=8 May 2017}}</ref> It is also available in some other NoSQL projects like MemcacheDB <ref>{{cite web|url=https://gitorious.org/mdb/memcachedb|title=gitorious.org Git - mdb:<nowiki />memcachedb.git/summary|website=gitorious.org|accessdate=8 May 2017}}</ref> and Mapkeeper.<ref>{{cite web|url=https://github.com/m1ch1/mapkeeper/|title=GitHub - m1ch1/mapkeeper: Thrift based key-value store with various storage backends, including MySQL, Berkeley DB, and LevelDB.|website=github.com|url-status=dead|archive-url=https://web.archive.org/web/20160209024233/https://github.com/m1ch1/mapkeeper/|archive-date=16 Feb 2016}}</ref> LMDB was used to make the in-memory store [[Redis]] persist data on disk. The existing back-end in [[Redis]] showed pathological behaviour in rare cases, and a replacement was sought. The baroque API of LMDB was criticized though, forcing a lot of coding to get simple things done. However, its performance and reliability during testing was considerably better than the alternative back-end stores that were tried.<ref>{{cite web|url=http://www.anchor.com.au/blog/2013/05/second-strike-with-lightning/|title=Second Strike With Lightning|publisher=Anchor|date=2013-05-09}}</ref>

An independent third-party software developer utilised the [[Python (programming language)|Python]] bindings to LMDB<ref>{{cite web | url=http://lmdb.readthedocs.org|title=Python bindings to LMDB}}</ref> in a high-performance environment and published, on the technology news site [[Slashdot]], how the system managed to successfully sustain 200,000 simultaneous read, write and delete operations per second (a total of 600,000 database operations per second).<ref>{{cite web | url=http://developers.slashdot.org/story/14/10/17/1547222/python-lmdb-in-a-high-performance-environment|title=Python-LMDB in a high-performance environment on Slashdot}}</ref><ref>{{cite web | url=http://lkcl.net/reports/python.lmdb.html|title=Open letter to Howard Chu and David Wilson regarding Python-LMDB}}</ref>

Line 93:

LMDB makes novel use of well-known computer science techniques such as [[copy-on-write]] semantics and [[B+ tree]]s to provide atomicity and reliability guarantees as well as performance that can be hard to accept, given the library's relative simplicity and that no other similar [[key-value store]] database offers the same guarantees or overall performance, even though the authors ''explicitly state'' in presentations that LMDB is read-optimised not write-optimised. Additionally, as LMDB was primarily developed for use in [[OpenLDAP]], its developers are focused mainly on the development and maintenance of OpenLDAP, not on LMDB per se. The developers limited time spent presenting the first benchmark results was therefore criticized as not stating limitations and for giving a "silver bullet impression" not adequate to address an engineers attitude<ref>{{cite web | url=http://banksco.de/p/lmdb-the-leveldb-killer.html|title=LMDB: The Leveldb Killer?}}</ref> ''(it has to be pointed out that the concerns raised however were later adequately addressed to the reviewer's satisfaction by the key developer behind LMDB.<ref>{{cite web | url=https://symas.com/is-lmdb-a-leveldb-killer/|title=Response to LMDB review|archive-url=https://web.archive.org/web/20201111195059/https://symas.com/is-lmdb-a-leveldb-killer/|url-status=dead |website=symas.com|archive-date=29 Aug 2013}}</ref>)''

The presentation did spark other database developers to dissect the code in-depth to understand how and why it works. Reviews run from brief <ref>{{cite web | url=http://kellabyte.com/2013/07/09/lightning-memory-mapped-database/|title=Lightning Memory-Mapped Database|archive-url=https://web.archive.org/web/20160314133119/http://kellabyte.com/2013/07/09/lightning-memory-mapped-database/|archive-date=9 Aug 2013}}</ref> to in-depth. Database developer Oren Eini wrote a 12-part series of articles on his analysis of LMDB, beginning July 9, 2013. The conclusion was in the lines of "impressive codebase ... dearly needs some love", mainly because of too long methods and code duplication.<ref>{{cite web | url=http://ayende.com/blog/162754/reviewing-lightning-memory-mapped-database-library-partial|title=Reviewing Lightning memory-mapped database library: Partial}}</ref> This review, conducted by a .NET developer with no former experience of [[C (programming language)|C]], concluded on August 22, 2013 with "beyond my issues with the code, the implementation is really quite brilliant. The way LMDB manages to pack so much functionality by not doing things is quite impressive... I learned quite a lot from the project, and it has been frustrating, annoying and fascinating experience".<ref>{{cite web | url=http://ayende.com/blog/162917/some-final-notes-about-lmdb-review|title=Some final notes about LMDB review}}</ref>.

Multiple other reviews cover LMDB<ref>{{cite web|url=https://mozilla.github.io/firefox-browser-architecture/text/0015-rkv.html|title=Design Review: Key-Value Storage|website=mozilla.github.io|quote=We propose the standardization of a simple key-value storage capability, based on LMDB, that is fast, compact, multi-process-capable, and equally usable from JS, Java, Rust, Swift, and C++.}}</ref> <ref>{{cite web|url=http://sampathherga.in/lmdb/|title=LMDB|publisher=Sampath Herga|access-date=2013-08-30|archive-url=https://web.archive.org/web/20130829025217/http://sampathherga.in/lmdb/|archive-date=2013-08-29|url-status=dead}}</ref> in various languages including Chinese.<ref>{{cite web | url=http://jianshu.io/p/yzFf8j|title=lmdb简介 - 简书}}</ref><ref>{{cite web|url=http://matao.writings.io/articles/1-lmdb|title=lmdb|accessdate=8 May 2017|archive-url=https://web.archive.org/web/20160305120551/http://matao.writings.io/articles/1-lmdb|archive-date=5 March 2016|url-status=dead}}</ref>