Lightning Memory-Mapped Database: Difference between revisions

Revision as of 04:56, 6 July 2021

Revision as of 00:58, 9 August 2021

Line 56:

LMDB was designed from the start to resist data loss in the face of system and application crashes. Its [[copy-on-write]] approach never overwrites currently-in-use data. Avoiding overwrites means the structure on disk/storage is always valid, so application or system crashes can never leave the database in a corrupted state. In its default mode, at worst a crash can lose data from the last not-yet-committed write transaction. Even with all asynchronous modes enabled, it is only an OS catastrophic failure or hardware power-loss<ref>{{cite web | url=https://bugs.openldap.org/show_bug.cgi?id=7668|title=LMDB Corruption detection}}</ref> event rather than merely an application crash that could potentially result in any data corruption.

Two academic papers from the [[USENIX Symposium on Operating Systems Design and Implementation|USENIX OSDI Symposium]]<ref>{{cite web | url=https://www.usenix.org/conference/osdi14|title=OSDI 2014|date=2013-02-08}}</ref> covered failure modes of DB engines (including LMDB) under a sudden power loss or system crash.<ref>{{cite book | url=https://www.usenix.org/conference/osdi14/technical-sessions/presentation/pillai|title=OSDI 2014, All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications|pages=433–448|isbn=9781931971164|year=2014|last1=Langston|first1=Mark C.|last2=Skelly|first2=Hal}}</ref><ref>{{cite book | url=https://www.usenix.org/conference/osdi14/technical-sessions/presentation/zheng_mai|title=OSDI 2014, Torturing Databases for Fun and Profit|pages=449–464|isbn=9781931971164|year=2014|last1=Langston|first1=Mark C.|last2=Skelly|first2=Hal}}</ref> The paper from Pillai et al., did not find any failure in LMDB that would occur in the real-world file systems considered; the single failure identified by the study in LMDB only relates to hypothetical file systems.<ref>{{cite web | url=http://www.openldap.org/lists/openldap-devel/201410/msg00004.html|title=Archive of discussion regarding the Usenix 2014 pillai paper}}</ref> The Mai Zheng et al. paper claims to point out failures in LMDB, but the conclusion depends on whether fsync or fdatasync is utilised. Using fsync ameliorates the problem. Selection of fsync or fdatasync is a compile-time switch which is not the default behavior in current GNU/Linux builds of LMDB, but is the default on macOS, *BSD, Android, and Windows. Default GNU/Linux builds of LMDB are therefore the only ones vulnerable to the problem discovered by the zhengmai researchers however LMDB may simply be rebuilt by GNU/Linux users to utilise fsync instead.<ref>{{cite web | url=http://www.openldap.org/lists/openldap-devel/201410/msg00008.html|title=LMDB Crash consistency discussion}}</ref>

When provided with a corrupt database, such as one produced by [[fuzzing]], LMDB may crash. LMDB's author considers the case unlikely to be concerning, but has nevertheless produced a partial fix in a separate branch.<ref name=fuzz>{{cite web |last1=Debroux |first1=Lionel |title=oss-security - Fun with DBM-type databases... |url=https://www.openwall.com/lists/oss-security/2018/06/17/1 |website=openwall.com |date=16 Jun 2018}}</ref>

Line 63:

In June 2013, [[Oracle Corporation|Oracle]] changed the license of [[Berkeley DB]] (a related project) from the [[Sleepycat license]] to the [[Affero General Public License]],<ref>{{cite web | url=https://oss.oracle.com/pipermail/bdb/2013-June/000056.html|title=Berkeley DB Release Announcement|publisher=[[Oracle Corporation]] |quote=Starting with the 6.0 / 12c releases, all Berkeley DB products are licensed under the GNU AFFERO GENERAL PUBLIC LICENSE (AGPL), version 3. This license is published by the Free Software Foundation (FSF) (1) and approved by the Open Source Initiative (2). Please review the terms of the license to ensure compliance before upgrading to the 12c release. Previous releases of Berkeley DB software will continue to be distributed under the Sleepycat license.}}</ref> thus restricting its use in a wide variety of applications. This caused the [[Debian|Debian project]] to exclude the library from 6.0 onwards. It was also criticized that this license is not friendly to commercial redistributors. The discussion was sparked over whether the same licensing change could happen to LMDB. Author Howard Chu made clear that LMDB is part of the OpenLDAP project, which had its BSD style license before he joined, and it will stay like it. No copyright is transferred to anybody by checking in, which would make a similar move like Oracle's impossible.<ref>{{cite mailing list | url=http://lists.debian.org/debian-devel/2013/07/msg00031.html |title=Berkeley DB 6.0 license change to AGPLv3 |date=July 2, 2013 |mailing-list=debian-devel |publisher=[[Debian]] |author=Ondřej Surý}}</ref><ref>{{cite news |url=http://www.infoworld.com/d/open-source-software/oracle-switches-berkeley-db-license-222097 |title=Oracle switches Berkeley DB license |author=Simon Phipps |date=July 5, 2013 |publisher=[[InfoWorld]]}}</ref><ref>{{cite web | url=http://developers.slashdot.org/story/13/07/05/1647215/oracle-quietly-switches-berkeleydb-to-agpl|title=Oracle Quietly Switches BerkeleyDB to AGPL|publisher=[[Slashdot]]}}</ref><ref>{{cite web |url=http://programmers-in-ua.blogspot.com/2013/07/oracle-berkeley-db.html |title=Oracle меняет лицензию Berkeley DB |publisher=[[Blogspot]] |website=Programmers in Ukraine |date=July 22, 2013 |language=ru |trans-title=Oracle Berkeley DB license changes}}</ref><ref>{{cite web | url=http://www.lemondeinformatique.fr/actualites/lire-oracle-passe-berkeley-db-sous-licence-gnu-agpl-54287.html|title=Oracle passe Berkeley DB sous licence GNU AGPL |language=fr |trans-title=Oracle Berkeley DB passes under GNU AGPL |date=July 8, 2013 |author= Jean Elyan |publisher=Le Monde Informatique}}</ref><ref>{{cite web | url=http://www.abclinuxu.cz/zpravicky/berkeley-db-6.0-vydana-pod-licenci-agplv3 |title=Berkeley DB 6.0 vydána pod licencí AGPLv3 |language=cs |trans-title=Berkeley DB 6.0 is released under the GPLv3 license |date=July 2, 2013 |author=Ondřej Surý |publisher=Abclinuxu}}</ref><ref>{{cite web | url=https://lwn.net/Articles/557820/ |title=Debian, Berkeley DB, and AGPLv3 |author=Nathan Willis |date=July 10, 2013 |publisher=[[LWN.net]]}}</ref><!-- Nathan Willis copied several posts from debian-devel to LWN.net, including url=https://lwn.net/Articles/558155/ and url=https://lwn.net/Articles/558154/; let's cite the original--><ref>{{cite mailing list | url=https://lists.debian.org/debian-devel/2013/07/msg00047.html |title=Berkeley DB 6.0 license change to AGPLv3 |date=July 2, 2013 |mailing-list=debian-devel |publisher=[[Debian]] |author=Dan Shearer}}</ref><ref>{{cite mailing list | url=https://lists.debian.org/debian-devel/2013/07/msg00067.html |title=Berkeley DB 6.0 license change to AGPLv3 |date=July 2, 2013 |mailing-list=debian-devel |publisher=[[Debian]] |author=Howard Chu}}</ref>

The Berkeley DB license issue has caused major GNU/Linux distributions such as [[Debian]] to completely phase out their use of Berkeley DB, with a preference for LMDB.<ref>{{cite mailing list | url=https://lists.debian.org/debian-devel/2014/06/msg00338.html | title=New project goal: Get rid of Berkeley DB (post jessie) | mailing-list=debian-devel | date=June 19, 2014 | author=Ondřej Surý |publisher=[[Debian]]}}</ref>

== API and uses ==