Lightning Memory-Mapped Database: Difference between revisions

Revision as of 23:04, 22 September 2024

Revision as of 18:46, 19 October 2024

Line 55:

LMDB was designed to resist data loss in the face of system and application crashes. Its [[copy-on-write]] approach never overwrites currently-in-use data. Avoiding overwrites means the structure on disk/storage is always valid, so application or system crashes can never leave the database in a corrupted state. In its default mode, at worst, a crash can lose data from the last not-yet-committed write transaction. Even with all asynchronous modes enabled, it is only an OS catastrophic failure or hardware power-loss<ref>{{cite web | url=https://bugs.openldap.org/show_bug.cgi?id=7668|title=LMDB Corruption detection}}</ref> event rather than merely an application crash that could potentially result in any data corruption.

Two academic papers from the USENIX OSDI Symposium<ref>{{cite web | url=https://www.usenix.org/conference/osdi14|title=OSDI 2014|date=2013-02-08}}</ref> covered failure modes of DB engines (including LMDB) under a sudden power loss or system crash.<ref>{{cite book | url=https://www.usenix.org/conference/osdi14/technical-sessions/presentation/pillai|title=OSDI 2014, All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications|pages=433–448|isbn=9781931971164|year=2014|last1=Langston|first1=Mark C.|last2=Skelly|first2=Hal}}</ref><ref>{{cite book | url=https://www.usenix.org/conference/osdi14/technical-sessions/presentation/zheng_mai|title=OSDI 2014, Torturing Databases for Fun and Profit|pages=449–464|isbn=9781931971164|year=2014|last1=Langston|first1=Mark C.|last2=Skelly|first2=Hal}}</ref> The paper from Pillai et al., did not find any failure in LMDB that would occur in the real-world file systems considered; the single failure identified by the study in LMDB only relates to hypothetical file systems.<ref>{{cite web | url=http://www.openldap.org/lists/openldap-devel/201410/msg00004.html|title=Archive of discussion regarding the Usenix 2014 pillai paper}}</ref> The Mai Zheng et al. paper claims to point out failures in LMDB, but the conclusion depends on whether fsync or fdatasync is utilised. Using fsync ameliorates the problem. The selection of fsync orover fdatasync is a compile-time switch that is not the default behavior in current Linux builds of LMDB but is the default on macOS, *BSD, Android, and Windows. Default Linux builds of LMDB are, therefore, the only ones vulnerable to the problem discovered by the zhengmai researchers however, LMDB may simply be rebuilt by Linux users to utilise fsync instead.<ref>{{cite web | url=http://www.openldap.org/lists/openldap-devel/201410/msg00008.html|title=LMDB Crash consistency discussion}}</ref>

When provided with a corrupt database, such as one produced by [[fuzzing]], LMDB may crash. LMDB's author considers the case unlikely to be concerning but has produced a partial fix in a separate branch.<ref name=fuzz>{{cite web |last1=Debroux |first1=Lionel |title=oss-security - Fun with DBM-type databases... |url=https://www.openwall.com/lists/oss-security/2018/06/17/1 |website=openwall.com |date=16 Jun 2018}}</ref>