Content deleted Content added
→Adoption: new |
m Replaced 1 bare URLs by {{Cite web}}; Replaced "Archived copy" by actual titles |
||
| (27 intermediate revisions by 14 users not shown) | |||
Line 1:
{{Lead too short|date=February 2020}}
{{Use dmy dates|date=January 2020|cs1-dates=y}}
{{Use list-defined references|date=January 2022}}
'''Large-file support''' ('''LFS''') is the term frequently applied to the ability to create files larger than either 2 or 4 [[Gibibyte|GiB]] on [[32-bit]] [[filesystem
==Details==
Traditionally, many operating systems and their underlying [[file system]] implementations used [[32-bit]] [[integer]]s to represent [[computer file|file]] sizes and positions. Consequently, no file could be larger than 2<sup>32</sup> − 1 bytes (4 GiB − 1). In many implementations, the problem was exacerbated by treating the sizes as [[Sign (mathematics)|signed]] numbers, which further lowered the limit to 2<sup>31</sup> − 1 bytes (2 GiB − 1). Files that were too large for 32-bit operating systems to handle came to be known as ''large files''.
While the limit was quite acceptable at a time when [[hard disk]]s were smaller, the general increase in storage capacity combined with increased server and desktop file usage, especially for [[database]] and [[multimedia]] files, led to intense pressure for OS vendors to overcome the limitation.
Line 10 ⟶ 12:
This switch caused deployment issues and required design modifications, the consequences of which can still be seen:
* The change to 64-bit file sizes frequently required incompatible changes to file system layout, which meant that large-file support sometimes necessitated a file system change. For example,
* To support binary compatibility with old [[
* To support writing [[porting|portable]] code that makes use of LFS where possible, [[C standard library]] authors devised mechanisms that, depending on [[C preprocessor|preprocessor]] constants, transparently redefined the functions to the 64-bit large-file aware ones.
* Many old interfaces, especially [[C (programming language)|C]]-based ones, explicitly specified argument types in a way that did not allow straightforward or transparent transition to 64-bit types. For example, the C functions <code>[[fseek]]</code> and <code>ftell</code> operate on file positions of type <code>long int</code>, which is typically 32 bits wide on 32-bit platforms, and cannot be made larger without sacrificing backward compatibility. (This was resolved by introducing new functions <code>fseeko</code> and <code>ftello</code> in [[POSIX]].<ref name="Unix_1996_LFS"/> On Windows machines, under Visual C++, functions <code>_fseeki64</code> and <code>_ftelli64</code> are used.)
==Adoption==
The usage of the large-file API in 32-bit programs had been incomplete for a long time. An analysis did show in 2002 that many base libraries of operating systems were still shipped without large-file support thereby limiting applications using them.<ref
The problem disappeared slowly with
A similar development can be seen in the mobile area. Google required to support 64-bit versions of applications in their app store by August 2019,<ref
Except for [[embedded systems]] with their special programs, the consideration of varying large-file support becomes obsolete in program code after 2020.
== Related problems ==
The [[year 2038 problem]] is well known for another case where a 32-bit "long" on 32-bit platforms will lead into problems. Just like the large-file limitation it will get obsolete when systems move to 64-bit only. In the meantime a 64-bit timestamp was introduced. In the Win32 API it is visible in functions having a "64" suffix along the earlier "32" suffix. When large-file support was added to the Win32 API it has led to functions having an additional "i64" suffix which sometimes makes for four combinations.(findfirst32, findfirst64, findfirst32i64, findfirst64i32).<ref name="Microsoft_2019_CRT"/> By comparison the UNIX98 API introduces functions with a "64" suffix when "_LARGEFILE64_SOURCE" is used.
Related to the large-file API there is a limitation of block numbers for [[mass storage]] media. With a common size of 512 bytes per [[Block (data storage)|data block]] the barrier resulting from 32-bit numbers did occur later. When [[hard disk drive]]s reached a size of 2 terabyte (around 2010) the [[master boot record]] had to be replaced by the [[GUID Partition Table]] which uses 64-bit for the LBA numbers ([[logical block addressing|logical block address]]). On [[Unix-like]] operating systems it did also require to enlarge the [[inode]] numbers which are used in some functions (stat64, setrlimit64). The [[Linux kernel]] introduced that in 2001 leading to version 2.4 which was picked up by the glibc in that year.<ref name="lfs_2015"/> As the large-file support and large-disk support was introduced at the same time the [[GNU C Library]] exports 64-bit inode structures on 32-bit architectures at the same time when the Unix LFS API is activated in program code.<ref name="Linux_stat.h"/>
When the kernel moved to 64-bit inodes the file system [[ext3]] used them internally in the driver by 2001. However the inode format on the storage media itself was stuck at 32-bit numbers.<ref name="lfs_2015"/> As mass storage devices moved to the [[Advanced Format]] of 4 kilobyte per block the actual limit of that file system format is at 8 or 16 terabyte.<ref name="lfs_2015"/> Handling larger disk partitions requires the usage of a different file system like [[XFS]] which was designed with 64-bit inodes from the start allowing for exabyte files and partitions.<ref name="Rutter_2020_inode"/><ref name="ext4_2019"/> The first 16 terabyte magnetic disk drives were delivered by mid 2019. [[Solid-state drive]] with 32 TiB for data centers were available as early as 2016 with some manufacturers forecasting 100 TiB SSD by 2020.<ref name="Scherer_2016"/>
==See also==
* [[2 GB limit]]
* [[RF64]] – 64-bit support for [[Broadcast Wave Format|BWF WAV]] audio files
* [[Comparison of text editors#Extra features|Comparison of large-file support in text editors]]
* [[FAT32+]]
* [[File size]]
* [[File spanning]]
* [[Long filename support]] (LFN)
* [[Year 2038 problem]]
Line 37 ⟶ 47:
<ref name="Solaris_1996">{{cite web |author=Solaris OS group |date=March 1996 |title=Large Files in Solaris: A White Paper |publisher=[[Sun Microsystems]] |url=http://www.sun.com/software/whitepapers/wp-largefiles/largefiles.pdf |archive-url=https://web.archive.org/web/20070228204423/http://www.sun.com/software/whitepapers/wp-largefiles/largefiles.pdf |archive-date=2007-02-28}}</ref>
<ref name="Unix_1996_LFS">{{cite web |date=1996-08-14 |title=Adding Large File Support to the Single UNIX Specification |publisher=X/Open Base Working Group |url=http://opengroup.org/platform/lfs.html |access-date=2006-09-10}}</ref>
<ref name="
<ref name="Kuhnt_2011_EDR">{{cite web |title=DR-DOS/OpenDOS Enhancement Project |author-first=Udo |author-last=Kuhnt |date=2011-07-21 |url=http://www.drdosprojects.de/ |access-date=2015-04-20 |url-status=dead |archive-url=https://web.archive.org/web/20160604014846/http://www.drdosprojects.de/ |archive-date=2016-06-04}}</ref>
<ref name="Largefile_Distros">{{cite web |url=https://ac-archive.sourceforge.net/largefile/distros.html |title=Distro Makers |date=2002-01-13}}</ref>
<ref name="ZLib_Changelog">{{Cite web| title=ChangeLog file for zlib | url=https://www.zlib.net/ChangeLog.txt | archive-url=https://web.archive.org/web/20031218063837/http://zlib.net:80/ChangeLog.txt | archive-date=2003-12-18}}</ref>
<ref name="Kolokythas_2007">{{cite web |title=Windows Server 2008: Microsofts letztes 32-Bit-Betriebssystem für Server |language=de |author-first=Panagiotis |author-last=Kolokythas |publisher=[[PC Welt]] |date=2007-05-28 |url=https://www.pcwelt.de/news/Windows-Server-2008-Microsofts-letztes-32-Bit-Betriebssystem-fuer-Server-334946.html}}</ref>
<ref name="RHEL_2014_32bit">{{cite web |title=Are 32-bit applications supported in RHEL 7 or later releases? |publisher=[[Red Hat]] |date=February 2014 |url=https://access.redhat.com/solutions/509373}}</ref>
<ref name="Cooke_2019_32bit">{{cite web |title=Intel 32bit packages on Ubuntu from 19.10 onwards |author-first=Will |author-last=Cooke |date=2019-06-02 |publisher=Canonical |url=https://discourse.ubuntu.com/t/intel-32bit-packages-on-ubuntu-from-19-10-onwards/11263}}</ref>
<ref name="Addams_2018_Nvidia">{{cite web |title=Nvidia discontinues support for 32-bit Windows platforms |author-first=Matthew |author-last=Addams |publisher=Windows Report |date=2018-04-12 |url=https://windowsreport.com/nvidia-32-bit-windows-end-support/}}</ref>
<ref name="Silver_2018_Apple">{{cite web |title=Mojave is Apple's last version of macOS to support 32-bit apps |publisher=[[Apple Insider]] |author-first=Steven |author-last=Silver |date=2018-06-05 |url=https://appleinsider.com/articles/18/06/05/mojave-is-apples-last-version-of-macos-to-support-32-bit-apps}}</ref>
<ref name="Microsoft_Windows7">{{cite web |title=Der Support für Windows 7 endet am 14. Januar 2020 |language=de |publisher=[[Microsoft]] |url=https://support.microsoft.com/de-de/help/4057281/windows-7-support-ended-on-january-14-2020 |access-date=2020-02-09}}</ref>
<ref name="Sebayang_2019_Android">{{cite web |title=Auf dem Weg zu reinen 64-Bit-Android-Apps |language=de |publisher=Golem |author-first=Andreas |author-last=Sebayang |date=2019-01-17 |url=https://www.golem.de/news/google-auf-dem-weg-zum-reinen-64-bit-android-1901-138792.html}}</ref>
<ref name="MW_2014">{{cite web |title=Google kündigt Ende von 32-Bit-Android-Apps per 2021 an |language=de |publisher=IT Magazin |author=mw |date=2019-01-17 |url=https://www.itmagazine.ch/Artikel/68830/Google_kuendigt_Ende_von_32-Bit-Android-Apps_per_2021_an.html}}</ref>
<ref name="Android_User_2014">{{cite web |title=64-Bit-Android: Diese Prozessoren gibt es, diese Veränderungen kommen |language=de |date=2014-08-26 |publisher=Android User |url=https://www.android-user.de/64-bit-android-diese-prozessoren-gibt-es-diese-veraenderungen-kommen/}}</ref>
<ref name="APT_2015">{{cite web |title=Platform-tools 23.1.0 Linux changed to 64-bit without notice |publisher=Android Public Tracker |date=2015-12-11 |url=https://issuetracker.google.com/issues/37074522 |quote=It turns out the android-sdk-linux/platform-tools content is 32-bit ELF in 23.0.1 but 64-bit ELF in 23.1_rc1 and 23.1.0. […] I set ANDROID_EMULATOR_FORCE_32BIT=true […] 23.0.1 is the last 32-bit Linux build.}}</ref>
<ref name="Tenzer_2019">{{cite web |title=Anteile der verschiedenen Android-Versionen an allen Geräten mit Android OS weltweit im Zeitraum 01. bis 07. Mai 2019 |language=de |publisher=Statista |author-first=F. |author-last=Tenzer |date=2019-11-14 |url=https://de.statista.com/statistik/daten/studie/180113/umfrage/anteil-der-verschiedenen-android-versionen-auf-geraeten-mit-android-os/}}</ref>
<ref name="Favero_2019">{{cite web |title=Ingress und Pokémon Go brauchen bald mindestens Android 5 |author-first=Elia |author-last=Del Favero |date=2019-06-10 |url=https://www.nau.ch/news/games/ingress-und-pokemon-go-brauchen-bald-mindestens-android-5-65541520}}</ref>
<ref name="Reddit_2019">{{cite web |title=Why is 32bit 0.159.0 version apk still not available? |publisher=Reddit |work=TheSilphRoad/ |date=December 2019 |url=https://www.reddit.com/r/TheSilphRoad/comments/dm6c51/why_is_32bit_01590_version_apk_still_not_available/}}</ref>
<ref name="Microsoft_2019_CRT">{{cite web |url=https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/findfirst-functions?view=vs-2019 |title=C Run-time library (CRT) reference: findfirst |publisher=[[Microsoft]] |access-date=2020-02-17}}</ref>
<ref name="lfs_2015">{{cite web |url=https://users.suse.com/~aj/linux_lfs.html |title=Large File Support in Linux |publisher=[[SUSE S.A.|SuSE GmbH]] |date=2015-02-15 |author-first=Andreas |author-last=Jaeger}}</ref>
<ref name="Linux_stat.h">linux/bits/stat.h: /* Note stat64 has the same shape as stat for x86-64. */</ref>
<ref name="Rutter_2020_inode">{{cite web |url=https://www.mjr19.org.uk/sw/inodes64.html |title=The 64 bit inode problem |author-first=M. J. |author-last=Rutter |access-date=2020-02-10}}</ref>
<ref name="ext4_2019">{{cite web |url=https://ext4.wiki.kernel.org/index.php/Ext4_Howto |title=Ext4 Howto |website=kernel.org |date=2019-02-11 |quote=Although very large fileystems are on ext4's feature list, current e2fsprogs currently still limits the filesystem size to 2^32 blocks (16TiB for a 4KiB block filesystem). Allowing filesystems larger than 16T is one of the very next high-priority features to complete for ext4.}}</ref>
<ref name="Scherer_2016">{{cite web |url=https://www.elektormagazine.de/news/samsungs-32-tb-ssd-der-anfang-vom-ende-der-festplatte |title=Samsungs 32-TB-SSD: Der Anfang vom Ende der Festplatte |language=de |publisher=[[Elektor]] |date=2016-08-15 |author-first=Thomas |author-last=Scherer}}</ref>
}}
==External links==
* {{cite web |author-first=Andreas |author-last=Jaeger |date=2005-02-15 |title=Large File Support in Linux |publisher=[[SUSE S.A.|SuSE GmbH]] |url=http://www.suse.de/~aj/linux_lfs.html |access-date=2006-09-10}}
[[Category:Computer file systems]]
| |||