Unicode in Microsoft Windows: Difference between revisions

Revision as of 20:41, 26 July 2024

Revision as of 22:31, 26 July 2024

Line 25:

== UTF-8 ==

Microsoft Windows ([[Windows XP]] and later) has a code page designated for [[UTF-8]], code page 65001<ref>{{cite web|title=Code Page Identifiers (Windows)|url=https://msdn.microsoft.com/en-us/library/windows/desktop/dd317756(v=vs.85).aspx|website=msdn.microsoft.com| date=7 January 2021 |language=en}}</ref> or <code>CP_UTF8</code>. For a long time, it was impossible to set the locale code page to 65001, leaving this code page only available for (a) explicit conversion functions such as MultiByteToWideChar and/or (b) the [[Win32 console]] command <code>chcp 65001</code> to translate stdin/out between UTF-8 and UTF-16. This meant that "narrow" functions, in particular <code>[[C file input/output#fopen|fopen]]</code> (which opens files), couldn't be called with UTF-8 strings, and in fact there was no way to open all possible files using <code>fopen</code> no matter what the locale was set to and/or what bytes were put in the string, as none of the available locales could produce all possible UTF-16 characters. This problem also applied to all other APIs that take or return 8-bit strings, including Windows ones such as <code>SetWindowText</code>.

Programs that wanted to use UTF-8, in particular code intended to be portable to other operating systems, needed a workaround for this deficiency. The usual work-around was to add new functions to open files that convert UTF-8 to UTF-16 using [[MultiByteToWideChar]] and call the "wide" function instead of <code>fopen</code>.<ref>{{cite web|url=https://stackoverflow.com/questions/166503/utf-8-in-windows|title=UTF-8 in Windows|publisher=[[Stack Overflow]]|access-date=July 1, 2011}}</ref> Dozens of multi-platform libraries added wrapper functions to do this conversion on Windows (and pass UTF-8 through unchanged on others), an example is a proposed addition to [[Boost (C++ libraries)|Boost]], {{tt|Boost.Nowide}}.<ref>{{cite web|url=https://github.com/boostorg/nowide|title=Boost.Nowide|website=[[GitHub]]}}</ref> Another popular work-around was to convert the name to the [[8.3 filename]] equivalent, this is necessary if the <code>fopen</code> is inside a library. None of these workarounds are considered good, as they require changes to the code that works on non-Windows.