Unicode in Microsoft Windows: Difference between revisions

Revision as of 22:15, 10 March 2023

Revision as of 22:31, 10 March 2023

Line 26:

Microsoft Windows ([[Windows XP]] and later) has a code page designated for [[UTF-8]], code page 65001<ref>{{cite web|title=Code Page Identifiers (Windows)|url=https://msdn.microsoft.com/en-us/library/windows/desktop/dd317756(v=vs.85).aspx|website=msdn.microsoft.com|language=en}}</ref> or <code>CP_UTF8</code>. For a long time, it was impossible to set the locale code page to 65001, leaving this code page only available for (a) explicit conversion functions such as MultiByteToWideChar and/or (b) the [[Win32 console]] command <code>chcp 65001</code> to translate stdin/out between UTF-8 and UTF-16. This meant that "narrow" functions, in particular <code>[[C file input/output#fopen|fopen]]</code> (which opens files), couldn't be called with UTF-8 strings, and in fact there was no way to open all possible files using <code>fopen</code> no matter what the locale was set to and/or what bytes were put in the string, as none of the available locales could produce all possible UTF-16 characters. This problem also applied to all other APIs that take or return 8-bit strings, including Windows ones such as <code>SetWindowText</code>.

OnPrograms allthat modernwanted to non-Windowsuse platformsUTF-8, thein file-nameparticular stringcode passedintended to <code>fopen</code>be isportable effectivelyto UTF-8.other Thisoperating producedsystems, anneeded incompatibilitya betweenworkaround otherfor platformsthis and Windowsdeficiency. The usual work-around was to add codenew functions to open files that convert UTF-8 to UTF-16 using [[MultiByteToWideChar]] and call the "wide" function instead of <code>fopen</code>.<ref>{{cite web|url=https://stackoverflow.com/questions/166503/utf-8-in-windows|title=UTF-8 in Windows|publisher=[[Stack Overflow]]|access-date=July 1, 2011}}</ref> Dozens of multi-platform libraries added wrapper functions to do this conversion on Windows (and pass UTF-8 through unchanged on others, often quite incompatible with each other), an example is a proposed addition to [[Boost (C++ libraries)|Boost]], {{tt|Boost.Nowide}}.<ref>{{cite web|url=https://github.com/boostorg/nowide|title=Boost.Nowide|website=[[GitHub]]}}</ref> Though this allowed code to be "portable", it still required unnecessary rewrites. Another popular work-around was to convert the name to the [[8.3 filename]] equivalent, this is necessary if the <code>fopen</code> is inside a library. functionNone thatof takesthese aworkarounds stringare filenameconsidered andgood, thusas callingthey anotherrequire functionchanges isto notthe possiblecode that works on non-Windows.

In April 2018 (or possibly November 2017<ref>{{cite web|title=Windows10 Insider Preview Build 17035 Supports UTF-8 as ANSI|url=https://news.ycombinator.com/item?id=15710685|website=Hacker News|access-date=7 May 2018}}</ref>), with insider build 17035 (nominal build 17134) for Windows 10, a "Beta: Use Unicode UTF-8 for worldwide language support" checkbox appeared for setting the locale code page to UTF-8.{{efn|1=Found under control panel, "Region" entry, "Administrative" tab, "Change system locale" button.}} This allows for calling "narrow" functions, including <code>fopen</code> and <code>SetWindowTextA</code>, with UTF-8 strings. However this is a system-wide setting and a program cannot assume it is set.