TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

WorstFit: Unveiling Hidden Transformers in Windows ANSI

373 点作者 notmine13374 个月前

29 条评论

vessenes4 个月前
This is a tough one. It’s systemic —- MS provides a “best fit” code mapping from wide Unicode to ASCII, which is a known, published, “vibes-based” mapper. This best fit parser is used a lottt of places, and I’m sure that it’s required for ongoing inclusion based on how MS views backward compatibility. It’s linked in by default everywhere, whether or not you know you included it.<p>The exploits largely revolved around either speccing an unusual code point that “vibes” into say a slash or a hyphen or quotes. These code points are typically evaluated one way (correct full Unicode evaluation) inside a modern programming language, but when passed to shell commands or other Win32 API things are vibes-downed. Crucially this happens <i>after</i> you check them, since it’s when you’ve passed control.<p>To quote the curl maintainer “curl is a victim” here — but who is the culprit? It seems certain that curl will be used to retrieve user supplied data automatically by a server in the future. When that server mangles user input in one way for validation and another when applied to system libraries, you’re going to have a problem.<p>It seems to me like maybe the solution is to provide an opt-out of “best fit” munging in the Win32 space, but I’m not a Windows guy, so I speculate. At least then open source providers could just add the opt out to best practices, and deal with the many terrible problems that things like a Unicode wide variant of “ or \ delivers to them.<p>And of course even if you do that, you’ll interact with officially shipped APIs and software that has <i>not</i> opted out.
评论 #42647998 未加载
评论 #42653554 未加载
评论 #42710007 未加载
评论 #42651922 未加载
mmastrac4 个月前
This is kind of unsurprising, but still new to me even as someone who did Windows development (and some Wine API hacking) for a decade around when this W&#x2F;A mess came about.<p>Windows is like the card game Munchkin, where a whole bunch of features can add up to a completely, unbelievably random over-powered exploit because of unintentional synergy between random bits.<p>I&#x27;m happy to see that they are converting the ANSI subsystem to UTF-8, which should, in theory, mitigate a lot of these problems.<p>I wonder if the Rust team is going to need YetAnotherFix to the process spawning API to fix this...
评论 #42660964 未加载
Joker_vD4 个月前
&gt; the only thing we can do is to encourage everyone, the users, organizations, and developers, to gradually phase out ANSI and promote the use of the Wide Character API,<p>This has been Microsoft&#x27;s official position since NT 3.5, if I remember correctly.<p>Sadly, one of the main hurdles is the way Microsoft&#x27;s own C&#x2F;C++ runtime library (msvcrt.dll) is implemented. Its non-standard &quot;wide&quot; functions like _wfopen(), _wgetenv(), etc. internally use W-functions from Win API. But the standard, &quot;narrow&quot; functions like fopen(), getenv(), etc., instead of using the &quot;wide&quot; versions and converting to-from Unicode themselves (and reporting conversion failures), simply use A-functions. Which, as you see, generally don&#x27;t report any Unicode conversion failures but instead try to gloss over them using best-fit approach.<p>And of course, nobody who ports software, written in C, to Windows wants to rewrite all of the uses of standard functions to use Microsoft&#x27;s non-portable functions because at this point, it becomes a full-blown rewrite.
评论 #42647989 未加载
评论 #42648790 未加载
评论 #42648957 未加载
评论 #42647980 未加载
评论 #42649272 未加载
评论 #42651051 未加载
Dwedit4 个月前
There are two ways to force the &quot;Ansi&quot; codepage to actually be UTF-8 for an application that you write (or an EXE that you patch).<p>One way is with a Manifest file, and works as of a particular build of Windows 10. This can also be applied to any EXE after building it. So if you want a program to gain UTF-8 support, you can hack it in. Most useful for console-mode programs.<p>The other way is to use the hacks that &quot;App Locale&quot; type tools use. One way involves undocumented function calls from NTDLL. I&#x27;m not sure exactly which functions you need to call, but I think it might involve &quot;RtlInitNlsTables&quot; and &quot;RtlResetRtlTranslations&quot; (not actually sure).
layer84 个月前
&gt; until Microsoft chooses to enable UTF-8 by default in all of their Windows editions.<p>I don’t know how likely this is. There are a lot of old applications that assume a particular code page, or assume 1 byte per character, that this would break. There are also more subtle variations of this, like applications assuming that converting from wide characters to ANSI can’t increase the number of bytes (and hence an existing buffer can be safely reused), which isn’t the case for UTF-8 (but for all, or almost all, existing code pages). It can open up new vulnerabilities.<p>It would probably cause much less breakage to remove the Best-Fit logic from the win32 xxxA APIs, and instead have all unmappable characters be replaced by a character without any common meta semantics, like “x”.
评论 #42650777 未加载
评论 #42649502 未加载
评论 #42657433 未加载
garganzol4 个月前
Microsoft was aware of this issue at least 1 year ago. I know this because they released a special code analysis rule CA2101 [1] that explicitly discouraged the use of the best-fit mapping. They mentioned security vulnerabilities in the rule’s description, but they were purposefully vague in details though.<p>[1] <a href="https:&#x2F;&#x2F;learn.microsoft.com&#x2F;en-us&#x2F;dotnet&#x2F;fundamentals&#x2F;code-analysis&#x2F;quality-rules&#x2F;ca2101" rel="nofollow">https:&#x2F;&#x2F;learn.microsoft.com&#x2F;en-us&#x2F;dotnet&#x2F;fundamentals&#x2F;code-a...</a>
cesarb4 个月前
&gt; However, resolving this problem isn’t that as simple as just replacing the main() with its wide-character counterpart. Since the function signature has been changed, maintainers would need to rewrite all variable definitions and argument parsing logics, converting everything from simple char * to wchar_t *. This process can be painful and error-prone.<p>You don&#x27;t need to convert everything from char * to wchar *. You can instead convert the wide characters you received to UTF-8 (or to something like Rust&#x27;s WTF-8, if you want to also allow invalid sequences like unpaired surrogates), and keep using &quot;char&quot; everywhere; of course, you have to take care to not mix ANSI or OEMCP strings with UTF-8 strings, which is easy if you simply use UTF-8 everywhere. This is the approach advocated by the classic <a href="https:&#x2F;&#x2F;utf8everywhere.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;utf8everywhere.org&#x2F;</a> site.
segasaturn4 个月前
I&#x27;ve been inadvertantly safe from this bug on my personal Windows computer for years thanks to having the UTF-8 mode set, as shown at the bottom of the article. I had it set due to some old, foreign games showing garbled nonsense text on my computer. Have not noticed any bugs or side effects despite it being labelled as &quot;Beta&quot;.
评论 #42649606 未加载
评论 #42651907 未加载
scoopr4 个月前
I was wondering if the beta checkbox the same thing as setting the ActiveCodePage to UTF-8 in the manifest, but the docs[0] clarify that GDI doesn&#x27;t adhere to per-process codepage, but only a single global one, which is what the checkbox sets.<p>Bit of a shame that you can&#x27;t fully opt-in to be UTF-8 with the *A API, for your own apps. But I think for the issues highlighted in the post, I think it would still be a valid workaround&#x2F;defence-in-depth thing.<p>[0] <a href="https:&#x2F;&#x2F;learn.microsoft.com&#x2F;en-us&#x2F;windows&#x2F;apps&#x2F;design&#x2F;globalizing&#x2F;use-utf8-code-page" rel="nofollow">https:&#x2F;&#x2F;learn.microsoft.com&#x2F;en-us&#x2F;windows&#x2F;apps&#x2F;design&#x2F;global...</a>
lifthrasiir4 个月前
Oh, my, freaking, god. I knew Windows API provides that sort of best-fit conversions, but didn&#x27;t realize that it was a default behavior for several ANSI functions in my native code page (949 [1])! At this point they should be just banned like gets.<p>[1] Yes, I know there is a UTF-8 code page (65001). That was really unusable for a long time and still is suffering compatibility issues to this day.
mouse_4 个月前
Unicode on modern systems is absolutely terrifying. Anyone remember the black dot of death? <a href="https:&#x2F;&#x2F;mashable.com&#x2F;article&#x2F;black-dot-of-death-unicode-imessage-iphone-crash" rel="nofollow">https:&#x2F;&#x2F;mashable.com&#x2F;article&#x2F;black-dot-of-death-unicode-imes...</a>
kazinator4 个月前
HN, Help! Before I dive into this, does anyone know whether this affects the argument parsing in Cygwin, that prepares the arguments for a regular int main(int argc, char *argv)?<p>TXR Lisp uses wchar_t strings, and the &quot;W&quot; functions on Windows. So that&#x27;s well and good. But it does start with a regular C main, relying on the Cygwin run-time for that.<p>If that&#x27;s vulnerable, I will hack it to have its own argument parsing, using the wide char command line.<p>Maybe I should ask this on the Cygwin mailing list.
评论 #42655676 未加载
评论 #42660625 未加载
评论 #42658570 未加载
bangaladore4 个月前
I tend to agree that this is not an issue with many of the applications that are mentioned in the post.<p>Fundamentally this boils down to essentially bugs in functions that are supposed to transform untrusted into trusted input like the example they gave:<p>`system(&quot;wget.exe -q &quot; . escapeshellarg($url));`<p>`escapeshellarg` is not producing a trusted output with some certain inputs.
评论 #42648430 未加载
sharpshadow4 个月前
That’s amazing great read. According to for example this[0] post it’s possible to change code pages in windows in various ways and would allow the use of multiple BestFit scenarios on the same OS without reboot. Even combining them should be possible.
评论 #42662833 未加载
pornel4 个月前
It would be easily fixable if CommandlineToArgvA was obtaining the command line itself. Then instead of converting to ANSI and then parsing that, it could parse args in Unicode, and then convert argument by argument to ANSI. The output would be ANSI compatible, but split and unescaped in the true form.<p>Unfortunately, the parsing is a two-step operation, with the application calling GetCommandLineA itself first and passing that to the parser, so a fix would need a hack to correlate the versions of the command line input without breaking when it&#x27;s given a different string.
nitwit0054 个月前
There are presumably some similar .Net COM issues when communicating with unmanaged code, as there is an attribute for controlling this conversion: <a href="https:&#x2F;&#x2F;learn.microsoft.com&#x2F;en-us&#x2F;dotnet&#x2F;api&#x2F;system.runtime.interopservices.bestfitmappingattribute" rel="nofollow">https:&#x2F;&#x2F;learn.microsoft.com&#x2F;en-us&#x2F;dotnet&#x2F;api&#x2F;system.runtime....</a><p>It directly mentions: &quot;Setting BestFitMappingAttribute parameters in this manner provides an added measure of security.&quot;
LudwigNagasena4 个月前
What would even be the proper way to do `system(&quot;wget.exe -q &quot; . escapeshellarg($url))`? It’s ridiculous that plaintext IPC is still the primary interface for many tools.
评论 #42658603 未加载
评论 #42656726 未加载
layer84 个月前
&gt; And yes, Python’s subprocess module can’t prevent this.<p>A reasonably sane solution would be for it to reject command line arguments on Windows that contain non-ASCII characters or ASCII characters that aren’t portable across code pages (not all code pages are a superset of US-ASCII), by default, and to support an optional parameter to allow the full range, documenting the risk.
评论 #42653332 未加载
radarsat14 个月前
Seems like a another possible fix would be to change the best fit mapping table to never generate control characters, but only alphanumerics. So map quote-like characters to &#x27;q&#x27; and so on.<p>This might be uglier and slightly change behaviour, but only for vulnerable applications.
ok1234564 个月前
Bush hid the facts
评论 #42649123 未加载
评论 #42648681 未加载
lilyball4 个月前
&gt; <i>Worse still, as the attack exploits behavior at the system level during the conversion process, no standard library in any programming language can fully stop our attack!</i><p>What happens if the standard library updates its shell escaping to also escape things like the Yen character and any other character that has a Best-Fit translation into a quote or backslash? Which is to say, what does Windows do for command-line splitting if it encounters a backslash-escaped nonspecial character in a quoted string? If it behaves like sh and the backslash simply disables special handling of the next character, then backslash-escaping any threat characters should work.
评论 #42653103 未加载
评论 #42658634 未加载
account424 个月前
Window&#x27;s A APIs and conversion functions are best ingored entirely.<p>Always use W functions and use your own converions (that can round-trip invalid UTF-16 like WTF-8) if you want to use an 8-bit encoding internally.<p>Most (all?) of the exploits here are already bugs because the applications don&#x27;t properly handle unicode data.
rubatuga4 个月前
From what I can tell the largest vulnerability is argument passing to executables in Windows. Essentially it is very difficult to safeguard it. I&#x27;ve seen some CLI programs use the &#x27;--&#x27; to signify user input at the end, maybe this would solve this for a single argument scenario. Overall, this is an excellent article and vulnerability discovery.
ppp9994 个月前
Character encoding has been such a mess for so long it&#x27;s crazy.
评论 #42662888 未加载
est4 个月前
I remember typing some prefix character in notepad.exe then your hole txt became messed up. Funny unicode times.
评论 #42654566 未加载
UltraSane4 个月前
The loosey-goosey mapping of code points to characters has always bothered me about Unicode.<p>To guard against this nasty issue that is going to take years to fix you can enable global UTF-8 support by doing<p>Settings &gt; Time &amp; language &gt; Language &amp; region &gt; Administrative language settings &gt; Change system locale, and check Beta: Use Unicode UTF-8 for worldwide language support. Then reboot the PC for the change to take effect.
评论 #42658640 未加载
Randor4 个月前
That was a long read. Just be happy that you never had to deal with Trigraphs. <a href="https:&#x2F;&#x2F;learn.microsoft.com&#x2F;en-us&#x2F;cpp&#x2F;c-language&#x2F;trigraphs?view=msvc-170" rel="nofollow">https:&#x2F;&#x2F;learn.microsoft.com&#x2F;en-us&#x2F;cpp&#x2F;c-language&#x2F;trigraphs?v...</a>
EdSharkey4 个月前
Distributing native binaries is so dangerous!
评论 #42655024 未加载
tiahura4 个月前
Imagine no Unicode, It’s easy if you try, No bytes that bloat our systems, No errors make us cry. Imagine all the coders, Living life in ASCII…<p>Imagine no emojis, Just letters, plain and true, No accents to confuse us, No glyphs in Sanskrit too. Imagine all the programs, Running clean and fast…<p>You may say I’m a dreamer, But I’m not the only one. I hope someday you’ll join us, And encoding wars will be done.
评论 #42653173 未加载
评论 #42654481 未加载