Fascinating topic. There are two ways the user/client/browser receives reports about the character encoding of content. And there are hefty caveats about how reliable those reports are.<p>(1) First, the Web server usually reports a character encoding, a.k.a. charset, in the HTTP headers that come with the content. Of course, the HTTP headers are not part of the HTML document but are rather part of the overhead of what the Web server sends to the user/client/browser. (The HTTP headers and the `head` element of an HTML document are entirely different.) One of these HTTP headers is called Content-Type, and conventionally this header often reports a character encoding, e.g., "Content-Type: text/html; charset=UTF-8". So this is one place a character encoding is reported.<p>If the actual content is <i>not</i> an (X)HTML file, the HTTP header might be the only report the user/client/browser receives about the character encoding. Consider accessing a plain text file via HTTP. The text file isn't likely to itself contain information about what character encoding it uses. The HTTP header of "Content-Type: text/plain; charset=UTF-8" might be the only character encoding information that is reported.<p>(2) Now, if the content is an (X)HTML page, a charset encoding is often also reported in the content itself, generally in the HTML document's head section in a meta tag such as '<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>' or '<meta charset="utf-8">'. Now just because an HTML document self-reports that it uses a UTF-8 (or whatever) character encoding, that's hardly a guarantee that the document does in fact use said character encoding.<p>Consider the case of a program that generates web pages using a boilerplate template still using an ancient default of ISO-8859-1 in the meta charset tag of its head element, even though the body content that goes into the template is being pulled from a database that spits out a default of utf-8. Boom. Mismatch. Janky code is spitting out mismatched and inaccurate character encoding information every day.<p>Or to consider web servers. Consider a web server whose config file contains the typo "uft-8" because somebody fat-fingered while updating the config (I've seen this in random web pages.). Or consider a web server that uses a global default of "utf-8" in its outgoing HTTP headers even when the content being served is a hodge-podge of UTF-8, WINDOWS-1251, WINDOWS-1252, and ISO-8859-1. This too happens all the time.<p>I think the most important takeaway is that with both HTTP headers and meta tags, there's no intrinsic link between the character encoding being reported and the <i>actual</i> character encoding of the content. What a Web server tells me and what's in the meta tag in the markup just count as two reports. They might be accurate, they might not be. If it really matters to me what the character encoding is, there's nothing for it but to determine the character encoding myself.<p>I have a Hacker News reader, <a href="https://www.thnr.net" rel="nofollow">https://www.thnr.net</a>, and my program downloads the URL for every HN story with an outgoing link. I have seen binary files sent with a "UTF-8" Content-Type header. I have seen UTF-8 files sent with a "inode/x-empty" Content-Type header. My logs have literally hundreds of goofy inaccurate reports of content types and character encodings. Because I'm fastidious and I want to know what a file actually is, I have a function `get_textual_mimetype` that analyzes the content of what the URL's web server sends me. My program downloads the content and uses tools such as `iconv` and `isutf8` to get some information about what encoding it might be. It uses `xmlwf` to check if it's well-formed XML. It uses `jq` to check whether it's valid JSON. It uses `libmagic`. There's a lot of fun stuff the program does to pin down with a high degree of certainty what the content is. I want my program to know whether the content is an application/pdf, an iamge/webp, a text/html, an application/xhtml+xml, a text/x-csrc, or whatever. Only a rigorous analysis will tell you the truth. (If anyone is curious, the source for `get_textual_mimetype` is in the repo for my HN reader project: <a href="https://github.com/timoteostewart/timbos-hn-reader/blob/main/utils_mimetypes_magic.py">https://github.com/timoteostewart/timbos-hn-reader/blob/main...</a> )