TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Shannon Entropy Imposes Fundamental Limits on Communication

77 点作者 harscoat超过 2 年前

10 条评论

vba616超过 2 年前
&quot;If someone tells you a fact you already know, they’ve essentially told you nothing at all.&quot;<p>I think it&#x27;s interesting to consider how this is&#x2F;is not true in a real life context.<p>If someone tells you a fact you already know, then it could be communicating several different things.<p><pre><code> 1. It might mean *they think* you don&#x27;t know it. 2. It might mean they want to make sure *you know* they know you know it. 3. It might mean they believe you know it and want to communicate they agree. 4. It might mean they are hoping for you to (or not to) contradict&#x2F;argue&#x2F;refute it. </code></pre> If, however, you can&#x27;t tell which of those (or some other alternative) is intended, <i>then</i> no information was communicated at all.<p>Except...there is a message communicated that they have <i>some</i> interest or concern regarding the topic, which is different from remaining silent.<p>Whenever someone says <i>anything</i>, true, false, nonsense, lie, they are communicating a true and possibly significant fact - that they chose to say that thing, in whatever circumstance.<p>I am thinking of &quot;Gödel, Escher, Bach&quot;, if you can&#x27;t tell.
评论 #32762282 未加载
评论 #32764776 未加载
zbobet2012超过 2 年前
&gt; That you can zip a large movie file, for example, owes to the fact that pixel colors have a statistical pattern, the way English words do. Engineers can build probabilistic models for patterns of pixel colors from one frame to the next.<p>This is, in fact, incorrect generally. Movie files are generally compressed via _lossy_ compression, which escapes the Shannon entropy limit. Entropy based encoding, as described in this article, is generally only the final stage in a video &quot;codec&quot; (algorithm to compress, and decompress video). The first stage relies of finding information in the visual field that humans are unlikely to notice the loss of and discarding it. The second stage does interframe (between frames) compression by simply searching for blocks of pixels that match. Entropy coding is generally the final stage.<p>This, by the way, is why zipping a video file can make it... larger.
评论 #32760608 未加载
评论 #32759942 未加载
ttctciyf超过 2 年前
&gt; Engineers can build probabilistic models for patterns of pixel colors from one frame to the next. The models make it possible to calculate the Shannon entropy [...] That value tells you the limit of “lossless” compression — the absolute most the movie can be compressed before you start to lose information about its contents.<p>While this is generally true in practice, strictly speaking the shortest possible encoding of a sequence of bits could be smaller than this would suggest, since it&#x27;s possible a pattern exists that has escaped notice.<p>For example: the first 10 billion digits of pi pass statistical tests for randomness, but can be generated by a short program which calculates them until the desired length is reached, in effect &quot;compressing&quot; them to the length of the generator program.<p>Because of considerations like this, Algorithmic Information Theory[1] equates the information content of a bit sequence with the length of the shortest program which will generate it - though this has the drawback of being generally uncomputable - an intriguingly different paradigm.<p>1: see <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Algorithmic_information_theory" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Algorithmic_information_theory</a>
评论 #32763079 未加载
评论 #32760692 未加载
评论 #32762093 未加载
评论 #32758661 未加载
评论 #32762869 未加载
评论 #32759176 未加载
acjohnson55超过 2 年前
The information theory class I took in grad school was one of the most mindbending classes I&#x27;ve ever taken. It made doing gnarly math problems fun.<p>Claude Shannon was a titan. He&#x27;s also known for publishing what some call the most important masters thesis of all time, <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;A_Symbolic_Analysis_of_Relay_and_Switching_Circuits" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;A_Symbolic_Analysis_of_Relay_a...</a>.
rkp8000超过 2 年前
For anyone interested in these ideas, I&#x27;d highly recommend reading Shannon&#x27;s original 1948 paper (linked in the article). While technical and precise it is actually quite digestible for someone with a bit of a quantitative background.
评论 #32757360 未加载
评论 #32757429 未加载
mpalmer超过 2 年前
Information theory is fascinating to anyone who likes the mathy underpinnings of seemingly non-mathy things, and it&#x27;s <i>everywhere</i>. Cool stuff.<p>Speaking of information and decompression, my brain initially interpreted the title as a breaking news headline about a female supervillain who&#x27;s also a theory nerd.<p>It was only the added context of the domain name that fixed the error.
javajosh超过 2 年前
I&#x27;m more interested in the problem of sending strings out into the world where more strings may &quot;stick&quot; to them, possibly producing more strings. You know, like life. Or software. Given a particular string, can do you determine its age? Can you determine which discrete strings came into contact with it along its journey and when? And so on. I&#x27;m not sure if this is interesting to anyone else, but it sure is interesting to me! Consider how these are the boundary conditions that justify software design decisions so primordial we don&#x27;t even consider them decisions anymore.
_wldu超过 2 年前
One example I like to use (when talking about entropy):<p>A four digit numeric PIN (that we know) has 0 bits of entropy. There is no uncertainty about what the PIN actually is. A randomly selected one (that we do not know) has just over 13 bits.<p>print(math.log(10)&#x2F;math.log(2)*4)<p>13.28771237954945<p>The more entropy, the more uncertain we are.<p>However, humans are not random. We use the year we were born, some keyboard sequence or some other predictable number as our PIN. We don&#x27;t know exactly how much entropy these PINS have (there is some degree of uncertainty), but we do know they are significantly less than 13 bits.
评论 #32759666 未加载
评论 #32761362 未加载
bob1029超过 2 年前
Information theory is so cool. I feel like having a solid grasp of the fundamentals is critical for achieving mastery in software development. Closely related would be digital signal processing and frequency domain techniques, both also extremely useful in many branches of software.<p>The actual information content of a thing has always surprised me, especially when factoring in slightly imperfect (but passable) representations such as JPEG or MP3. To lose some information seems entirely acceptable in many cases.
ffhhj超过 2 年前
The Dartmoth Workshop:<p><a href="http:&#x2F;&#x2F;www-formal.stanford.edu&#x2F;jmc&#x2F;slides&#x2F;dartmouth&#x2F;dartmouth&#x2F;node1.html" rel="nofollow">http:&#x2F;&#x2F;www-formal.stanford.edu&#x2F;jmc&#x2F;slides&#x2F;dartmouth&#x2F;dartmout...</a>