Visual Information Theory

I really enjoyed this article, it was very accessible. Recently I've been studying some literature on stochastic optimal control, and I've bumped into the KL-divergence concept a number of times, but never really understood it. I expected this article be a fun read, but I never expected to learn something so directly useful! Information theory really does show up everywhere.

I often avoid these sort of visual methods, because:<p>- they only work with 2 variables, but many interesting problems require digging into more variables<p>- they only work with medium-sized numbers, and aren&#x27;t readable when P&lt;0.01 or P&gt;0.99<p>So they&#x27;re great for gaining intuition (like the Simpson&#x27;s Paradox example), but when you try to solve a real problem you find yourself boxed in.

After careful consideration I&#x27;ve always enjoyed how:<p><pre><code> p(rain,coat) = p(rain) * p(coat | rain) </code></pre> Can be pronounced: &quot;the probability of rain, and coat (wearing) is the probability of rain times the probability of (my wearing a) coat given rain&quot;. This intuitively showcases how the order of independent events doesn&#x27;t effect the outcome, since after all:<p><pre><code> p(coat,rain) = p(rain,coat) = p(rain) * p(coat | rain)</code></pre>

&quot;I love the feeling of having a new way to think about the world. I especially love when there’s some vague idea that gets formalized into a concrete concept. Information theory is a prime example of this.&quot;<p>THIS! THIS is why I love programming and electronics&#x2F;mechanical engineering. I live for the new ways to think.

Off-topic, but does anyone know what was used to draw the graphs? They look really clean<p>I&#x27;m guessing LaTeX? The font looked like LaTex&#x27;s font

i like forward to every article colah writes - the way he explained backpropagation a few weeks ago was really interesting - never thought about it that way but was very helpful!

It rains 25% of the time in California? Sounds like an unpleasant place.

The visual cortex is one of the largest and most powerful cortices in the human brain.<p>But it may be that vision&#x27;s supposed to work in conjunction with the other senses.<p>I think visual explanations work well for very simple visuals. As soon as higher order factors need to be factored-in, visual explanations are only sensible to the highly trained expert (think Feynman diagrams).<p>Nice essay, nevertheless. A lot of time and work went into it, and I can appreciate that.