Shameless plug: I'm trying to start an online reading group for "Elements of Information Theory" by Cover and Thomas.<p>If you're interested, please join (<a href="https://www.reddit.com/r/mathreadinggroup/" rel="nofollow">https://www.reddit.com/r/mathreadinggroup/</a>) and the discord (<a href="https://discord.gg/D9FjZXs" rel="nofollow">https://discord.gg/D9FjZXs</a>).
This is an interesting topic and is closely related to what I am working on at the moment.<p>One interesting thing I can say is that while subsets of a set are the language of probability and of logic (the existential quantifier and the universal quantifier are the two adjoints of the inverse image function), we have the dual to be the language of entropy: partitions of a set.<p>You can unify the two via a self-dual set theory, where you essentially replace the notions of subset (i.e., subobjects) and partitions (i.e., quotient objects) with an abstract subobject notion. This was part of the basis for my MSc and then my supervisor and a colleague of mine [1] showed that you can have what they call a Noetherian form over the category of sets. It is essentially by taking the pullback of the subobject and quotient object functors, though I can't remember if it is the usual pullback when seen as a category of categories.<p>[1] This is not all published, but you could have a look at the papers of Zurab Janelidze. The key point was his work on functorially self-dual group theory.
It's funny seeing this at the top of HN today. I just wrote a utility in Rust this past weekend to calculate the Shannon and metric entropy of a file (<a href="https://github.com/tuxxy/entropy/" rel="nofollow">https://github.com/tuxxy/entropy/</a>).<p>I really love the Shannon entropy equation. I don't know why, but I just find it so fascinating and elegant. I love it so much that I even got it tattooed on my wrist recently haha! (<a href="https://twitter.com/__tux/status/1257737842308022275/photo/1" rel="nofollow">https://twitter.com/__tux/status/1257737842308022275/photo/1</a>)
I like the original paper by Shannon "A Mathematical Theory of Communication" (<a href="http://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf" rel="nofollow">http://people.math.harvard.edu/~ctm/home/text/others/shannon...</a>) a lot. It is quite readable and also probably the most important paper in the field of information theory.
I was introduced first to "gray code" as an undergraduate for error correction. For those new to this topic it could be where you can start for an introduction.