Fifty Years of Shannon Theory (1998) [pdf]

For anyone interested in a quick visual overview of some important ideas in information theory, I&#x27;d like to give a shout out to colah&#x27;s &quot;Visual Information Theory&quot; post: <a href="http:&#x2F;&#x2F;colah.github.io&#x2F;posts&#x2F;2015-09-Visual-Information&#x2F;" rel="nofollow">http:&#x2F;&#x2F;colah.github.io&#x2F;posts&#x2F;2015-09-Visual-Information&#x2F;</a>

I read Shannon&#x27;s original papers and have been trying to grok information theory off and on with only limited success for several years. The book &quot;Information Theory: A Tutorial Introduction&quot; by Stone really helped the principles sink in. For anyone who&#x27;d like a textual introduction, I highly recommend it. It&#x27;s semi-technical - high-school level math helps explain much of the foundational concepts.<p>If you&#x27;re feeling more adventurous, Cover and Thomas&#x27;s textbook is the information theory bible. It&#x27;s very dense but absolutely packed with insight.

Recently, I read a great book that goes into Shannon a bit. It&#x27;s called &quot;The Information: A History, A Theory, A Flood&quot; by James Gleik.<p>It starts by talking about African drum speech, and how it is a very simplified version of speech, and how it deals with error correction over great distances. From there it goes into a lot of great historical explanations, including Shannon.

Tip: Claude Shannon&#x27;s new biography [<a href="https:&#x2F;&#x2F;www.amazon.com&#x2F;Mind-Play-Shannon-Invented-Information-ebook&#x2F;dp&#x2F;B01M5IJN1P&#x2F;ref=sr_1_1?s=books&amp;ie=UTF8&amp;qid=1515747422&amp;sr=1-1&amp;keywords=claude+shannon" rel="nofollow">https:&#x2F;&#x2F;www.amazon.com&#x2F;Mind-Play-Shannon-Invented-Informatio...</a>] is an enjoyable read.

Are there any online tutorials or courses that guide us step by step to create a non trivial fun project that uses Shannon Theory? For example a program that simulates encoding a message on one end and decoding it on the other end.

Claude Shannon&#x27;s &quot;A Mathematical Theory of Communication&quot; is available online, and in bookstores.<p><a href="http:&#x2F;&#x2F;affect-reason-utility.com&#x2F;1301&#x2F;4&#x2F;shannon1948.pdf" rel="nofollow">http:&#x2F;&#x2F;affect-reason-utility.com&#x2F;1301&#x2F;4&#x2F;shannon1948.pdf</a><p>Khan Academy has done a information theory<p><a href="https:&#x2F;&#x2F;www.khanacademy.org&#x2F;computing&#x2F;computer-science&#x2F;informationtheory" rel="nofollow">https:&#x2F;&#x2F;www.khanacademy.org&#x2F;computing&#x2F;computer-science&#x2F;infor...</a>

Focused on Information Theory in grad school at Stony Brook in 1996&#x2F;97, and enjoyed Nam Phamdo&#x27;s courses. Sadly after reading 8-9 pages I feel like much of this review is over my head now. The words make sense but the concepts are pretty dense.<p>I do remember the algorithms (Huffman coding), but the high level concepts are harder to remember - information being directly correlated to randomness (entropy). Information is the unknown (random?) message, that sounds more like noise. Channel capacity makes sense: based on noise there&#x27;s an upper limit on how fast you can communicate bits and groups of bits.