[messel]

Focused on Information Theory in grad school at Stony Brook in 1996/97, and enjoyed Nam Phamdo's courses. Sadly after reading 8-9 pages I feel like much of this review is over my head now. The words make sense but the concepts are pretty dense.

I do remember the algorithms (Huffman coding), but the high level concepts are harder to remember - information being directly correlated to randomness (entropy). Information is the unknown (random?) message, that sounds more like noise. Channel capacity makes sense: based on noise there's an upper limit on how fast you can communicate bits and groups of bits.

[varlock]

Re: information is the unknown message. Think of it this way.

You read the newspaper each day. Every day it's got the same news: "Today the sun shines." This is no useful information (no entropy) and you can predict it very well.

Second scenario. You read the newspaper, but each day it states that either "Today the sun shines" or "Today is raining" with 50/50 probability. That's maximum entropy. That's useful information, which you can't reliably predict.