Intuition of information theory I am reading the book "Elements of Information Theory" by Cover and

Isaiah Farrell 2022-05-28 Answered
Intuition of information theory
I am reading the book "Elements of Information Theory" by Cover and Thomas and I am having trouble understanding conceptually the various ideas.
For example, I know that H(X) can be interpreted as the average encoding length. But what does H ( Y | X ) intuitively mean?
And what is mutual information? I read things like "It is the reduction in the uncertainty of one random variable due to the knowledge of the other". This doesn't mean anything to me as it doesn't help me explain in words why I ( X ; Y ) = H ( Y ) H ( Y | X ). Or explain the chain rule for mutual information.
I also encountered the Data processing inequality explained as something that can be used to show that no clever manipulation of the data can improve the inferences that can be made from the data. I ( X ; Y ) I ( X ; Z ). If I had to explain this result to someone in words and explain why it should be intuitively true I would have absolutely no idea what to say. Even explaining how "data processing" is related to a markov chains and mutual information would baffle me.
I can imagine explaining a result in algebraic topology to someone since there is usually an intuitive geometric picture that can be drawn. But with information theory if I had to explain a result to someone at comparable level to a picture I would not be able to.
When I do problems its just abstract symbolic manipulations and trial and error. I am looking for an explanation (not these blah gives information about blah explanations) of the various terms that will make the solutions to problems appear in a meaningful way.
Right now I feel like someone trying to do algebraic topology purely symbolically without thinking about geometric pictures.
Is there a book that will help my curse?
You can still ask an expert for help

Expert Community at Your Service

  • Live experts 24/7
  • Questions are typically answered in as fast as 30 minutes
  • Personalized clear answers
Learn more

Solve your problem for the price of one coffee

  • Available 24/7
  • Math expert for every subject
  • Pay only if we can solve it
Ask Question

Answers (2)

komizmtk
Answered 2022-05-29 Author has 8 answers
Christopher Olah wrote an excellent intuitive explanation of Information Theory called - Visual Information Theory. It provides thougtful visualizations for understanding these concepts.
In addition there was a paper that introduced a tool for visualizing mutual information called The Mutual Information Diagram for Uncertainty Visualization that may be useful.
Not exactly what you’re looking for?
Ask My Question
Briana Petty
Answered 2022-05-30 Author has 3 answers
Since you have an intuitive understanding of entropy based on the compression theorem, you should look into the operational meaning of mutual information, which is the channel coding theorem.
It says if you have a noisy channel with a joint distribution p(X,Y), then it can transmit information encoded in X to a receiving party with access to Y at a rate of I(X;Y) bits per symbol.
Not exactly what you’re looking for?
Ask My Question

Expert Community at Your Service

  • Live experts 24/7
  • Questions are typically answered in as fast as 30 minutes
  • Personalized clear answers
Learn more

You might be interested in

asked 2021-02-23
Interpreting z-scores: Complete the following statements using your knowledge about z-scores.
a. If the data is weight, the z-score for someone who is overweight would be
-positive
-negative
-zero
b. If the data is IQ test scores, an individual with a negative z-score would have a
-high IQ
-low IQ
-average IQ
c. If the data is time spent watching TV, an individual with a z-score of zero would
-watch very little TV
-watch a lot of TV
-watch the average amount of TV
d. If the data is annual salary in the U.S and the population is all legally employed people in the U.S., the z-scores of people who make minimum wage would be
-positive
-negative
-zero
asked 2022-07-02
Second Order Approximation for a Polynomial
if I have an expression: L = 12 a 3 d 3 4 w a 3 d 2 + 16 a 2 d 2 4 w a 2 d + 6 a d + 1 12 a 3 d 3 4 w a 3 d 2 4 a 2 w d + 16 a 2 d 2 + 7 a d a w + 1 what is the second order approximation in d w ?
I know that ( d w ) 2 can be ignored but what about d 2 w 3 . At this instant (without knowing the actual values of d wrt w) can we ignore this too? What about if we have (d/w=0.001)? Also how would the first order approximation in (d/w) be different in both cases?
asked 2022-05-01
Prove that the difference between numbers with the same sum of digits is a multiple of 9.
asked 2022-06-21
How to solve log x 3 = log x ?
asked 2021-08-11
add or subtract as indicated, and express answers in lowest terms. 16+18.
asked 2022-06-13
Let X be a locally compact Hausdorff space and C b (X) the set of all continuous functions with support compact. The dual of C b is isometric isomorphism of M(X), with M(X) the set of all regular Borel measures on X? Do the measures need to be finite as well?
asked 2022-02-12
How do you write the ratios 12:15 in simplest form?

New questions