
Lecture 20 of 53 · Information Theory
Himanshu Tyagi
Information Theory · 53 lectures
53 lectures
How to model uncertainty?
Basic concepts of probability
Estimates of random variables
Limit theorems
Source model
Motivating examples
A compression problem
Shannon entropy
Random hash
Generating almost random bits
Generating samples from a distribution using uniform randomness
Typical sets and entropy
Hypothesis testing and estimation
Examples
The log-likelihood ratio test
Kullback-Leibler divergence and Stein’s lemma
Multiple hypothesis testing
Error analysis of multiple hypothesis testing
24 Mutual information
26 Measures of information
27 Chain rules
28 Shape of measures of information
29 Data processing inequality
30 Proof of Fano’s inequality
31 Variational formulae
32 Capacity as information radius
33 Proof of Pinsker’s inequality
34 Continuity of entropy
35 Lower bound for compression
38 Lower bound for random number generation
39 Strong converse
40 Lower bound for minmax statistical estimation
41 Variable length source codes
44 Huffman code
42 Kraft’s inequality
43 Shannon code
45 Minmax Redundancy
46 Type based universal compression
Information Theory Review
47 Arithmetic code
48 Online probability assignment
49 Compression of databases: A scheme
50 Compression of databases: A lower bound
51 Repetition code
52 Channel capacity
53 Sphere packing bound for BSC
54 Random coding bound for BSC
55 Random coding bound for general channel
57 Converse proof for channel coding theorem
58 Additive Gaussian Noise channel
59 Mutual information and differential entropy
60 Channel coding theorem for Gaussan channel
61 Parallel channels and water-filling