Entropy and Mutual Information: Entropy, joint entropy and conditional entropy, Relative entropy and mutual information, chain rules for entropy, relative entropy and mutual information, Jensen’s inequality and log-sum inequality. Differential Entropy: Differential entropy and discrete entropy, joint and conditional differential entropy, properties of differential entropy, relative entropy and mutual information. Entropy Rates of Stochastic Process: Markov Chain, Entropy rate and hidden Markov models. Source Coding: Kraft inequality, optimal codes, Huffman code and its optimality, Shannon-Fano-Elias coding, arithmetic coding. Channel Capacity: Binary symmetric channels and properties of channel capacity, channel coding theorems, joint source and channel coding theorem. Block coding and decoding, BCH, RS codes, Convolutional coding, Viterbi Decoder, Turbo codes, decoding techniques: STBC, SFBC, STFBC. Gaussian Channel: Introduction to Gaussian Channel, Band limited channel, Parallel Gaussian Channel, Gaussian Channel with feedback.
