Hsuan-Tien Lin > Courses > Introduction to Information Theory, Fall 2019

Hsuan-Tien Lin

Home | MOOCs | AIsk | Courses | Research Group | Awards | Publications | Presentations | Programs/Data

Introduction to Information Theory, Fall 2019

Course Description

The course addresses fundamental questions in quantitatively defining, evaluating, or estimating information. What is information? How many bits do we need to store some data? How many bits do we need to communicate some messages? How many bits do we need to produce some strings? How many bits do we need to learn some concepts? How many bits do we need to implement some formula? The course answers these questions in a broad and connected manner, and gives students some starting points to understand important areas like compression, error correcting code, machine learning, and neural computation.

People

instructor: Hsuan-Tien Lin (htlin AT csie . ntu . edu . tw) [office hour: after classes, or by appointment]
TAs and TA hour: iit_ta AT csie . ntu . edu . tw
- Chien-Min Yu: Mondays 16:30--17:30 in CSIE R536

Course Information

Time: Tuesdays 14:20 to 16:20
Room: CSIE Building, R111
Textbook: none
Language: Mandarin teaching with boards
Grading: 100% homework (tentative)
Full Course Website: http://ceiba.ntu.edu.tw/1081iit

Announcements

homework -log2 sqrt(Omega) announced on 01/07/2020
homework 3 announced on 12/11/2019
homework 2 announced on 11/05/2019
homework 1 announced on 10/01/2019
policy announced on 09/10/2019

Class Policy

main policy

Course Plan (tentative)

date	syllabus	todo/done	suggested reading
09/10	course introduction; what is information?		course introduction handout our cool discussion on what information is
09/17	information in storage: deterministic and probabilistic information		handout IAC notes 1.1.1, 1.1.2, 1.1.3
09/24	information in storage: entropy		handout IAC notes 1.2.1, 1.2.4
10/01	information in storage: Shannon's first theorem	homework 1 released	handout IAC notes 1.3.1, 1.3.2 (almost)
10/08	information in storage: source coding		handout IAC notes 1.3.2, 1.3.3, 1.4.1, 1.4.2
10/15	information in storage: Huffman coding		handout IAC notes 1.4.3, 1.4.4
10/22	information in storage: scalar quantization		handout IAC notes 1.5.1
10/29	information in storage: mutual information		handout IAC notes 1.2.2, 1.2.3
11/05	information in storage: vector quantization	homework 1 due; homework 2 released	handout IAC notes 1.5.2
11/12	information in storage: rate-distortion tradeoff		handout IAC notes 1.5.3
11/19	no class as instructor attends ACML
11/26	information in communication: channel capacity		handout IAC notes 3.2.1, 3.2.3, 3.3
12/03	information in computation: computational compression	homework 3 released	handout IAC notes 4.1.1, 4.1.2
12/10	information in computation: Kolmogorov complexity	homework 2 due	handout IAC notes 4.1.2
12/17	information in computation: algorithmic entropy		handout IAC notes 4.1.4
12/24	information in computation/learning: universal probability		handout Algorithmic Information Theory: Review for Physicists and Natural Scientists 3.2, 3.6
12/31	information in learning: Solomonoff inference and PAC	homework -log2 sqrt(Omega) due	handout Tutorial on Practical Prediction Theory for Classification Section 4
01/07	information in learning: more about PAC; summary		handout
01/14	winter vacation begins (really?)	homework 3 due

Last updated at CST 13:08, October 04, 2023
Please feel free to contact me: htlin.email.png