Hsuan-Tien Lin > Courses > Machine Learning Foundations/Techniques, Fall 2020

Machine Learning Foundations/Techniques, Fall 2020

Course Description

Machine learning allows computational systems to adaptively improve their performance with experience accumulated from the data observed. This course introduces the basics of learning theories, the design and analysis of learning algorithms, and some applications of machine learning.

People

instructor: Hsuan-Tien Lin (htlin AT csie . ntu . edu . tw) [office hour: after classes, or by appointment]
TAs and TA hour: html_ta AT csie . ntu . edu . tw
- Si-An Chen, D09, Mondays 10:00-11:00, CSIE R536
- Chi-Pin Huang, B07, Mondays 14:00-15:00, CSIE Basement (Red Sofa)
- Yu-Chu Yu, R09, Tuesdays 09:00-10:00, CSIE R536
- Yi-Hung Chiu, B05, Wednesdays 09:00-10:00, CSIE Basement (Red Sofa)
- Wei-I Lin, B05, Wednesdays 17:30-18:30, CSIE R536 (call 02-33664888x536 if you cannot go to the 5th floor)
- Yu-Hsiang Huang, B07, Thursdays 09:00-10:00, CSIE Basement (Red Sofa)
- Yu-Chen Lin, B06, Thursdays 11:00-12:00, CSIE Basement (Red Sofa)

Course Information

Time: Tuesdays 10:20 to 12:10; Fridays 10:20 to 12:10
Room: NTU Lecture Hall, Multi-Purpose Classroom Building with syncronized live lecture in NTU CSIE Building, R103 if too many people are in the room
Textbook: Learning from Data, by Yaser Abu-Mostafa, Malik Magdon-Ismail and Hsuan-Tien Lin
Language: Mandarin teaching with English slides
Grading:
- Machine Learning Foundations: 100% homework by homework sets 1-4 (tentative)
- Machine Learning Techniques: 50% homework by homework sets 5-6, 50% project (tentative)
screencast link: https://www.csie.ntu.edu.tw/~htlin/ml20fall/screencast.php
sign-up: Thanks to the generous support of NTU EECS College and Office of Academic Affairs for TA resources, we can add everyone to any of the two classes. We will start distributing the selection code 10 minutes prior to the 9/18 Friday class.

Announcements

homework 6 announced on 12/25/2020, due 01/15/2021
homework 5 announced on 12/04/2020, due 12/25/2020
final project announced on 11/29/2020, due 01/19/2021
homework 4 partially announced on 11/13/2020, fully announced on 11/16/2020, due 12/04/2020
homework 3 announced on 10/30/2020, due 11/20/2020
homework 2 extended to 11/06/2020
homework 2 announced on 10/16/2020, due 10/30/2020
homework 1 announced on 09/25/2020, due 10/16/2020
homework 0 slightly updated for clarification on 09/20/2020
homework 0 announced on 09/15/2020, self-graded

Class Policy

main policy

Course Plan (tentative)

Machine Learning Foundations

date	syllabus	todo/done	suggested reading
09/15	course introduction		course slides
09/18	topic 1: when can machines learn? the learning problem;		course slides; LFD 1.0, 1.1.1, 1.2.4
09/22	learning to answer yes/no		course slides; LFD 1.1.2, 3.1 ; course slides on machine learning and artificial intelligence
09/25	types of learning	homework 1 announced	notes about convergence of PLA ; course slides; LFD 1.2; LFD 1.3
09/26	no class on this make-up day because it will be ineffective to take four hours within two days
09/29	feasibility of learning		notes about proof of Hoeffding ; course slides; LFD 1.3
10/02	no class because of long-weekend of mid-autumn festival
10/06	topic 2: why can machines learn? training versus testing		course slides; LFD 2.0, 2.1.1
10/09	no class because of long-weekend of double-ten holiday		suggested watching: Theory of Generalization :: Restriction of Break Point Theory of Generalization :: Bounding Function: Basic Cases Theory of Generalization :: Bounding Funciton: Inductive Cases Theory of Generalization :: A Pictorial Proof suggested reading: course slides; LFD 2.0, 2.1.1
10/13	the VC dimension		course slides; LFD 2.2
10/16	noise and error	homework 1 due; homework 2 announced	course slides; LFD 1.4
10/20	topic 3: how can machines learn? linear regression		course slides; LFD 3.2
10/23	logistic regression		course slides; LFD 3.3
10/27	linear models for classification		course slides; LFD 3.3 (for SGD part only)
10/30	nonlinear transformation	homework 2 due; homework 3 announced	course slides; LFD 3.4
11/03	topic 4: how can machines learn better? hazard of overfitting		course slides; LFD 4.0, 4.1
11/06	regularization		course slides; LFD 4.2
11/10	validation		course slides; LFD 4.3
11/13	three learning principles; machine learning for modern artificial intelligence	homework 3 due; homework 4 announced	course slides; LFD 5; course slides

Machine Learning Techniques

date	syllabus	todo/done	suggested reading
11/17	topic 4: how can machines learn by embedding numerous features? linear support vector machine		course slides; LFD e-8.1
11/20	no class because of NTU sports day
11/24	dual support vector machine; kernel support vector machine		course slides; LFD e-8.2; course slides; LFD e-8.3
11/27	soft-margin support vector machine	homework 4 due; homework 5 announced	course slides; LFD e-8.4
12/01	topic 7: how can machines learn by distilling hidden features? neural network		course slides; LFD e-7.1, e-7.2, e-7.3, e-7.4 (selected parts)
12/04	matrix factorization		course slides; extended reading: Matrix Factorization Techniques for Recommender Systems (Koren, Bell and Folinsky)
12/08	guest lecture: Machine Learning and Data in Big Tech Companies by Dr. Scott Chen
12/11	no class because of NeurIPS 2020
12/15	neural networks, matrix factorization (unfinished parts)
12/18	topic 2: how can machines learn by combining predictive features? blending and bagging		course slides; extended reading: A linear ensemble of individual and blended models for music rating prediction (Chen et al.) Bagging predictors (Breiman)
12/22	adaptive boosting		course slides; extended reading: A short introduction to boosting (Freund and Schapire)
12/25	decision tree (selected) and random forest (selected)	homework 5 due; homework 6 announced	course slides; course slides; extended reading: Classification and regression trees (overview of decision tree by Loh) Classification and regression trees (book of CART by Breiman et al.) Random forest (Breiman)
12/29	gradient boosted decision tree; deep learning basics (selected)		course slides; course slides; LFD e-7.6 extended reading: Greedy Function Approximation: A Gradient Boosting Machine (Friedman)
01/01	no class because of long-weekend of new year's day
01/05	modern deep learning: activation		course slides; extended reading: ReLU: Deep sparse rectifier neural networks (Glorot, Bordes and Bengio) leaky ReLU: Rectifier Nonlinearities Improve Neural Network Acoustic Models (Maas, Hannun and Ng) parametric ReLU: Delving Deep into Rectifiers: Surpassing Human-Level Performance on Image Net Classification (He, Zhang, Ren and Sun) Dive into Deep Learning Section 4.1.2
01/08	modern deep learning: initialization, optimization, regularization		course slides; extended reading: ReLU: Deep sparse rectifier neural networks (Glorot, Bordes and Bengio) leaky ReLU: Rectifier Nonlinearities Improve Neural Network Acoustic Models (Maas, Hannun and Ng) parametric ReLU: Delving Deep into Rectifiers: Surpassing Human-Level Performance on Image Net Classification (He, Zhang, Ren and Sun) Dive into Deep Learning Section 4.1.2 course slides; extended reading: Gloret initialization: Understanding the difficulty of training deep feedforward neural networks (Gloret and Bengio) He initialization: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (He et al.) backprop and momentum: Learning representations by back-propagating errors (Rumelhart, Hinton, and Williams); On the Momentum Term in Gradient Descent Learning Algorithms (Qian) adam: Adam: A Method for Stochastic Optimization (Kingma and Ba) Dive into Deep Learning Sections 4.8, 11.6, 11.8, 11.10 course slides (not taught); extended reading: Dropout: A Simple Way to Prevent Neural Networks from Overfitting (Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov) Dive into Deep Learning Sections 4.6
01/12	finale and award ceremony		course slides
01/15	no class and winter vacation started (really?)	homework 6 due
01/19	no class and winter vacation started (really?)	final project due