Optimization Methods for Deep Learning

This is a tentative page copied from the past course. Materials are being updated.

Instructor: Chih-Jen Lin, Room 413, CSIE Building.
The best way to contact me is via e-mails.
TA: ?? (email: ?? at ntu.edu.tw). TA hour: ??
Your HW/exam scores will be here
Time: Tuesday 10:20am-1pm
We will do two 10-minute breaks at around 11:10am and 12:10pm. The class ends at 1pm.

Place: room 101, CSIE building
FAQ of this course
We will pre-record most lectures and broadcast them in the class. In the class I will give additional comments while the video is being played.

Course Outline

Deep learning involves a difficult non-convex optimization problem. The goal of this course is to study the implementation of optimization methods for deep learning. We will run this course in the following formats:

lectures (by the instructor)
project/paper presentations (by students)

For potential students: you want to make sure that you are interested in optimization for deep learning.

Slides and recordings

This section (and slides) will be continuously updated.

Optimization problems for deep learning
- Linear classification ( part 1: slides video )
- Fully-connected networks ( part 1: slides video )
- Convolutional networks ( part 1: slides video part 2: slides video part 3: slides video part 4: slides video )
Stochastic gradient methods for deep learning
- Gradient descent ( part 1: slides video )
- Stochastic gradient methods
  - part 1: slides video: part 1 part 2
  - part 2: slides video: part 1 part 2
  - a note on different momentum update rules: slides
- Gradient calculation
  - Vector form ( part 1: slides video )
  - Gradient calculation ( part 1: slides video part 2: slides video part 3: slides video )
- Implementation
  - part 1: slides video
  - We will partially cover two sets of slides from the course "numerical methods" ( part 1: slides video part 2: slides video )
  - part 2: slides video 1 video 2 video 3
  - part 3: slides video
- Automatic differentiation
  - part 1: slides video
  - part 2: slides video
- Newton method
  - Basic ( part 1: slides video part 2: slides video )
  - Algorithms ( part 1: slides video part 2: slides video part 3: slides video )
  - Gauss Newton matrix-vector product
    - slides
    - Using only backward process ( videos: part1, part2, part3, part4, part5 )
    - Using forward and backward processes ( videos: part1, part2, part3 )
Projects
- Project 1: using a linear classifier to train the ?? set
- Project 2: using CNN to train the ?? set and do the profiling
- Project 3: Making our own implementation competitive with PyTorch.
- Project 1: simple experiments using SG methods. Presentation: March 15. Discussion.
- Project 2: more experiments on SG methods. Presentation: March 29. Discussion.
- Project 3: checking running time of major operations. Presentation: April 19. Discussion.
- Project 4: Making the MATLAB Implementation Competitive with Tensorflow. Presentation: May 10. Discussion.
- Project 5: An investigation of Python profilers. Presentation: June 7. Discussion.
- Project 6: Robustness of newton methods and running time analysis. Presentation: June 21. Discussion
Final Project
You must choose the project topic before the end of week 4 and let TAs know. See the following possible topics. If you want to do something else, you must discuss with the teacher first.
The final project is research oriented. We expect that you conduct deep investigation.
- A study on convergence results of stochastic gradient methods
  You may survey existing works and point out possible future directions. You may see an earlier course project report at UIUC as an example
- A study on autodiff in PyTorch
  In our lecture, we only introduced the basic concept of automatic differentiation. Can you investigate the implementation in PyTorch?
- A study on GPU for matrix-matrix products
  In our lecture, we discussed block algorithms for reducing the cost of memory access in matrix-matrix products. It's a general illustration. For GPU, some architectural properties make it very efficient for matrix-matrix products. Can you investigate what these properties are and conduct experiments to confirm them?
- A study on Adam and AdamW
  In our discussion we see many issues of Adam/AdamW remain to be investigated. For example, the original implementation in BERT does not include bias correction. Can you confirm the importance of bias correction? Then in the AdamW paper, it decouples the weight decay step. Can you design experiments to show that the setting in AdamW is better? Moreover, in our notes we mentioned an issue in the Hugging face implementation of AdamW. Is this implementation really "wrong"? Can the Hugging face implementation still work well in practice? Also how about the convergence?
- Newton for KimCNN
  From ?? (Wang's paper), Newton methods may not be able to handle networks with many layers. However, some applications need only a few layers. An example is the CNN for texts discussed in our project ??. Can you implement a Newton method and compare with stochastic gradient? (may need to discuss with Yaxu)
- BERT versus linear methods: In this talk (?? link to Bloomberg talk), for some document sets, linear methods are highly competitive with BERT. However, in ??, their linear results are worse than BERT on ?? sets. Can you conduct some investigation? (?? need to say more)
Course Schedule
The schedule is subject to change.
- Week 9: final project progress presentation
- Weeks 14-15: research paper presentation
- Week 16: final project presentation
Exams
No exam

Grading
100% Projects and presentations.

Issues related to COVID-19
- According to school's regulation, all students must wear masks
- If the covid situation becomes serious, we will move the course online.
- If you are sick, please do not come to the class.
Acknowledgements: we thank many people for helping to prepare materials for this course.
Last modified:

Optimization Methods for Deep Learning

Course Outline

Slides and recordings

Projects

Final Project

Course Schedule

Exams

Grading

Issues related to COVID-19