Optimization Methods for Deep Learning
Course Outline
Deep learning involves a difficult non-convex optimization
problem. The goal of this course is to study the implementation
of optimization methods for deep learning.
We will run this course in the following formats:
- lectures (by the instructor)
- project presentations (by students): we will have many.
For potential students: you want to make sure that you are
interested in optimization for deep learning.
We will heavily use the software
simpleNN
Among the various types of networks, we will pay more attention
to CNN.
You will get hands-on experiences in implementing a deep
learning code
This course was first offered at UCLA in winter 2019 (see course page here; 21 enrolled and 19 finished).
Slides, Reading list, and Projects
This section (and slides) will be continuously updated.
March 2
March 9
- Slides: we continue the discussion on optimization problems
March 16
March 23
- We continue our discussion on stochastic gradient methods
March 30
April 6
April 13
- Slides: we continue the discussion on implementation issues
April 20
April 27
May 4
- Project 4 presentation. Discussion
- We continue the discussion on automatic differentiation
May 11
May 18
May 25
June 1
- We continue the discussion on Gauss Newton matrix-vector product
June 8
Other possible projects
- math formulation of batch normalization
- try other networks
- implement vTP in mex
- Checking some details of Tensorflow
- SG vs LBFGS
- Different Newton implementations
- Repeat some exps in Wilson et al. (2017)
Exams
No exam
Grading
100% Projects.
Issues due to the Coronavirus outbreak
- We run the course online by a conferencing system.
- If you are sick, please do not come to the class.
- For course registration, you DON'T need my permission and can do it online by yourself.
Acknowledgements: the following people have greatly helped to prepare
materials for this course (including creating the software used for the course
and trying some projects).
Chien-Chih Wang, Kent Loong Tan, Pin-Yen Lin, Cheng-Hung Liu (former
and current members in my group), Pengrui Quan (UCLA), and Leonardo Galli (University of Florence)
Last modified: