Security and Privacy of Machine Learning, Spring 2020

Thursdays 2:20 - 5:20pm, CSIE Building, R110

Instructor: Shang-Tse Chen
  • Email: stchen at
  • Office hour: after classes, or by appointment
Modern machine learning models has reached and even surpassed human performance in many areas. However, many of the successful cases only hold in clean and controlled settings, which can be far from real scenarios. This course will introduce you to potential vulnerabilities of ML models. We will design and implement various attacks during model training and testing phases, as well as methods to make ML models more robust. We will also cover other important aspects of ML, including privacy and fairness.

Course Schedule Evolving

We will use NTU COOL for slides, homework assignments, announcement, and discussion.

Date Topics Reading Note
3/5 * Course introduction
* Evasion attacks (i.e., adversarial examples)
* Intriguing properties of neural networks
* Explaining and harnessing adversarial examples
* Towards Evaluating the Robustness of Neural Networks
3/12 * Empirical defenses to evasion attacks
* Towards Deep Learning Models Resistant to Adversarial Attacks
* Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
HW1 out
3/19 * Theoretical analysis of adversarial examples
* Adversarially Robust Generalization Requires More Data
* Robustness May Be at Odds with Accuracy
* Adversarial Examples Are Not Bugs, They Are Features
* Adversarial examples from computational constraints
3/26 * Certified Defenses
* Evaluating robustness of neural networks with mixed integer programming
* Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability

Presentation: Attack other classifiers (SVM, decision tree, etc)
  * Evasion Attacks Against Machine Learning at Test Time
  * Robust Decision Trees Against Adversarial Examples
  * Analyzing the Robustness of Nearest Neighbors to Adversarial Examples
4/2 Spring break  
4/9 * Certified Defenses
* Provable defenses against adversarial examples via the convex outer adversarial polytope
* Certified Adversarial Robustness via Randomized Smoothing
Presentation: Transferability / Black-box attack
  * Delving into Transferable Adversarial Examples and Black-box Attacks
  * The Space of Transferable Adversarial Examples
  * Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
4/16 * Poisoning attacks
* Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks
* Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners
Presentation: Attack against generative models / reinforcement learning
  * Adversarial examples for generative models
  * Rob-GAN: Generator, Discriminator, and Adversarial Attacker
  * Adversarial Attacks on Neural Network Policies
  * Adversarial Policies: Attacking Deep Reinforcement Learning
HW1 due
4/23 * Robust statistics
Presentation: Adversarial attack & defense beyond images (e.g., NLP, audio)
  * Audio Adversarial Examples: Targeted Attacks on Speech-to-Text
  * Devil’s Whisper: A General Approach for Physical Adversarial Attacks against Commercial Black-box Speech Recognition Devices
  * HotFlip: White-Box Adversarial Examples for Text Classification
  * Adversarial Examples for Evaluating Reading Comprehension Systems
Final project proposal due
4/30 * Robust statistics Presentation: Detection of adversarial examples
  * Detecting Adversarial Samples from Artifacts
  * NIC: Detecting Adversarial Samples with Neural Network Invariant Checking
  * FakeSpotter: A Simple yet Robust Baseline for Spotting AI-Synthesized Fake Faces
5/7 * Confidentiality of ML models
* Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
* Stealing Machine Learning Models via Prediction APIs
Presentation: Adversarial attack beyond L_p constraints
  * Adversarial Patch
  * Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifiers
  * Generating Semantic Adversarial Examples with Differentiable Rendering
  * AT-GAN: An Adversarial Generator Model for Non-constrained Adversarial Examples
HW2 out
5/14 * Differential privacy
* section 1 of this tutorial
Presentation: Model interpretation / visualization
  * Feature Visualization
  * Differentiable Image Parameterizations
  * Interpreting Adversarially Trained Convolutional Neural Networks
  * Massif: Interactive Interpretation of Adversarial Attacks on Deep Learning
5/21 * Differential privacy
* Deep Learning with Differential Privacy
* Scalable Private Learning with PATE
Presentation: Attack against unsupervised methods (e.g., dimension reduction / clustering)
  * Is Data Clustering in Adversarial Settings Secure
  * Suspicion-Free Adversarial Attacks on Clustering Algorithms
  * Adversarial Attacks on Node Embeddings via Graph Poisoning
5/28 * Differential privacy
Presentation: Federated learning
  * Advances and Open Problems in Federated Learning
  * Analyzing Federated Learning through an Adversarial Lens
  * Deep Leakage from Gradients
HW2 due
6/4 * Fairness
Presentation: Connection between adversarial robustness and differential privacy
  * Certified Robustness to Adversarial Examples with Differential Privacy
  * A unified view on differential privacy and robustness to adversarial examples
6/11 * Final project presentation
6/18 * Final project presentation
6/25 Summer vacation starts!
Final project report due


  • There will be two homework assignments.
  • You will need to do some programming with standard deep learning libraries (e.g., PyTorch, Tensorflow).
  • We will most likely use a mid-sized dataset, such as CIFAR-10. It will be easier for you if you have access to GPUs.
  • Try applying for AWS education grant if you need computing resources.

Paper Presentation

  • A group of students (size TBD based on class size) will present and lead the discussion on an extended topic related to this course.
  • The presentation including QA should be within 50 minutes.
  • Topics and related papers will be announced soon here.

Reading Critique

  • Choose a paper from the suggested reading list and write a 1-page summary.
  • The summary should address the strength and weakness of the paper and questions you have about the paper.
  • The summary is due at noon before each class, starting from the 2nd week.
  • You only need to turn in 10 summaries in total.
  • Please ask the questions you have to me or the student speaker of that week in class.
  • Student speaker of that week do not need to write the summary.

Final Project

  • You will work in groups on a topic related to this course.
  • Example project format:
    • Tackle an open problem (not necessarily need to be successful)
    • Improve algorithms in a paper with techniques that you come up with
    • Apply the techniques you learned to novel applications
    • Benchmark algorithms from multiple papers and get some insights
    • Literature survey of some related areas that we did not cover
  • You need to turn in a 1-page proposal by 4/23.
  • Presentation should be similar to a conference talk (25 minutes presentation + 5 minites QA).
  • Final report should be typed with Latex (with NeurIPS format) and no more than 6 pages.

Grading Policy

  • Homework: 30% (15% x 2)
  • Reading critique: 10%
  • Paper presentation: 20%
  • Project: 40%
    • Proposal (5%)
    • Presentation (15%)
    • Final report (20%)
  • All due times are at noon.
    • No late submission is accepted.
    • Exception: you email Shang-Tse and get the approval before the deadline.