Security and Privacy of Machine Learning, Fall 2020

Fridays 9:10am - 12:10pm, CSIE Building, R105

Instructor: Shang-Tse Chen
  • Email: stchen at
  • Office hour: after classes, or by appointment
TA: De-Yang Hong
  • Email: r08922097 at
  • Office hour: Fridays 2-3pm, CSIE Building R340
Modern machine learning models has reached and even surpassed human performance in many areas. However, many of the successful cases only hold in clean and controlled settings, which can be far from real scenarios. This course will introduce you to potential vulnerabilities of ML models. We will design and implement various attacks during model training and testing phases, as well as methods to make ML models more robust. We will also cover other important aspects of ML, including privacy and fairness.

Course Schedule Evolving

We will use NTU COOL for slides, homework assignments, announcement, and discussion.

Date Topics Reading Note
9/18 * Course introduction
* Evasion attacks (i.e., adversarial examples)
* Intriguing properties of neural networks
* Explaining and harnessing adversarial examples
* Towards Evaluating the Robustness of Neural Networks
9/25 * Empirical defenses to evasion attacks
* Towards Deep Learning Models Resistant to Adversarial Attacks
* Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
HW1 out
10/2 Holiday; no class
10/9 Holiday; no class
10/16 * Theoretical analysis of adversarial examples
* Adversarially Robust Generalization Requires More Data
* Robustness May Be at Odds with Accuracy
* Adversarial Examples Are Not Bugs, They Are Features
* Adversarial examples from computational constraints
Presentation: Attack other classifiers (SVM, decision tree, etc)
  * Evasion Attacks Against Machine Learning at Test Time
  * Robust Decision Trees Against Adversarial Examples
  * Analyzing the Robustness of Nearest Neighbors to Adversarial Examples
10/23 * Certified Defenses
* Evaluating robustness of neural networks with mixed integer programming
* Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability
Presentation: Transferability / Black-box attack
  * Delving into Transferable Adversarial Examples and Black-box Attacks
  * The Space of Transferable Adversarial Examples
  * Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
HW1 due
HW2 out
10/30 * Certified Defenses
* Provable defenses against adversarial examples via the convex outer adversarial polytope
* Certified Adversarial Robustness via Randomized Smoothing
* Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks
Presentation: Attack against generative models / reinforcement learning
  * Adversarial examples for generative models
  * Rob-GAN: Generator, Discriminator, and Adversarial Attacker
  * Adversarial Attacks on Neural Network Policies
  * Adversarial Policies: Attacking Deep Reinforcement Learning
11/6 * Poisoning attacks & defenses
* Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners
* Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness
* Spectral Signatures in Backdoor Attacks
Presentation: Adversarial attack & defense beyond images (e.g., NLP, audio)
  * Audio Adversarial Examples: Targeted Attacks on Speech-to-Text
  * Devil’s Whisper: A General Approach for Physical Adversarial Attacks against Commercial Black-box Speech Recognition Devices
  * HotFlip: White-Box Adversarial Examples for Text Classification
  * Adversarial Examples for Evaluating Reading Comprehension Systems
Final project proposal due
11/13 * Student presentations
Presentation: Detection of adversarial examples
  * Detecting Adversarial Samples from Artifacts
  * NIC: Detecting Adversarial Samples with Neural Network Invariant Checking
  * FakeSpotter: A Simple yet Robust Baseline for Spotting AI-Synthesized Fake Faces
Presentation: Adversarial attack beyond L_p constraints
  * Adversarial Patch
  * Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifiers
  * Generating Semantic Adversarial Examples with Differentiable Rendering
  * AT-GAN: An Adversarial Generator Model for Non-constrained Adversarial Examples
Presentation: Model interpretation / visualization
  * Feature Visualization
  * Differentiable Image Parameterizations
  * Interpreting Adversarially Trained Convolutional Neural Networks
  * Massif: Interactive Interpretation of Adversarial Attacks on Deep Learning

11/20 * NTU sports day; no class
HW2 due
11/27 * Confidentiality of ML models
* Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
* Stealing Machine Learning Models via Prediction APIs
12/4 * Differential privacy
* section 1 of this tutorial
* Deep Learning with Differential Privacy
* Scalable Private Learning with PATE
Presentation: Attack against unsupervised methods (e.g., dimension reduction / clustering)
  * Is Data Clustering in Adversarial Settings Secure
  * Suspicion-Free Adversarial Attacks on Clustering Algorithms
  * Adversarial Attacks on Node Embeddings via Graph Poisoning
12/11 * Differential privacy
* Auditing Differentially Private Machine Learning: How Private is Private SGD?
* Learning from Mixtures of Private and Public Populations
* GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators
Presentation: Federated learning
  * Advances and Open Problems in Federated Learning
  * Analyzing Federated Learning through an Adversarial Lens
  * Deep Leakage from Gradients
12/18 * Fairness
* On Formalizing Fairness in Prediction with Machine Learning
* Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification
* Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
Presentation: Connection between adversarial robustness and differential privacy
  * Certified Robustness to Adversarial Examples with Differential Privacy
  * A unified view on differential privacy and robustness to adversarial examples
12/25 * Final project presentation
1/1 Holiday; no class
1/8 * Final project presentation
1/15 Summer vacation starts!
Final project report due


  • There will be two homework assignments.
  • You will need to do some programming with standard deep learning libraries (e.g., PyTorch, Tensorflow).
  • We will most likely use a mid-sized dataset, such as CIFAR-10. It will be easier for you if you have access to GPUs.
  • Try applying for AWS education grant if you need computing resources.

Paper Presentation

  • A group of students (size TBD based on class size) will present and lead the discussion on an extended topic related to this course.
  • The presentation including QA should be within 50 minutes.

Reading Critique

  • Choose a paper from the suggested reading list and write a 1-page summary.
  • The summary should address the strength and weakness of the paper and questions you have about the paper.
  • The summary is due at noon before each class, starting from the 2nd week.
  • You only need to turn in 10 summaries in total.
  • Please ask the questions you have to me or the student speaker of that week in class.
  • Student speaker of that week do not need to write the summary.

Final Project

  • You will work in groups on a topic related to this course.
  • Example project format:
    • Tackle an open problem (not necessarily need to be successful)
    • Improve algorithms in a paper with techniques that you come up with
    • Apply the techniques you learned to novel applications
    • Benchmark algorithms from multiple papers and get some insights
    • Literature survey of some related areas that we did not cover
  • You need to turn in a 1-page proposal by 11/6.
  • Presentation should be similar to a conference talk (25 minutes presentation + 5 minites QA).
  • Final report should be typed with Latex (with NeurIPS format) and no more than 6 pages.

Grading Policy

  • Homework: 30% (15% x 2)
  • Reading critique: 10%
  • Paper presentation: 20%
  • Project: 40%
    • Proposal (5%)
    • Presentation (15%)
    • Final report (20%)
  • All due times are at noon.
    • No late submission is accepted.
    • Exception: you email Shang-Tse and get the approval before the deadline.