Security and Privacy of Machine Learning, Fall 2020

Fridays 9:10am - 12:10pm, CSIE Building, R105

Instructor: Shang-Tse Chen

Email: stchen at csie.ntu.edu.tw
Office hour: after classes, or by appointment

TA: De-Yang Hong

Email: r08922097 at ntu.edu.tw
Office hour: Fridays 2-3pm, CSIE Building R340

Modern machine learning models has reached and even surpassed human performance in many areas. However, many of the successful cases only hold in clean and controlled settings, which can be far from real scenarios. This course will introduce you to potential vulnerabilities of ML models. We will design and implement various attacks during model training and testing phases, as well as methods to make ML models more robust. We will also cover other important aspects of ML, including privacy and fairness.

Course Schedule Evolving

We will use NTU COOL for slides, homework assignments, announcement, and discussion.

Date	Topics	Reading	Note
9/18	* Course introduction * Evasion attacks (i.e., adversarial examples)	* Intriguing properties of neural networks * Explaining and harnessing adversarial examples * Towards Evaluating the Robustness of Neural Networks	slides
9/25	* Empirical defenses to evasion attacks	* Towards Deep Learning Models Resistant to Adversarial Attacks * Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples	slides HW1 out
10/2	Holiday; no class
10/9	Holiday; no class
10/16	* Theoretical analysis of adversarial examples	* Adversarially Robust Generalization Requires More Data * Robustness May Be at Odds with Accuracy * Adversarial Examples Are Not Bugs, They Are Features * Adversarial examples from computational constraints Presentation: Attack other classifiers (SVM, decision tree, etc) * Evasion Attacks Against Machine Learning at Test Time * Robust Decision Trees Against Adversarial Examples * Analyzing the Robustness of Nearest Neighbors to Adversarial Examples	slides
10/23	* Certified Defenses	* Evaluating robustness of neural networks with mixed integer programming * Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability Presentation: Transferability / Black-box attack * Delving into Transferable Adversarial Examples and Black-box Attacks * The Space of Transferable Adversarial Examples * Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples	slides HW1 due HW2 out
10/30	* Certified Defenses	* Provable defenses against adversarial examples via the convex outer adversarial polytope * Certified Adversarial Robustness via Randomized Smoothing * Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks Presentation: Attack against generative models / reinforcement learning * Adversarial examples for generative models * Rob-GAN: Generator, Discriminator, and Adversarial Attacker * Adversarial Attacks on Neural Network Policies * Adversarial Policies: Attacking Deep Reinforcement Learning	slides
11/6	* Poisoning attacks & defenses	* Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners * Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness * Spectral Signatures in Backdoor Attacks Presentation: Adversarial attack & defense beyond images (e.g., NLP, audio) * Audio Adversarial Examples: Targeted Attacks on Speech-to-Text * Devil’s Whisper: A General Approach for Physical Adversarial Attacks against Commercial Black-box Speech Recognition Devices * HotFlip: White-Box Adversarial Examples for Text Classification * Adversarial Examples for Evaluating Reading Comprehension Systems	slides Final project proposal due
11/13	* Student presentations	Presentation: Detection of adversarial examples * Detecting Adversarial Samples from Artifacts * NIC: Detecting Adversarial Samples with Neural Network Invariant Checking * FakeSpotter: A Simple yet Robust Baseline for Spotting AI-Synthesized Fake Faces Presentation: Adversarial attack beyond L_p constraints * Adversarial Patch * Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifiers * Generating Semantic Adversarial Examples with Differentiable Rendering * AT-GAN: An Adversarial Generator Model for Non-constrained Adversarial Examples Presentation: Model interpretation / visualization * Feature Visualization * Differentiable Image Parameterizations * Interpreting Adversarially Trained Convolutional Neural Networks * Massif: Interactive Interpretation of Adversarial Attacks on Deep Learning
11/20	* NTU sports day; no class		HW2 due
11/27	* Confidentiality of ML models	* Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures * Stealing Machine Learning Models via Prediction APIs	slides
12/4	* Differential privacy	* section 1 of this tutorial * Deep Learning with Differential Privacy * Scalable Private Learning with PATE Presentation: Attack against unsupervised methods (e.g., dimension reduction / clustering) * Is Data Clustering in Adversarial Settings Secure * Suspicion-Free Adversarial Attacks on Clustering Algorithms * Adversarial Attacks on Node Embeddings via Graph Poisoning	slides
12/11	* Differential privacy	* Auditing Differentially Private Machine Learning: How Private is Private SGD? * Learning from Mixtures of Private and Public Populations * GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators Presentation: Federated learning * Advances and Open Problems in Federated Learning * Analyzing Federated Learning through an Adversarial Lens * Deep Leakage from Gradients	slides
12/18	* Fairness	* On Formalizing Fairness in Prediction with Machine Learning * Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification * Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting Presentation: Connection between adversarial robustness and differential privacy * Certified Robustness to Adversarial Examples with Differential Privacy * A unified view on differential privacy and robustness to adversarial examples	slides
12/25	* Final project presentation
1/1	Holiday; no class
1/8	* Final project presentation
1/15	Summer vacation starts!		Final project report due

Homework

There will be two homework assignments.
You will need to do some programming with standard deep learning libraries (e.g., PyTorch, Tensorflow).
We will most likely use a mid-sized dataset, such as CIFAR-10. It will be easier for you if you have access to GPUs.
Try applying for AWS education grant if you need computing resources.

Paper Presentation

A group of students (size TBD based on class size) will present and lead the discussion on an extended topic related to this course.
The presentation including QA should be within 50 minutes.

Reading Critique

Choose a paper from the suggested reading list and write a 1-page summary.
The summary should address the strength and weakness of the paper and questions you have about the paper.
The summary is due at noon before each class, starting from the 2nd week.
You only need to turn in 10 summaries in total.
Please ask the questions you have to me or the student speaker of that week in class.
Student speaker of that week do not need to write the summary.

Final Project

You will work in groups on a topic related to this course.
Example project format:

Tackle an open problem (not necessarily need to be successful)
Improve algorithms in a paper with techniques that you come up with
Apply the techniques you learned to novel applications
Benchmark algorithms from multiple papers and get some insights
Literature survey of some related areas that we did not cover

You need to turn in a 1-page proposal by 11/6.
Presentation should be similar to a conference talk (25 minutes presentation + 5 minites QA).
Final report should be typed with Latex (with NeurIPS format) and no more than 6 pages.

Grading Policy

Homework: 30% (15% x 2)
Reading critique: 10%
Paper presentation: 20%
Project: 40%

Proposal (5%)
Presentation (15%)
Final report (20%)

All due times are at noon.

No late submission is accepted.
Exception: you email Shang-Tse and get the approval before the deadline.