Course syllabus and policies: Course handout.
Teaching staff:
- Instructor and office hours:
- Head TA: Harris Chan and John Giorgi
Contact emails:
- Instructor: csc413-2022-01@cs.toronto.edu
- TAs and instructor: csc413-2022-01-tas@cs.toronto.edu
Please do not send the instructor or the TAs email about the class directly to their personal accounts.
Piazza: Students are encouraged to sign up Piazza to join course discussions. If your question is about the course material and doesn’t give away any hints for the homework, please post to Piazza so that the entire class can benefit from the answer.
Lecture and tutorial hours:
| Time | Location | |
|---|---|---|
| Lecture | Tuesday 6-8 pm | YouTube |
| Tutotiral | Tuesday 8-9 pm | Zoom |
Online lectures and tutorials: The access to online lectures and tutorials will be communicated via course mailing list. Course videos and materials belong to your instructor, the University, and/or other sources depending on the specific facts of each situation, and are protected by copyright. Do not download, copy, or share any course or student materials or videos without the explicit permission of the instructor. For questions about recording and use of videos in which you appear please contact your instructor.
Announcements:
- Mar 26: Programming Assignment 4 handout and the starter code (a4_dcgan.ipynb, a4_GCN.ipynb and a4_dqn.ipynb) are now online. Make sure you create a copy in your own Drive before making edits, or else the changes will not be saved.
- Mar 1: Homework 4 handout is released, due April 1st.
- Mar 1: Homework 3 handout updated to Version 1.1.
- Feb 27: Programming Assignment 3 handout and the starter code (nmt.ipynb, bert.ipynb and clip.ipynb) are now online. Make sure you create a copy in your own Drive before making edits, or else the changes will not be saved.
- Feb 6: Homework 3 handout is due Mar 11th.
- Feb 6: Homework 3 handout is released.
- Feb 11: Programming Assignment 2 handout and the starter code updated to Version 1.4.
- Feb 6: Homework 2 handout updated to Version 1.1.
- Feb 5: Programming Assignment 2 handout and the starter code are now online. Make sure you create a copy in your own Drive before making edits, or else the changes will not be saved.
- Feb 2: Programming Assignment 1 handout updated to Version 1.2.
- Jan 30: Programming Assignment 1 handout and the starter code updated to Version 1.1.
- Jan 29: Homework 2 handout is due Feb 11th.
- Jan 22: Programming Assignment 1 handout and the starter code are now online. Make sure you create a copy in your own Drive before making edits, or else the changes will not be saved.
- Jan 20: Homework 1 handout updated to Version 1.1.
- Jan 15: Homework 1 handout is due Jan 28th.
Course Overview:
It is very hard to hand design programs to solve many real world problems, e.g. distinguishing images of cats v.s. dogs. Machine learning algorithms allow computers to learn from example data, and produce a program that does the job. Neural networks are a class of machine learning algorithm originally inspired by the brain, but which have recently have seen a lot of success at practical applications. They’re at the heart of production systems at companies like Google and Facebook for image processing, speech-to-text, and language understanding. This course gives an overview of both the foundational ideas and the recent advances in neural net algorithms.
Assignments:
| Handout | Due | |
|---|---|---|
| Homework 1 | Jan. 15(out), due Jan. 28 | |
| Programming Assignment 1 | starter code(make a copy in your own Drive), pdf | Jan. 22(out), due Feb. 04 |
| Homework 2 | Jan. 29(out), due Feb. 11 | |
| Programming Assignment 2 | starter code (make a copy in your own Drive), pdf | Feb. 05(out), due Feb. 18 |
| Homework 3 | Feb. 19(out), due Mar. 11 | |
| Programming Assignment 3 | nmt.ipynb (make a copy in your own Drive), bert.ipynb, clip.ipynb, pdf | Feb. 26(out), due Mar. 18 |
| Homework 4 | Mar. 19(out), due Apr. 01 | |
| Programming Assignment 4 | a4_dcgan.ipynb, a4_GCN.ipynb, a4_dqn.ipynb (make a copy in your own Drive), pdf | Mar. 26(out), due Apr. 08 |
| Course Project | handout |
Midterm Online Quiz: Feb. 11
The midterm online quiz will cover the lecture materials up to lecture 4 and Homework 2. The quiz will be hosted on Quecus for 24 hours. The exact details will be announced soon.
Calendar:
Suggested readings included help you understand the course material. They are not required, i.e. you are only responsible for the material covered in lecture. Most of the suggested reading listed are more advanced than the corresponding lecture, and are of interest if you want to know where our knowledge comes from or follow current frontiers of research.
| Date | Topic | Slides | Suggested Readings | |
|---|---|---|---|---|
| Lecture 1 | Jan 11 | Introduction & Linear Models | Slides | Roger Grosse’s notes: Linear Regression, Linear Classifiers, Training a Classifier |
| Tutorial 1 | Jan 11 | Multivariable Calculus Review | ipynb | iPython notebook: ipynb, you may view the notebook via Colab. |
| Lecture 2 | Jan 18 | Multilayer Perceptrons & Backpropagation | Slides | Roger Grosse’s notes: Multilayer Perceptrons, Backpropagation |
| Tutorial 2 | Jan 18 | Autograd and PyTorch | ipynb | iPython notebook: ipynb, you may view the notebook via Colab. |
| Lecture 3 | Jan 25 | Distributed Representations & Optimization | Slides | Roger Grosse’s notes: Automatic Differentiation, Distributed Representations, Optimization |
| Tutorial 3 | Jan 25 | How to Train Neural Networks | Slides, code demo | iPython notebook: ipynb, you may view the notebook via Colab. |
| Lecture 4 | Feb 01 | Convolutional Neural Networks and Image Classification | Slides | Roger Grosse’s notes: ConvNets, Image Classification. Related papers: Yann LeCun’s 1998 LeNet, AlexNet. |
| Tutorial 4 | Feb 01 | Convolutional Neural Networks | ipynb | iPython notebook: ipynb, you may view the notebook via Colab |
| Lecture 5 | Feb 08 | Interpretability | Slides | Related papers: Sanity Check for Saliency Maps, SmoothGrad, Towards a rigorous science of interpretable machine learning. |
| Tutorial 5 | Feb 08 | How to Write a Good Course Project Report | Slides | |
| Midterm Quiz | Feb 11 | |||
| Lecture 6 | Feb 15 | Optimization & Generalization | Slides | Roger Grosse’s notes: Generalization, Exploding Vanishing Gradients. Related papers: Dropout, ResNet |
| Tutorial 6 | Feb 15 | Best Practices of ConvNet Applications | Slides | |
| Reading Week | Feb 22 | |||
| Lecture 7 | Mar 01 | Recurrent Neural Networks and Attention | Slides | Roger Grosse’s notes: RNNs, Exploding Vanishing Gradients. Related papers: LSTM, ResNet, Neural machine translation, Show, attend and tell.t |
| Tutorial 7 | Mar 01 | Recurrent Neural Networks | Colab | |
| Lecture 8 | Mar 08 | Transformers and Autoregressive Models | Slides | Related papers: Transformers, BERT pre-training, PixelRNNs, WaveNet, PixelCNNs. |
| Tutorial 8 | Mar 08 | NLP and Transformers | Slides | |
| Lecture 9 | Mar 15 | Reversible Models & Generative Adversarial Networks | Slides | Related papers: Generative adversarial networks, CycleGANs, RealNVP. |
| Tutorial 9 | Mar 15 | Information Theory | Slides | |
| Lecture 10 | Mar 22 | Generative Models & Reinforcement Learning | Slides | Related papers: RealNVP, Variational Auto-encoder, Policy Gradients for Robotics, Proximal Policy Optimization Algorithm. |
| Tutorial 10 | Mar 22 | Generative Adversarial Networks | Slides | |
| Lecture 11 | Mar 29 | Q-learning & the Game of Go | Slides | Related papers: Deep Q Network, AlphaGo, AlphaZero. |
| Tutorial 11 | Mar 29 | Policy Gradient and Reinforcement Learning | Slides, Colab | |
| Lecture 12 | Apr 05 | Recent Trends in Deep Learning |
Resource:
| Type | Name | Description |
|---|---|---|
| Related Textbooks | Deep Learning (Goodfellow at al., 2016) | The Deep Learning textbook is a resource intended to help students and practitioners enter the field of machine learning in general and deep learning. |
| Information Theory, Inference, and Learning Algorithms (MacKay, 2003) | A good introduction textbook that combines information theory and machine learning. | |
| General Framework | PyTorch | An open source deep learning platform that provides a seamless path from research prototyping to production deployment. |
| Computation Platform | Colab | Colaboratory is a free Jupyter notebook environment that requires no setup and runs entirely in the cloud. |
| GCE | Google Compute Engine delivers virtual machines running in Google’s innovative data centers and worldwide fiber network. | |
| AWS-EC2 | Amazon Elastic Compute Cloud (EC2) forms a central part of Amazon.com’s cloud-computing platform, Amazon Web Services (AWS), by allowing users to rent virtual computers on which to run their own computer applications. |