Course syllabus and policies: Course handout.
Teaching staff:
- Instructor and office hours:
- Head TA: John Giorgi and Rex Ma
Contact emails:
- Support ticketing system: csc413-2023-01@cs.toronto.edu
- TA contact: csc413-2023-01-tas@cs.toronto.edu
- Emergency contact: csc413-2023-01-emerg@cs.toronto.edu
Please do not email the instructor or the TAs about the class directly to their personal accounts. Use the ticketing system for most of the requests, e.g., enrollment, coursework extensions, late submissions, etc.
Piazza: Students are encouraged to sign up Piazza to join course discussions. If your question is about the course material and doesn’t give away any hints for the homework, please post to Piazza so that the entire class can benefit from the answer.
Lecture and tutorial hours:
| Time | Location | |
|---|---|---|
| Lec0101/2001 Tutorial | Tues 1-3 pm Thur 1-2 pm | MP102 SS2117 |
| Lec2501/5101 Tutorial | Tues 6-8 pm Tues 8-9 pm | BA1190 & YouTube BA1190 & Zoom |
| Lec0201/2101 Tutorial (week after) | Thur 2-4 pm Tues 3-4 pm | SS2102 BA1180 |
| Lec2502/5201 Tutorial | Thur 6-8 pm Thur 8-9 pm | BA1170 BA1170 |
Online lectures and tutorials: The access to online lectures and tutorials will be communicated via course mailing list. Course videos and materials belong to your instructor, the University, and/or other sources depending on the specific facts of each situation, and are protected by copyright. Do not download, copy, or share any course or student materials or videos without the explicit permission of the instructor. For questions about recording and use of videos in which you appear please contact your instructor.
Accessibility: The CSC413 teaching team is fully committed to ensuring accessibility for all our students. For the students looking for additional academic accommodations or accessibility services registration, please visit www.accessibility.utoronto.ca. Students are encouraged to review the course syllabus at the beginning of a course and discuss questions regarding their accommodations for the course with their Accessibility Advisor. Once registered, students should send the Letter of Academic Accommodations to our ticketing system at csc413-2023-01@cs.toronto.edu as soon as possible by Friday, January 27th, 2023.
Waitlist and enrollment: CSC413 and CSC2516 always had long waiting lists for the last few years. The hard enrollment cap is determined by teaching resources available at the department level. Note that waitlists typically expire one week after the course starts. Once waitlists are removed, students are responsible for trying to enroll in the course on ACORN in a first-come, first-serve fashion. If you have further questions, please get in touch with CS undergrad office or CS graduate office.
Announcements:
- Jan 16: Assignment 1 handout and the starter code are now online. Make sure you create a copy in your own Drive before making edits, or else the changes will not be saved. Assignment 1 is due Feb 3rd.
- Jan 29: Assignment 2 handout and the starter code are now online. Make sure you create a copy in your own Drive before making edits, or else the changes will not be saved. Assignment 2 is due Feb 24th.
- Feb 2: Assignment 2 handout updated to Version 1.1
- Feb 9: Assignment 2 handout updated to Version 1.2
- Feb 14: Assignment 2 handout updated to Version 1.3
- Mar 4: Assignment 3 handout and the starter code nmt.ipynb, bert.ipynb, clip.ipynb are up. Assignment 3 is now due Mar 31st.
- Mar 17: Assignment 4 handout and the starter code gnn.ipynb, dqn.ipynb are up. Assignment 4 is now due Apr 11st.
- Mar 21: Assignment 4 handout updated to Version 1.1
Course Overview:
It is very hard to hand design programs to solve many real world problems, e.g. distinguishing images of cats v.s. dogs. Machine learning algorithms allow computers to learn from example data, and produce a program that does the job. Neural networks are a class of machine learning algorithm originally inspired by the brain, but which have recently have seen a lot of success at practical applications. They’re at the heart of production systems at companies like Google and Facebook for image processing, speech-to-text, and language understanding. This course gives an overview of both the foundational ideas and the recent advances in neural net algorithms.
Assignments:
| Handout | Due | |
|---|---|---|
| Assignment 1 | pdf, starter code(make a copy in your own Drive) | Jan. 15(out), due Feb. 3 |
| Assignment 2 | pdf, starter code(make a copy in your own Drive) | Jan. 29(out), due Feb. 24 |
| Assignment 3 | pdf, nmt.ipynb, bert.ipynb, clip.ipynb(make a copy in your own Drive) | |
| Assignment 4 | pdf, gnn.ipynb, dqn.ipynb | |
| Course Project | handout |
Lateness and grace days: Every student has a total of 7 grace days to extend the coursework deadlines through the semester. Each grace day allows for a 24 hours deadline extension without late penalty. That is, you may apply the grace days on a late submission to remove its late penalty. The maximum extension you may request is up to the remaining grace days you have banked up. We will keep track of the grace days on MarkUs. After the grace period, assignments will be accepted up to 3 days late, but 10% will be deducted for each day late, rounded up to the nearest day. After that, submissions will not be accepted and will receive a score of 0.
Midterm Online Quiz: Feb. 9
The midterm online quiz will cover the lecture materials up to Lecture 4 and Assignment 2 (written part only). The quiz will be hosted on Quecus for 24 hours. The exact details will be announced soon.
Calendar:
Suggested readings included help you understand the course material. They are not required, i.e. you are only responsible for the material covered in lecture. Most of the suggested reading listed are more advanced than the corresponding lecture, and are of interest if you want to know where our knowledge comes from or follow current frontiers of research.
| Date | Topic | Slides | Suggested Readings | |
|---|---|---|---|---|
| Lecture 1 | Jan 10 | Introduction & Linear Models | Slides | Roger Grosse’s notes: Linear Regression, Linear Classifiers, Training a Classifier |
| Tutorial 1 | Jan 10 | Multivariable Calculus Review | ipynb | iPython notebook: ipynb, you may view the notebook via Colab. |
| Lecture 2 | Jan 17 | Multilayer Perceptrons & Backpropagation | Slides | Roger Grosse’s notes: Multilayer Perceptrons, Backpropagation |
| Tutorial 2 | Jan 17 | Autograd and PyTorch | ipynb | iPython notebook: ipynb, you may view the notebook via Colab. |
| Lecture 3 | Jan 24 | Distributed Representations & Optimization | Slides | Roger Grosse’s notes: Automatic Differentiation, Distributed Representations, Optimization |
| Tutorial 3 | Jan 24 | How to Train Neural Networks | Slides, ipynb | iPython notebook: ipynb, you may view the notebook via Colab. |
| Lecture 4 | Jan 31 | Convolutional Neural Networks and Image Classification | Slides | Roger Grosse’s notes: ConvNets, Image Classification. Related papers: Yann LeCun’s 1998 LeNet, AlexNet. |
| Tutorial 4 | Jan 31 | Convolutional Neural Networks | ipynb | iPython notebook: ipynb, you may view the notebook via Colab. |
| Lecture 5 | Feb 07 | Optimization & Generalization | Slides | Roger Grosse’s notes: Generalization, Exploding Vanishing Gradients. Related papers: Dropout, ResNet |
| Tutorial 5 | Feb 07 | How to Write a Good Course Project Report | Slides | |
| Midterm Quiz | Feb 09 | |||
| Lecture 6 | Feb 14 | Interpretability and Large-scale Generative Models | Slides | Related papers: Sanity Check for Saliency Maps, SmoothGrad, Towards a rigorous science of interpretable machine learning, Batch size scaling, Model scaling |
| Tutorial 6 | Feb 14 | Best Practices of ConvNet Applications | Slides | |
| Reading Week | Feb 21 | |||
| Lecture 7 | Feb 28 | Recurrent Neural Networks and Attention | Slides | Roger Grosse’s notes: RNNs, Exploding Vanishing Gradients. Related papers: LSTM, ResNet, Neural machine translation, Show, attend and tell |
| Tutorial 7 | Feb 28 | Recurrent Neural Networks | ipynb | iPython notebook: ipynb, you may view the notebook via Colab. |
| Lecture 8 | Mar 07 | Transformers and Autoregressive Models | Slides | Related papers: Transformers, BERT pre-training, PixelRNNs, WaveNet, PixelCNNs. |
| Tutorial 8 | Mar 07 | NLP and Transformers | (Google) Slides | |
| Lecture 9 | Mar 14 | Large Language Models | Slides | |
| Tutorial 9 | Mar 14 | Large Language Models | Slides | |
| Lecture 10 | Mar 21 | Generative Models of Images and Graph Neural Networks | Slides | |
| Tutorial 10 | Mar 21 | Applications of Deep Generative Models | (Google) Slides | |
| Lecture 11 | Mar 28 | Q-learning & the Game of Go | Slides | Related papers: Deep Q Network, AlphaGo, AlphaZero. |
| Tutorial 11 | Mar 28 | Reinforcement Learning | Slides | |
| Lecture 12 | Apr 04 | Recent Trends in Deep Learning |
Resource:
| Type | Name | Description |
|---|---|---|
| Related Textbooks | Deep Learning (Goodfellow at al., 2016) | The Deep Learning textbook is a resource intended to help students and practitioners enter the field of machine learning in general and deep learning. |
| Information Theory, Inference, and Learning Algorithms (MacKay, 2003) | A good introduction textbook that combines information theory and machine learning. | |
| General Framework | PyTorch | An open source deep learning platform that provides a seamless path from research prototyping to production deployment. |
| Computation Platform | Colab | Colaboratory is a free Jupyter notebook environment that requires no setup and runs entirely in the cloud. |
| GCE | Google Compute Engine delivers virtual machines running in Google’s innovative data centers and worldwide fiber network. | |
| AWS-EC2 | Amazon Elastic Compute Cloud (EC2) forms a central part of Amazon.com’s cloud-computing platform, Amazon Web Services (AWS), by allowing users to rent virtual computers on which to run their own computer applications. |