CSC413/2516 Winter 2023
Neural Networks and Deep Learning


Course syllabus and policies: Course handout.

Teaching staff:

  • Instructor and office hours:

Contact emails:

Please do not email the instructor or the TAs about the class directly to their personal accounts. Use the ticketing system for most of the requests, e.g., enrollment, coursework extensions, late submissions, etc.

Piazza: Students are encouraged to sign up Piazza to join course discussions. If your question is about the course material and doesn’t give away any hints for the homework, please post to Piazza so that the entire class can benefit from the answer.

Lecture and tutorial hours:

  Time Location
Lec0101/2001
Tutorial
Tues 1-3 pm
Thur 1-2 pm
MP102
SS2117
Lec2501/5101
Tutorial
Tues 6-8 pm
Tues 8-9 pm
BA1190 & YouTube
BA1190 & Zoom
Lec0201/2101
Tutorial (week after)
Thur 2-4 pm
Tues 3-4 pm
SS2102
BA1180
Lec2502/5201
Tutorial
Thur 6-8 pm
Thur 8-9 pm
BA1170
BA1170

Online lectures and tutorials: The access to online lectures and tutorials will be communicated via course mailing list. Course videos and materials belong to your instructor, the University, and/or other sources depending on the specific facts of each situation, and are protected by copyright. Do not download, copy, or share any course or student materials or videos without the explicit permission of the instructor. For questions about recording and use of videos in which you appear please contact your instructor.

Accessibility: The CSC413 teaching team is fully committed to ensuring accessibility for all our students. For the students looking for additional academic accommodations or accessibility services registration, please visit www.accessibility.utoronto.ca. Students are encouraged to review the course syllabus at the beginning of a course and discuss questions regarding their accommodations for the course with their Accessibility Advisor. Once registered, students should send the Letter of Academic Accommodations to our ticketing system at csc413-2023-01@cs.toronto.edu as soon as possible by Friday, January 27th, 2023.

Waitlist and enrollment: CSC413 and CSC2516 always had long waiting lists for the last few years. The hard enrollment cap is determined by teaching resources available at the department level. Note that waitlists typically expire one week after the course starts. Once waitlists are removed, students are responsible for trying to enroll in the course on ACORN in a first-come, first-serve fashion. If you have further questions, please get in touch with CS undergrad office or CS graduate office.



Announcements:

  • Jan 16: Assignment 1 handout and the starter code are now online. Make sure you create a copy in your own Drive before making edits, or else the changes will not be saved. Assignment 1 is due Feb 3rd.

Course Overview:

It is very hard to hand design programs to solve many real world problems, e.g. distinguishing images of cats v.s. dogs. Machine learning algorithms allow computers to learn from example data, and produce a program that does the job. Neural networks are a class of machine learning algorithm originally inspired by the brain, but which have recently have seen a lot of success at practical applications. They’re at the heart of production systems at companies like Google and Facebook for image processing, speech-to-text, and language understanding. This course gives an overview of both the foundational ideas and the recent advances in neural net algorithms.


Assignments:

  Handout Due
Assignment 1 pdf, starter code(make a copy in your own Drive) Jan. 15(out), due Feb. 3
Assignment 2   Jan. 29(out), due Feb. 24
Assignment 3   Feb. 19(out), due Mar. 24
Assignment 4   Mar. 12(out), due Apr. 07
Course Project handout due Apr. 14


Lateness and grace days: Every student has a total of 7 grace days to extend the coursework deadlines through the semester. Each grace day allows for a 24 hours deadline extension without late penalty. That is, you may apply the grace days on a late submission to remove its late penalty. The maximum extension you may request is up to the remaining grace days you have banked up. We will keep track of the grace days on MarkUs. After the grace period, assignments will be accepted up to 3 days late, but 10% will be deducted for each day late, rounded up to the nearest day. After that, submissions will not be accepted and will receive a score of 0.

Midterm Online Quiz: Feb. 9

The midterm online quiz will cover the lecture materials up to Lecture 4 and Assignment 2 (written part only). The quiz will be hosted on Quecus for 24 hours. The exact details will be announced soon.


Calendar:

Suggested readings included help you understand the course material. They are not required, i.e. you are only responsible for the material covered in lecture. Most of the suggested reading listed are more advanced than the corresponding lecture, and are of interest if you want to know where our knowledge comes from or follow current frontiers of research.

  Date     Topic Slides Suggested Readings
Lecture 1 Jan 10 Introduction & Linear Models Slides Roger Grosse’s notes: Linear Regression, Linear Classifiers, Training a Classifier
Tutorial 1 Jan 10 Multivariable Calculus Review ipynb iPython notebook: ipynb, you may view the notebook via Colab.
Lecture 2 Jan 17 Multilayer Perceptrons & Backpropagation Slides Roger Grosse’s notes: Multilayer Perceptrons, Backpropagation
Tutorial 2 Jan 17 Autograd and PyTorch ipynb iPython notebook: ipynb, you may view the notebook via Colab.
Lecture 3 Jan 24 Distributed Representations & Optimization Slides Roger Grosse’s notes: Automatic Differentiation, Distributed Representations, Optimization
Tutorial 3 Jan 24 How to Train Neural Networks Slides, ipynb iPython notebook: ipynb, you may view the notebook via Colab.
Lecture 4 Jan 31 Convolutional Neural Networks and Image Classification    
Tutorial 4 Jan 31 Convolutional Neural Networks    
Lecture 5 Feb 07 Interpretability and Large-scale Generative Models    
Tutorial 5 Feb 07 How to Write a Good Course Project Report    
Midterm Quiz Feb 09      
Lecture 6 Feb 14 Optimization & Generalization    
Tutorial 6 Feb 14 Best Practices of ConvNet Applications    
Reading Week Feb 21      
Lecture 7 Feb 28 Recurrent Neural Networks and Attention    
Tutorial 7 Feb 28 Recurrent Neural Networks    
Lecture 8 Mar 07 Transformers and Autoregressive Models    
Tutorial 8 Mar 07 NLP and Transformers    
Lecture 9 Mar 14 Large Language Models    
Tutorial 9 Mar 14 Large Language Models    
Lecture 10 Mar 21 Generative Models of Images and Graph Neural Networks    
Tutorial 10 Mar 21 GANs, VAEs and Diffusion Models    
Lecture 11 Mar 28 Q-learning & the Game of Go    
Tutorial 11 Mar 28 Policy Gradient and Reinforcement Learning    
Lecture 12 Apr 04 Recent Trends in Deep Learning    

Resource:

Type Name Description
Related Textbooks Deep Learning (Goodfellow at al., 2016) The Deep Learning textbook is a resource intended to help students and practitioners enter the field of machine learning in general and deep learning.
  Information Theory, Inference, and Learning Algorithms (MacKay, 2003) A good introduction textbook that combines information theory and machine learning.
General Framework PyTorch An open source deep learning platform that provides a seamless path from research prototyping to production deployment.
Computation Platform Colab Colaboratory is a free Jupyter notebook environment that requires no setup and runs entirely in the cloud.
  GCE Google Compute Engine delivers virtual machines running in Google’s innovative data centers and worldwide fiber network.
  AWS-EC2 Amazon Elastic Compute Cloud (EC2) forms a central part of Amazon.com’s cloud-computing platform, Amazon Web Services (AWS), by allowing users to rent virtual computers on which to run their own computer applications.