CSC413/2516 Winter 2023
Neural Networks and Deep Learning

Course syllabus and policies: Course handout.

Teaching staff:

  • Instructor and office hours:
  • Head TA: John Giorgi and Rex Ma

Contact emails:

Please do not email the instructor or the TAs about the class directly to their personal accounts. Use the ticketing system for most of the requests, e.g., enrollment, coursework extensions, late submissions, etc.

Piazza: Students are encouraged to sign up Piazza to join course discussions. If your question is about the course material and doesn’t give away any hints for the homework, please post to Piazza so that the entire class can benefit from the answer.

Lecture and tutorial hours:

  Time Location
Tues 1-3 pm
Thur 1-2 pm
Tues 6-8 pm
Tues 8-9 pm
BA1190 & YouTube
BA1190 & Zoom
Tutorial (week after)
Thur 2-4 pm
Tues 3-4 pm
Thur 6-8 pm
Thur 8-9 pm

Online lectures and tutorials: The access to online lectures and tutorials will be communicated via course mailing list. Course videos and materials belong to your instructor, the University, and/or other sources depending on the specific facts of each situation, and are protected by copyright. Do not download, copy, or share any course or student materials or videos without the explicit permission of the instructor. For questions about recording and use of videos in which you appear please contact your instructor.

Accessibility: The CSC413 teaching team is fully committed to ensuring accessibility for all our students. For the students looking for additional academic accommodations or accessibility services registration, please visit Students are encouraged to review the course syllabus at the beginning of a course and discuss questions regarding their accommodations for the course with their Accessibility Advisor. Once registered, students should send the Letter of Academic Accommodations to our ticketing system at as soon as possible by Friday, January 27th, 2023.

Waitlist and enrollment: CSC413 and CSC2516 always had long waiting lists for the last few years. The hard enrollment cap is determined by teaching resources available at the department level. Note that waitlists typically expire one week after the course starts. Once waitlists are removed, students are responsible for trying to enroll in the course on ACORN in a first-come, first-serve fashion. If you have further questions, please get in touch with CS undergrad office or CS graduate office.


  • Jan 16: Assignment 1 handout and the starter code are now online. Make sure you create a copy in your own Drive before making edits, or else the changes will not be saved. Assignment 1 is due Feb 3rd.
  • Jan 29: Assignment 2 handout and the starter code are now online. Make sure you create a copy in your own Drive before making edits, or else the changes will not be saved. Assignment 2 is due Feb 24th.
  • Feb 2: Assignment 2 handout updated to Version 1.1
  • Feb 9: Assignment 2 handout updated to Version 1.2
  • Feb 14: Assignment 2 handout updated to Version 1.3
  • Mar 4: Assignment 3 handout and the starter code nmt.ipynb, bert.ipynb, clip.ipynb are up. Assignment 3 is now due Mar 31st.
  • Mar 17: Assignment 4 handout and the starter code gnn.ipynb, dqn.ipynb are up. Assignment 4 is now due Apr 11st.
  • Mar 21: Assignment 4 handout updated to Version 1.1

Course Overview:

It is very hard to hand design programs to solve many real world problems, e.g. distinguishing images of cats v.s. dogs. Machine learning algorithms allow computers to learn from example data, and produce a program that does the job. Neural networks are a class of machine learning algorithm originally inspired by the brain, but which have recently have seen a lot of success at practical applications. They’re at the heart of production systems at companies like Google and Facebook for image processing, speech-to-text, and language understanding. This course gives an overview of both the foundational ideas and the recent advances in neural net algorithms.


  Handout Due
Assignment 1 pdf, starter code(make a copy in your own Drive) Jan. 15(out), due Feb. 3
Assignment 2 pdf, starter code(make a copy in your own Drive) Jan. 29(out), due Feb. 24
Assignment 3 pdf, nmt.ipynb, bert.ipynb, clip.ipynb(make a copy in your own Drive) Feb. 19(out), due Mar. 24 Mar. 04(out), due Mar. 31
Assignment 4 pdf, gnn.ipynb, dqn.ipynb Mar. 12(out), due Apr. 07 Mar. 17(out), due Apr. 11
Course Project handout due Apr. 14 due Apr. 19 (Peer review due Apr. 22)

Lateness and grace days: Every student has a total of 7 grace days to extend the coursework deadlines through the semester. Each grace day allows for a 24 hours deadline extension without late penalty. That is, you may apply the grace days on a late submission to remove its late penalty. The maximum extension you may request is up to the remaining grace days you have banked up. We will keep track of the grace days on MarkUs. After the grace period, assignments will be accepted up to 3 days late, but 10% will be deducted for each day late, rounded up to the nearest day. After that, submissions will not be accepted and will receive a score of 0.

Midterm Online Quiz: Feb. 9

The midterm online quiz will cover the lecture materials up to Lecture 4 and Assignment 2 (written part only). The quiz will be hosted on Quecus for 24 hours. The exact details will be announced soon.


Suggested readings included help you understand the course material. They are not required, i.e. you are only responsible for the material covered in lecture. Most of the suggested reading listed are more advanced than the corresponding lecture, and are of interest if you want to know where our knowledge comes from or follow current frontiers of research.

  Date     Topic Slides Suggested Readings
Lecture 1 Jan 10 Introduction & Linear Models Slides Roger Grosse’s notes: Linear Regression, Linear Classifiers, Training a Classifier
Tutorial 1 Jan 10 Multivariable Calculus Review ipynb iPython notebook: ipynb, you may view the notebook via Colab.
Lecture 2 Jan 17 Multilayer Perceptrons & Backpropagation Slides Roger Grosse’s notes: Multilayer Perceptrons, Backpropagation
Tutorial 2 Jan 17 Autograd and PyTorch ipynb iPython notebook: ipynb, you may view the notebook via Colab.
Lecture 3 Jan 24 Distributed Representations & Optimization Slides Roger Grosse’s notes: Automatic Differentiation, Distributed Representations, Optimization
Tutorial 3 Jan 24 How to Train Neural Networks Slides, ipynb iPython notebook: ipynb, you may view the notebook via Colab.
Lecture 4 Jan 31 Convolutional Neural Networks and Image Classification Slides Roger Grosse’s notes: ConvNets, Image Classification. Related papers: Yann LeCun’s 1998 LeNet, AlexNet.
Tutorial 4 Jan 31 Convolutional Neural Networks ipynb iPython notebook: ipynb, you may view the notebook via Colab.
Lecture 5 Feb 07 Optimization & Generalization Slides Roger Grosse’s notes: Generalization, Exploding Vanishing Gradients. Related papers: Dropout, ResNet
Tutorial 5 Feb 07 How to Write a Good Course Project Report Slides  
Midterm Quiz Feb 09      
Lecture 6 Feb 14 Interpretability and Large-scale Generative Models Slides Related papers: Sanity Check for Saliency Maps, SmoothGrad, Towards a rigorous science of interpretable machine learning, Batch size scaling, Model scaling
Tutorial 6 Feb 14 Best Practices of ConvNet Applications Slides  
Reading Week Feb 21      
Lecture 7 Feb 28 Recurrent Neural Networks and Attention Slides Roger Grosse’s notes: RNNs, Exploding Vanishing Gradients. Related papers: LSTM, ResNet, Neural machine translation, Show, attend and tell
Tutorial 7 Feb 28 Recurrent Neural Networks ipynb iPython notebook: ipynb, you may view the notebook via Colab.
Lecture 8 Mar 07 Transformers and Autoregressive Models Slides Related papers: Transformers, BERT pre-training, PixelRNNs, WaveNet, PixelCNNs.
Tutorial 8 Mar 07 NLP and Transformers (Google) Slides  
Lecture 9 Mar 14 Large Language Models Slides  
Tutorial 9 Mar 14 Large Language Models Slides  
Lecture 10 Mar 21 Generative Models of Images and Graph Neural Networks Slides  
Tutorial 10 Mar 21 Applications of Deep Generative Models (Google) Slides  
Lecture 11 Mar 28 Q-learning & the Game of Go Slides Related papers: Deep Q Network, AlphaGo, AlphaZero.
Tutorial 11 Mar 28 Reinforcement Learning Slides  
Lecture 12 Apr 04 Recent Trends in Deep Learning    


Type Name Description
Related Textbooks Deep Learning (Goodfellow at al., 2016) The Deep Learning textbook is a resource intended to help students and practitioners enter the field of machine learning in general and deep learning.
  Information Theory, Inference, and Learning Algorithms (MacKay, 2003) A good introduction textbook that combines information theory and machine learning.
General Framework PyTorch An open source deep learning platform that provides a seamless path from research prototyping to production deployment.
Computation Platform Colab Colaboratory is a free Jupyter notebook environment that requires no setup and runs entirely in the cloud.
  GCE Google Compute Engine delivers virtual machines running in Google’s innovative data centers and worldwide fiber network.
  AWS-EC2 Amazon Elastic Compute Cloud (EC2) forms a central part of’s cloud-computing platform, Amazon Web Services (AWS), by allowing users to rent virtual computers on which to run their own computer applications.