Spring 2020 / DM873
Deep Learning

General Information

Machine learning has become a part in our everydays life, from simple product recommendations to personal electronic assistant to self-driving cars. More recently, through the advent of potent hardware and cheap computational power, “Deep Learning” has become a popular and powerful tool for learning from complex, large-scale data.

In this course, we will discuss the fundamentals of deep learning and its application to various different fields. We will learn about the power but also the limitations of these deep neural networks. At the end of the course, the students will have significant familiarity with the subject and will be able to apply the learned techniques to a broad range of different fields.

Mainly, the following topics will be covered:

  • feedforward neural networks
  • recurrent neural networks
  • convolutional neural networks
  • backpropagation algorithm
  • regularization
  • factor analysis

Lectures

# Date Content Slides Comments
1 Mon, 10.02.2020 Introduction here
2 Tue, 11.02.2020 Feed Forward Networks &
Linear Algebra
FF
LinAlg
DLB, Chap. 2
3 Mon, 17.02.2020 Feed Forward Networks Part 2 here DLB, Chap. 6
4 Tue, 18.02.2020 Feed Forward Networks Part 2 continuation
5 Mon, 24.02.2020 Statistics here DLB, Chap. 3
6 Tue, 25.02.2020 Machine Learning Basics here DLB, Chap. 5
7 Mon, 02.03.2020 Linear Regression here ISL, Chap. 2 & 3
8 Tue, 03.03.2020 Logistic Regression here ISL, Chap. 4.1-4.3
9 Mon, 09.03.2020 Regularization here
10 Mon, 16.03.2020 Convolution here
Video 0: Recap
Video 1: Motivation
Video 2: Convolution
Video 3: Pooling
Video 4: Varaints
Video 5: Efficient Convolution
Video 6: Keras
11 Tue, 17.03.2020 Gradient-Based Learning here
Video 1: Loss Functions Part 1
Video 2: Loss Functions Part 2
Video 3: Gradient Descent
Video 4: SGD
Video 5: Problems in Deep Learning
Video 6: Optimizers
12 Mon, 23.03.2020 Backpropagation here
Video 1: Backpropagation
Video 2: Extension to Vectors
13 Tue, 24.03.2020 Recurrent Networks here
Video 1: Introduction
Video 2: Network unfolding
Video 3: RNNs
Video 4: Leaky Units
Video 5: LSTM
Video 6: RNN in Keras
14 Mon, 30.03.2020 Auto Encoders here
Video 1: Introduction
Video 2: Undercomplete AE
Video 3: Regularized AE
15 Mon, 27.04.2020 Generative Networks slides
Video 1: Introduction
Video 2: VAE
Video 3: GAN
16 Mon, 04.05.2020 Advanced Networks slides
Video 1: Introduction
Video 2: Batch Normalization
Video 3: Style Transfer
Video 4: Style Networks
Video 5: Modern Networks
17 Mon, 18.05.2020

Exercises

# Date Questions Download Solutions
1 Wed, 26.02.2020 Introduction to Keras Intro
Exercises
Solution
2 Wed, 04.03.2020 Regression Exercises
Dataset
Solution
3 Tue, 10.03.2020 Classification Exercises
Keras 2
Solution
4 Wed, 18.03.2020 Additional Info
Convolution
Exercises Solution
Screeenshot
5 Wed, 25.03.2020 Advanced Keras Exercises
Advanced Keras
6 Wed, 15.04.2020 Q & A Session Video Data Generators
7 Wed, 29.04.2020
6 Tue, 28.04.2020 Moved to 15.04.2020
8 Tue, 05.05.2020
8 Tue, 19.05.2020 Moved to 29.04.2020

Project

Aim of the Project

In light of the recent COVID-19 pandemic, we will take a look at one of the severe symptoms of the disease, pneumonia. The dataset we have consists of x-ray images of patients who either have pneumonia or are healthy. You must make your own model to detect whether a patient has developed pneumonia, or is healthy.

To spice things up and to also give you deeper insights in KERAS, you are also supposed to implement your own layers and mix them with existing other layers. Make use of the upcoming Q&A sessions in case you are stuck and require help from Mathias.

You are supposed to work in groups of 4-5 people. In light of the ongoing lock-down, this might be a bit more challenging than usual. Nevertheless, it will help you since you can spread the tasks amongst the team members. In the case you have trouble finding members, please write Mathias to work out a solution.

Further Information

You will find all necessary information in the project description. It is fundamentally important that you start early on, since the models might take some time to train.

Deadline

  • Project Hand-In:
    Tuesday the 24th of May 2020 at 23:59.

    This includes:
    • Project report via Blackboard
    • Your code as nextcloud-link via email
    • Your model (as .h5 file) as nextcloud-link via email

Downloads

Procedure of the oral exam

Due to the ongoing Corona crisis, the exam will be conducted online via Zoom

The exam will last about 15-20 minutes. At the beginning, one topic from the list below will be drawn randomly using a random number generator. For each topic the examinee should be prepared to make a short presentation of 5 minutes (sharp). The examinee will have 2 minutes to briefly study the notes for the drawn topic before the presentation. The notes may be consulted during the presentation if needed but it will negatively influence the evaluation of the examinee's performance. The student can either use a presentation or a shared whiteboard for quick sketches, etc. (see technical details below).

After the short presentation, additional questions about the presentation's topic but also about other topics in the curriculum will be asked.

Technical Details and precise procedure:

  1. We will have aZoom meeting. Please join well ahead of time (since there might be no-shows). You will be placed in the waiting room.
  2. In the waiting room, please monitor the chat to see when it will be your turn.
  3. Please log-in with your full name such that we can identify you
  4. Once connected, please enable your webcam and microphone. Have your student-ID (or any other photo-ID) ready
  5. We will use a random generator and assign your topic
  6. While you prepare for your talk (1-2 mins), we will send you a link to a shared whiteboard
  7. We have decided to use Jamboard (https://jamboard.google.com/). Give it a try and feel comfortable with it. Also note that there is a virtual laser pointer where you can highlight things in your drawing.
  8. While giving your presentation, you can either choose to use this whiteboard or share your screen and use prepared power-point slides
  9. Either way, please be prepared and maybe organize a test run with your friends see whether you presentation is suitable for an online meeting
  10. After the exam you will receive your grade and put back in the waiting room. You are then free to leave
  11. Also, please read through the official guidelines published from SDU and in particular make sure that your computer works, in particular your microphone and webcam.

Below is the list of possible topics and some suggested content. The listed content are only suggestions and are not necessarily complete nor must everything be covered in the short presentation. It is the responsibility of the examinee to gather and select among all relevant information for each topic from the course material. On the course website you can find suggested readings for each of these topics.

Topics for the Oral Exam:

  1. Feed-Forward Networks
    • Function Principle
    • Output Units
    • Hidden Units
    • ...
  2. Backpropagation
    • Function Principle
    • Computational Graphs
    • Backpropagation through time
    • ...
  3. Regularization
    • Over/Underfitting & Model Capacity
    • Parameter Penalties
    • Bagging
    • Dropout
    • ...
  4. Convolutional Neural Networks
    • Function Principle
    • Pooling
    • Initialization of the kernels
    • ...
  5. Recurrent Neural Networks
    • Function Principle
    • Problems with long term memory
    • Long Short Term Memory
    • ...
  6. Autoencoders and GANs
    • Autoencoders
    • Variational Autoencoders
    • GANs
    • ...
  7. Normalization & Style Transfer
    • Batch Normalization
    • Instance Normalization
    • Style Trannsfer Methods
    • ...