Fall 2020 / DM873 / DS809
Deep Learning

Important Information

Given the current situation with COVID, there might be constant change to schedule and format of the course. It looks like we will have the fist session on campus. Nevertheless, I will live stream the sessions as well (link will be found on blackboard) so students can decide whether they want to participate on campus or online.

This deep learning course runs with two course codes: DS809 and DM873. Since DS809 is a 5 ECTS course, it only constitutes of the first half of this course (till week 43). Therafter, the lectures will only be relevant for students taking DM873.

General Information

Machine learning has become a part in our everydays life, from simple product recommendations to personal electronic assistant to self-driving cars. More recently, through the advent of potent hardware and cheap computational power, “Deep Learning” has become a popular and powerful tool for learning from complex, large-scale data.

In this course, we will discuss the fundamentals of deep learning and its application to various different fields. We will learn about the power but also the limitations of these deep neural networks. At the end of the course, the students will have significant familiarity with the subject and will be able to apply the learned techniques to a broad range of different fields.

Mainly, the following topics will be covered:

• feedforward neural networks
• recurrent neural networks
• convolutional neural networks
• backpropagation algorithm
• regularization
• factor analysis

Lectures

Exercises

Project 1 (DM873 / DS809)

Aim of the Project

This is a simple project where you are suppsoed to put your learnings into practice. You will solve a simple image classification task to tell dogs from cats a part (of course).

You are supposed to work in groups of 4-5 people. In light of the ongoing lock-down, this might be a bit more challenging than usual. Nevertheless, it will help you since you can spread the tasks amongst the team members. In the case you have trouble finding members, please write Mathias & Juan to work out a solution.

You will find all necessary information in the project description. Please read the project description carefully and also note the two Q&A sessions.

Deadline

• Project Hand-In:
  Tuesday the 15th of November 2020 at 23:59.
  This includes:
  • Your code as nextcloud-link via email
  • Your model (as .h5 file) as nextcloud-link via email

Competition

To spice things up and motivate you to explore the possibilities of neural networks, we will also host a competition. The team producing the best network (i.e., the network producing the best accuracy) will win a Raspberry Pi. The best network will be evaluated based on our own dataset. Please follow the rules and only utilize networks which have been built and trained according to the rules (for instance, do not use pre-trained networks, only use the training data provided, etc.).

Downloads

• Project Description (read carefully)
• Dataset

Solution

This solution was provided by Filip Juric and achieved a test perfromance of around 90 percent. Thanks for that!

• Notebook

Project 2 (DM873 only)

In light of the recent COVID-19 pandemic, we will take a look at one of the severe symptoms of the disease, pneumonia. The dataset we have consists of x-ray images of patients who either have pneumonia or are healthy. You must make your own model to detect whether a patient has developed pneumonia, or is healthy.

To spice things up and to also give you deeper insights in KERAS, you are also supposed to implement your own layers and mix them with existing other layers. Make use of the upcoming Q&A sessions in case you are stuck and require help from Mathias.

You are supposed to work in groups of 4-5 people. In light of the ongoing lock-down, this might be a bit more challenging than usual. Nevertheless, it will help you since you can spread the tasks amongst the team members. In the case you have trouble finding members, please write Mathias to work out a solution.

Further Information

You will find all necessary information in the project description. It is fundamentally important that you start early on, since the models might take some time to train.

Deadline

• Project Hand-In:
  Tuesday the 18th of December 2020 at 23:59.
  This includes:
  • Brief Report via Blackboard
  • Your code as nextcloud-link via email
  • Your model (as .h5 file) as nextcloud-link via email

Downloads

• Project Description
• Dataset (link to nextcloud)

Procedure of the oral exam

It is not yet clear whether the exam will be via zoom or on Campus. The instructions below are assuming the exam is being held via zoom. In case the exam will be held on Campus, the procedure will stay the same and you can ignore the steps concerning zoom and just follow the normal procedures for oral exams.

The exam will last about 15-20 minutes. At the beginning, one topic from the list below will be drawn randomly using a random number generator. For each topic the examinee should be prepared to make a short presentation of 5 minutes (sharp). The examinee will have 2 minutes to briefly study the notes for the drawn topic before the presentation. The notes may be consulted during the presentation if needed but it will negatively influence the evaluation of the examinee's performance. The student can either use a presentation or a shared whiteboard for quick sketches, etc. (see technical details below).

After the short presentation, additional questions about the presentation's topic but also about other topics in the curriculum will be asked.

Technical Details and precise procedure:

1. We will have a zoom meeting. Please join well ahead of time (since there might be no-shows). You will be placed in the waiting room.
2. In the waiting room, please monitor the chat to see when it will be your turn.
3. Please log-in with your full name such that we can identify you
4. Once connected, please enable your webcam and microphone. Have your student-ID (or any other photo-ID) ready
5. We will use a random generator and assign your topic
6. While giving your presentation, you can either choose to use the zoom whiteboard or use prepared power-point slides (or a combination of both)
7. Either way, please be prepared and maybe organize a test run with your friends see whether you presentation is suitable for an online meeting
8. After the exam you will receive your grade and put back in the waiting room. You are then free to leave
9. Also, please read through the official guidelines published from SDU and in particular make sure that your computer works, in particular your microphone and webcam.

Below is the list of possible topics and some suggested content. The listed content are only suggestions and are not necessarily complete nor must everything be covered in the short presentation. It is the responsibility of the examinee to gather and select among all relevant information for each topic from the course material. On the course website you can find suggested readings for each of these topics.

Topics for the Oral Exam:

1. Feed-Forward Networks (DS809 / DM873)
• Function Principle
• Output Units
• Hidden Units
• Network Training
• ...
2. Backpropagation (DS809 / DM873)
• Function Principle
• Computational Graphs
• Stochastic Gradient Descent / Optimizers
• ...
3. Regularization (DS809 / DM873)
• Over/Underfitting & Model Capacity
• Parameter Penalties
• Bagging
• Dropout
• ...
4. Convolutional Neural Networks (DS809 / DM873)
• Function Principle
• Pooling
• Initialization of the kernels
• ...
5. Recurrent Neural Networks (DM873 only)
• Function Principle
• Problems with long term memory
• Long Short Term Memory
• Backpropagation through time
• ...
6. Autoencoders and GANs (DM873 only)
• Autoencoders
• Variational Autoencoders
• GANs
• ...
7. Normalization & Style Transfer (DM873 only)
• Batch Normalization
• Instance Normalization
• Style Transfer Methods
• ...

Materials

• All lecture slides are relevant for the exams.
• All readings noted in the lecture list are relevant for the exam.
• Ian Goodfellow, Yoshua Bengio, Aaron Courville - The Deep Learning Book
• Gareth James, Daniela Witten, Trevor Hastie - Robert Tibshirani An Introduction to Statistical Learning (ISL)