Introduction to Machine Learning (Autumn 2023)

Abstract

The course is an introductory course to machine learning its mathematical foundation.

 We recommend that you do not take this cour if you have already taken the course Machine Learning on the Bachelor programme in Data Science at ITU or a similar.

Description

The objectives of this course are to provide students with the fundamental knowledge and skills required to design, build, evolve and evaluate (smaller) machine learning programs. The course requires programming skills equivalent to a completed bachelor in software development (SWU) hence substantial programming experience is expected. The course  contains significant amounts of mathematics and the student should expect to spend some time to  get comfortable with the mathematics and how it is used within Machine learning and programming.

Having knowledge within signals and machine learning (e.g. deep learning) are  important skills needed in many modern innovative businesses. The course will introduce specific machine learning techniques and apply them to relevant small-scale problems such as recognition, regression, dimensionality reduction and clustering. Through the course the student should be able to understand the mathematical foundation and the use the techniques in their own applications and within more advanced topics using machine learning.

Contents:
There is a strong synergy between  machine learning, linear algebra and optimization.  Understanding machine learning requires a good understanding of linear algebra and optimization and vice versa, these topics are nicely explained in the context of machine learning.

The course introduces the basic methodology of machine learning in primarily covering linear aspects of

• regression
• classification
• clustering
• dimensionality reduction

and hence relying on a good foundation of linear algebra.  The course will  also touch upon non-linear models and neural networks.

The course  will present  fundamental mathematical models for machine learning as well as techniques to implement them. The in-class exercises will mostly cover fundamental mathematical topics and small hands-on programming exercises. You will in the exercises and mandatory assignments be getting hands-on experience with using the techniques described during the lectures. You will not be reimplementing advanced methods, but will rather use them for analysis and interpretation.  In the exercises you will be using Python and image analysis / machine learning packages such as OpenCV, PyTorch, Numpy. Datasets of various datatypes will be used in the exercises to illustrate the theory.

In particular the course covers:

•  Linear algebra (matrix multiplication, transformations, solving linear set of equations, subspaces and SVD)
•  Basics of images e.g. pixel-based and local processing of images (smoothing, edges, conversion between color spaces) and color image processing.
• Transformations of data and  data wrangling
  
•  Regression, classification, dimensionality reduction and clustering, including neural networks, deep learning, support vector machines and principal component analysis / Auto encoders, k-means and  mean-shift algorithms,   and recommender systems
• Regularization, non-linear optimization (gradient descent), multivariate differentiation including the chain rule.
  
• Python, OpenCV, Numpy,PyTorch

During the course students should particularly focus on  

• the mathematical models underlying modern machine learning and computer vision techniques. The course will contain significant amounts of mathematics which is essential for understanding the machine learning models.  IT IS ABSOLUTELY not something that can be avoided.
• The techniques for estimating the parameters of the model given the definition of optimality.
• evaluating  and comparing models on a given problem using established
  

Interpretation
This means that you should be able to use and explain the mathematics that is taught in the course. Similarly you should be able to explain the models of machine learning in terms of mathematics and indicate how they can be implemented, but (and this is important) you will NOT be asked to reimplement specific / advanced ML methods (such as KNN, neural networks etc). There already exist very good implementations and no need to make your own implementation. It is MUCH more important that you can

• understand and describe the underlying (mathematical) models and how model parameters are optimally found.
  

• select an appropriate Machine learning model for a given problem.

• interpret the output of the model (eg. what do the principal components represent)

• compare and evaluate the results from the model

Formal prerequisites

The course assumes the student has the ability to program in some imperative programming language (Java, Python, C/C++/C#.  You should be  a rather proficient programmer as expected on 1. semester on computer science. The course uses concepts from mathematics and in particular linear algebra. Fulfilling the general admittance requirements for CS and SWU ensures the mathematical prerequisites. We advice that students refresh their memory on the distance formula, basic trgonometry, vectors, rules of  differentiation (e.g. chain rule), formulas for straight lines and polynomia, prior to the course.

Intended learning outcomes

After the course, the student should be able to:

• Define, describe and relate concepts, mechanisms and mathematical models underpinning machine learning (ML) methods.
• Analyze and explain key aspects of constructing medium-sized machine learning applications.
• Explain, design and implement classification, regression, clustering and dimensionality reduction methods and apply them to specific problems
• Evaluate, select and adapt appropriate machine learning techniques by applying the theoretical concepts and practical techniques from the course and evaluate the performance.
• Clearly explain and employ basic linear algebra to machine learning.
• Apply the theory and implement rudimentary research papers within machine learning as expected on a master level
• Be able to describe linear and non-linear models of machine learning and how the model parameters are optimally found.

Learning activities

The course includes both lectures and in-class exercises, as well as tutorials and exercise sessions. Students should plan to spend around 11-12 hours per week on these activities, which will include completing mandatory assignments . The course material builds on itself from week to week, so it's important for students to keep up with the planned activities in order to progress. Students are  to work systematically as expected on Computer Science master level.

Throughout the course, students will have many opportunities to gain hands-on experience with the mathematical fundamentals of machine learning models and applying and evaluating machine learning techniques in Python. Exercises and assignments are clearly marked with their level of difficulty and which ones are required for the mandatory activities. There are also plenty of extra exercises available for practice.

During the oral examination, students will be assessed based on their ability to relate the exercises and assignments to the theory and mathematical foundation of the course. Students should expect to purchase a few books to cover different aspects of the course.

Mandatory activities

Throughout the course, there will be 2-3 mandatory assignments that must be handed in to the TA. These assignments will typically be extensions of specific exercises and may extend over several weeks. After each hand-in, students will receive written and/or oral formative feedback to help them reflect on their work. If a student fails to hand in or does not receive approval on their assignment, they will have one week to make revisions and resubmit.

It's important to note that the mandatory assignments serve a pedagogical purpose. They are designed to ensure scaffolding and clear communication/ feedback about essential course content, and expectations for the exam.

The student will receive the grade NA (not approved) at the ordinary exam, if the mandatory activities are not approved and the student will use an exam attempt.

Course literature

Chirag Shah A Hands-on Introduction to Machine Learning

http://www.deeplearningbook.org/

The following book is TENTATIVE. DO NOT BUY YET

Introduction to Linear Algebra 5. edition Gilbert Stang

Various notes

Several online lectures

Student Activity Budget

Estimated distribution of learning activities for the typical student

• Preparation for lectures and exercises: 30%
• Lectures: 33%
• Exercises: 29%
• Assignments: 8%

Ordinary exam

Exam type:
B: Oral exam, External (7-point scale)
Exam variation:
B22: Oral exam with no time for preparation.

reexam

Exam type:
B: Oral exam, External (7-point scale)
Exam variation:
B22: Oral exam with no time for preparation.

Time and date

Ordinary Exam Tue, 9 Jan 2024, 09:00 - 21:00
Ordinary Exam Wed, 10 Jan 2024, 09:00 - 21:00
Ordinary Exam Thu, 11 Jan 2024, 09:00 - 21:00
Ordinary Exam Fri, 12 Jan 2024, 09:00 - 21:00
Ordinary Exam Mon, 15 Jan 2024, 09:00 - 21:00
Ordinary Exam Tue, 16 Jan 2024, 09:00 - 21:00
Ordinary Exam Thu, 18 Jan 2024, 09:00 - 21:00
Ordinary Exam Fri, 19 Jan 2024, 09:00 - 21:05
Reexam Wed, 6 Mar 2024, 09:00 - 18:00
Reexam Fri, 8 Mar 2024, 09:00 - 18:00