Skip to content

Data Engineering🔗

The amount of data in the world, the form these data take, and the ways to interact with data have all increased exponentially in recent years. The extraction of useful knowledge from data has long been one of the grand challenges of computer science, and the dawn of "big data" has transformed the landscape of data storage, manipulation, and analysis. In this module, we will look at the tools used to store and interact with data.

The objective of this class is that students gain:

  • First hand experience with and detailed knowledge of computing models, notably cloud computing
  • An understanding of distributed programming models and data distribution
  • Broad knowledge of many databases and their respective strengths

As a part of the Data and Decision Sciences Master's program, this module aims specifically at providing the tool set students will use for data analysis and knowledge extraction using skills acquired in the Algorithms of Machine Learning and Digital Economy and Data Uses classes.

Class structure🔗

The class is structured in four parts:

Data engineering fundamentals🔗

In this primer class, students will cover the basics of Linux command line usage, git, ssh, and data manipulation in python. The format of this class is an interactive capture-the-flag event.

Data storage🔗

This module covers Database Management Systems with a focus on SQL systems. For evaluation, students will install and manipulate data in PostgreSQL and MongoDB and compare the two systems.

Data computation🔗

A technical overview of the computing platforms used in the data ecosystem. We will briefly cover cluster computing and then go in depth on cloud computing, using Google Cloud Platform as an example. Finally, a class on GPU computing will be given in coordination with the deep learning section of the AML class.

Data distribution🔗

In the final module, we cover the distribution of data, with a focus on distributed programming models. We will introduce functional programming and MapReduce, then use these concepts in a practical session on Spark. Finally, students will do a graded exercise with Dask.