The best complete summary for Data Engineering for MADS (EBM213A05), it includes: Lectures, Readings & Book Chapters (Mandatory + Optional), Weekly Quizzes and the latest Practice Exam. Enhanced with a dynamic table of contents and meticulous organization for readability and easy studying.
100%...
Answers for Quiz 1, Quiz 5, Quiz 6 and a Full Practice Exam for Data Engineering for MADS (EBM213A05)
Extensive Summary of Verhoef book (Chapter 1-5) + Lecture 2 for Data Engineering for MADS (EBM213A05)
Readings Summary for Data Engineering for MADS (Mandatory & Optional Papers + Book Chapters)
All for this textbook (5)
Written for
Rijksuniversiteit Groningen (RuG)
Marketing Analytics & Data Science
Data Engineering for MADS (EBM213A05)
All documents for this subject (5)
1
review
By: maxvanhalteren1 • 1 year ago
Seller
Follow
madsmadlad
Reviews received
Content preview
SUMMARY OF EVERYTHING
LECTURES + READINGS + CHAPTERS +
AVAILABLE WEEKLY-QUIZZES + 2022 PRACTICE EXAM
Bonus: Week 7 lecture substituted for more extensive material.
E n h a n c e d w i t h a d y n a m ic t a b le o f c o n t e n t s .
, MADS MADLAD |2
Note from MADS MADLAD:
Thank you for buying my summary. I sincerely hope it helps you excel and learn
from this course. When I was writing these I sometimes struggled with this
program, but there were no summaries available.
This is why I decided to write something that is truly complete with a lot of
effort put into it. It helped me and my friends get good grades, but I also
always had you in mind, the future reader. When necessary, I always went the
extra mile to make this summary, more readable, organized and complete.
If you feel like it, leave me a review of how the course is going using this
summary, it will make my day to hear your opinion good or bad!
Check out my other extensive summaries for other MADS courses:
Available on:
Contact info:
If you need help or have an inquiry, contact me: https://www.georgedreemer.com
Connect with me on LinkedIn: https://www.linkedin.com/in/georgedreemer/
Donations:
By no means am I looking for fellow students to send me money! But if you feel like sending
me some ETH or BTC, you can do so here:
--> ETH: 0x123e086c6808459e7fC6Ac7F64a77dBA1dDe0149
--> BTC: bc1qgwzc82vph5v8rmzef4ywechjf85772n7m2e22g
, MADS MADLAD |3
wishes you good luck & perseverance.
Grades Testimony:
, MADS MADLAD |4
Table of Contents
Week 1 (Lectures+Readings) ............................................................................ 14
Lecture 0 (Intro) & 1.1 (MD->MQ->RQ) & HBR article ................................... 14
Management Dilemma and Questions, Research Questions ..................... 14
Management Dilemma......................................................................... 14
From Management Dilemma to Management Questions: ................... 15
Management Question defined: .......................................................... 15
From Management Question to Research Question ............................ 15
Management Question vs. Research Question:.................................... 16
From Research Question to Analysis Questions: .................................. 16
7 steps of the opportunity tree ............................................................ 16
Lecture 1.2 - supplement (Wehkamp lecture 2022) ...................................... 19
7 Elements of a Data Strategy .................................................................... 19
Data Warehouse vs. Data Lake .................................................................. 19
4 Ways to work with Data .......................................................................... 21
Data Science – mix of 3 fields of knowledge .............................................. 21
Domain Expertise (Business) ................................................................ 22
Technical Data Engineering .................................................................. 22
Math & Statistical Knowledge .............................................................. 22
Reading – HBR Article (optional) ................................................................... 24
4 Steps to Management Questions ............................................................ 24
Reading – Book: Verhoef et al. (11.6 required +11.1 optional) ..................... 25
7 Steps of Opportunity finding ................................................................... 25
Reading – Book: Business Research Methods (2.1 optional + 2.2 required) .. 28
Week 2 (Readings+Lecture) ............................................................................. 32
Reading – Book: Verhoef et al.(Ch.1, 2.1-2.5, 3.4, 3.9, 4, 5) .......................... 32
Chapter 1 (required): “Data science and big data”........................................ 32
Chapter 2.1-2.5 (required): “Creating value with data science” .................... 35
2.1 INTRODUCTION ................................................................................ 35
2.2 DATA SCIENCE VALUE CREATION MODEL ......................................... 35
, MADS MADLAD |5
2.3 VALUE CREATION OBJECTIVES .......................................................... 36
2.3.1 Balance between V2F and V2C: ............................................... 36
2.3.2 V2S – Value to Society ................................................................. 37
2.3.3 Metrics for V2F and V2C .......................................................... 37
2.4 DATA ASSETS .................................................................................... 38
2.5 DATA ANALYTICS .................................................................................. 38
2.5.1 The power of visualization and storytelling ............................. 39
Chapter 3.4 & 3.9 (required): “CUSTOMER METRICS”................................... 39
3.4 CUSTOMER (FEEDBACK) METRICS .................................................... 39
First Dimension: Time span - Forward- vs. Backward-looking Metrics .. 40
Second Dimension: Measurement scale............................................... 41
3.4.1 Is there a silver metric? ........................................................... 41
3.4.2 Other theoretical relationship metrics .................................... 42
3.4.3 Customer equity drivers .............................................................. 42
3.4.4 Internal data sources ................................................................... 43
3.4.5 Online sources – Customer Reviews ............................................ 43
3.9 CUSTOMER METRICS ........................................................................ 44
3.9.1 Customer acquisition metrics ...................................................... 45
3.9.2 Customer development metrics .................................................. 45
3.9.3 Customer value metrics ............................................................... 46
3.9.4 Customer equity .......................................................................... 48
3.9.5 New big data metrics................................................................... 49
Chapter 4 (required): “Data Assets”.............................................................. 51
4.2 DATA SOURCES AND THE DIFFERENT TYPES OF DATA .......................... 51
4.2.1 External data sources vs. Internal data sources ........................... 51
4.2.2 Structured vs. Unstructured data ................................................ 52
4.2.3 Market data................................................................................. 53
4.2.5 Brand data ................................................................................... 54
4.2.7 Customer data ............................................................................. 56
4.3 USING THE DIFFERENT DATA SOURCES IN THE ERA OF BIG DATA ........ 57
, MADS MADLAD |6
4.4 DATA QUALITY AND DATA CLEANSING................................................. 58
4.4.1 Data Quality ................................................................................ 58
4.4.2 Data Cleansing ............................................................................. 59
4.4.3 Missing value and data fusion ..................................................... 60
Chapter 5 (required): “Data storing and integration” ................................... 60
5.1INTRODUCTION .................................................................................... 60
5.2 STORING AND INTEGRATING DATA SOURCES IN DATA WAREHOUSES. 60
5.2.1 Storing data in the data warehouse............................................. 61
5.2.2 The data model in a data warehouse .......................................... 62
5.2.3 Data integration into the data warehouse................................... 64
5.2.3.1 Extraction ............................................................................. 65
5.2.3.2 Transformation ..................................................................... 65
5.2.3.3 Loading ................................................................................. 65
5.3 STORING AND INTEGRATING DATA SOURCES IN DATA LAKES .............. 66
5.4 CHALLENGES OF DATA INTEGRATION IN THE ERA OF BIG DATA .......... 68
5.4.1 The technical challenges of integrated data ................................ 68
5.4.1.1 Integration at the individual level ......................................... 69
5.4.1.2 Integration at the intermediate level .................................... 69
5.4.1.3 Integration at the time level ..................................................... 69
5.4.2 The analytical challenges of integrated data ............................... 70
5.4.3 The business challenges of integrated data ................................. 70
5.4.3.1 Dealing with different data types ......................................... 70
5.4.3.2 Declared data: customer descriptors .................................... 71
5.4.3.3 Appended data ..................................................................... 71
5.4.3.4 Overlaid data ........................................................................ 71
5.4.3.5 Implied data ......................................................................... 72
Lecture 2 – From ‘raw’ data to your own table: conceptual introduction ..... 73
Our first choice: level of analysis (aggregation level) ................................. 73
Four Types of Information ......................................................................... 73
Joining tables, aggregation, adding, external data ..................................... 74
, MADS MADLAD |7
INNER_JOIN.......................................................................................... 74
LEFT_JOIN /RIGHT_JOIN ....................................................................... 74
FULL_OUTER_JOIN ............................................................................... 75
Week 3 (Lecture) ............................................................................................. 79
Lecture – Working with SQL, Azure Data, Creating our own table: From ‘raw’
data to your own table with SQL................................................................... 79
Every SQL query to retrieve data consists of maximum 6 SQL instructions 79
Week 4 (Lecture) ............................................................................................. 80
Lecture – Combining Wehkamp data with external data in RStudio ............. 80
Two dimensions: Data source & Data type ................................................ 80
Sources for external data ........................................................................... 80
Integrating external data sources .............................................................. 82
Useful commands in R Studio .................................................................... 82
Week 5 (Readings+Lecture) ............................................................................. 83
Reading – Wickham (2014) “Tidy Data” ........................................................ 83
Section 2: Three characteristics of a Tidy dataset ...................................... 83
Section 2.1: Data structure ................................................................... 83
Section 2.2: Data Semantics ................................................................. 84
Section 2.3: Tidy data ........................................................................... 84
Section 3: Operations to make a messy dataset tidy.................................. 85
Section 3.1: Column headers are values, not variable names ............... 85
3.2: Multiple variables are stored in one column ................................. 87
3.3: Variables are stored in both rows and columns............................. 88
3.4: Multiple types of observational units are stored in the same table
............................................................................................................. 89
3.5: A single observational unit is stored in multiple tables.................. 89
Section 4: Tidy tools (tools that input & output tidy data) ......................... 90
Section 4.1: Data Manipulation ............................................................ 90
Section 4.2: Visualization ..................................................................... 90
Section 4.3: Modeling .......................................................................... 91
, MADS MADLAD |8
Reading – de Jonge, E. and van der Loo, M. (2013) "An introduction to data
cleaning with R" ............................................................................................ 92
1 Introduction ............................................................................................ 92
1.1: Statistical analysis in 5 steps.......................................................... 92
1.2 Some general background in R ....................................................... 93
1.2.1 Variable types and indexing techniques .................................. 93
1.2.2 Special values .......................................................................... 93
2 From raw data to technically correct data .............................................. 93
2.1: Technically correct data in R.......................................................... 93
2.2: Reading text data into a R data.frame ........................................... 94
2.2.1 read.table() and its cousins...................................................... 94
2.3: Type conversion ............................................................................ 98
2.3.1 Introduction to R’s typing system ............................................ 98
2.3.2 Recoding factors ...................................................................... 98
2.3.3 Converting dates ..................................................................... 99
2.4: character manipulation ............................................................... 100
2.4.1 String normalization .............................................................. 101
2.4.2 Approximate string matching ................................................ 102
2.5 Character encoding issues ............................................................ 105
3: From technically correct data to consistent data ................................. 105
3.1 Detection and localization of errors ............................................. 106
3.1.1 Missing values ....................................................................... 106
3.1.2 Special values ........................................................................ 107
3.1.3 Outliers .................................................................................. 108
3.1.4 Obvious inconsistencies......................................................... 109
3.2 Correction .................................................................................... 111
3.2.1 Simple transformation rules .................................................. 112
3.2.2 Deductive correction ............................................................. 114
3.2.3 Deterministic imputation....................................................... 115
3.3 Imputation ................................................................................... 116
, MADS MADLAD |9
3.3.1 Basic numeric imputation models ......................................... 117
3.3.2 Hot deck imputation .............................................................. 117
3.3.3 kNN-imputation ..................................................................... 118
3.3.4 Minimal value adjustment ......................................................... 119
Reading – Donders, A.G.T, van der Heijden, G.J.M.G, Stijnen, T and Moons,
K.G.M. (2006) "Review: A gentle introduction to imputation of missing values"
................................................................................................................... 120
Reading – Schafer, J.L. and Graham, J.W. (2002), "Missing Data: Our View of
the State of the Art" ................................................................................... 120
Lecture 5 – Data cleaning, outliers, missing data (1/2) ............................... 121
Data Quality ............................................................................................. 121
Data Cleaning........................................................................................... 121
Steps in Cleansing (lecture) ................................................................ 122
Parsing (1) ...................................................................................... 122
Correcting (2) ................................................................................. 123
Standardizing (3) ............................................................................ 123
Matching (4) ................................................................................... 124
Consolidating (5) ............................................................................ 124
Formatting the data (Tidy Data) ............................................................... 125
Packages for Tidying Data .................................................................. 125
After Data Cleaning: Sanity Check ............................................................ 126
Outliers .................................................................................................... 126
What are outliers? ............................................................................. 126
Why do we address outliers? ............................................................. 126
Detecting Outliers .............................................................................. 127
Treating Outliers ................................................................................ 128
Missing Data ............................................................................................ 129
MCAR, MAR, MNAR – Graphical representation (lecture) ............... 130
MCAR, MAR, MNAR: Academic & Simple explanation (ext. source) ... 131
, M A D S M A D L A D | 10
Lecture 5 – Detecting outliers/Mahalanobis distance, Multiple imputation
(2/2)............................................................................................................ 132
Detecting outliers/Mahalanobis Distance ................................................ 132
How to do it in R: ............................................................................... 135
Imputation ............................................................................................... 137
How to do it in R................................................................................. 138
Week 6 (Readings+Lecture) ........................................................................... 142
Reading – Book: Chapter 7 and 8 ................................................................ 142
Chapter 7: Data Analytics............................................................................ 142
7.2 THE POWER OF ANALYTICS ................................................................ 142
7.3 STRATEGIES FOR ANALYZING DATA ................................................... 142
7.3.1 PROBLEM SOLVING ................................................................... 143
7.3.2 DATA EXPLOITATION ................................................................. 143
7.3.3 DATA MINING ............................................................................ 144
7.3.4 COLLATERAL CATCH................................................................... 145
7.4 TYPES OF DATA ANALYTICS ................................................................ 146
7.4.1 DESCRIPTIVE ANALYTICS ............................................................ 147
7.4.2 DIAGNOSTIC ANALYTICS ............................................................ 147
7.4.3 PREDICTIVE ANALYTICS.............................................................. 148
7.4.4 PRESCRIPTIVE ANALYTICS .......................................................... 148
7.5 HOW BIG DATA AND AI CHANGE ANALYTICS ..................................... 149
7.5.1.1 DATA SCIENCE ........................................................................ 149
7.5.1.2 AI ............................................................................................ 150
7.5.1.3 MACHINE LEARNING (ML) ...................................................... 150
7.5.1.4 DEEP LEARNING (DL) .............................................................. 151
7.6 ANALYTICAL METHODS AND TECHNIQUES......................................... 151
Chapter 8: Data Exploration ........................................................................ 152
8.1 INTRODUCTION.................................................................................. 152
8.2 DESCRIPTIVE ANALYSES – REPORTING ............................................... 153
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller madsmadlad. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $16.29. You're not tied to anything after your purchase.
