Data mining is an automatic or semi-automatic technical process that analyses large amounts of scattered information to make sense of it and turn it into knowledge. It looks for anomalies, patterns or correlations among millions of records to predict results, as indicated by the SAS Institute, a wo...
1. Data Mining
Data mining is an automatic or semi-automatic technical process that analyses large
amounts of scattered information to make sense of it and turn it into knowledge. It
looks for anomalies, patterns or correlations among millions of records to predict
results, as indicated by the SAS Institute, a world leader in business analytics.
Data mining is the process of using statistical analysis and machine learning to
discover hidden patterns, correlations, and anomalies within large datasets. This
information can aid you in decision-making, predictive modelling, and understanding
complex phenomena.
In the meantime, information continues to grow and grow. A 2017 research on big
data reveals that 90% of world data is from after 2014 and its volume doubles every
1.2 years. In this context, data mining is a strategic practice considered important by
almost 80% of organisations that apply business intelligence, according to Forbes.
Thanks to the joint action of analytics and data mining, which combines statistics,
Artificial Intelligence and automatic learning, companies can create models to
discover connections between millions of records. Some of the possibilities of data
mining include:
• To clean data of noise and repetitions.
• Extract the relevant information and use it to evaluate possible results.
• Make better and faster business decisions.
Data mining is the process of discovering patterns, correlations, and anomalies
within large datasets with the goal of extracting useful information and converting it
into an understandable structure for further use. It is a key component of data
analysis and an essential part of the knowledge discovery in databases (KDD)
process, which involves methods from statistics, machine learning, artificial
intelligence, and database management.
2
, 3
Technology Solutions
2. The Data Mining Process
Data mining involves a series of steps to extract valuable insights from large
datasets. While the exact process can vary depending on the specific project and
tools used, the general workflow typically includes the following stages:
1. Data Collection and Preparation
Gather data: Collect data from various sources, such as databases, files,
sensors, and APIs.
Clean and pre-process: Handle missing values, outliers, inconsistencies,
and inconsistencies to ensure data quality.
Transform data: Convert data into a suitable format for analysis, such as
normalizing or standardizing.
Data Collection and Preparation: The Foundation of Data Mining
Data collection and preparation are crucial initial steps in the data mining process.
They lay the groundwork for accurate and meaningful analysis. Here's a detailed
breakdown of these essential tasks:
Data Collection
Identifying data sources: Determine where the necessary data resides. This
might include databases, files, APIs, sensors, or even manual collection.
Data acquisition: Gather the data from the identified sources. This may
involve using tools like SQL queries, web scraping, or data extraction APIs.
Data integration: Combine data from multiple sources if needed. This might
involve joining tables, merging datasets, or resolving inconsistencies.
3
, 4
Technology Solutions
Data Cleaning and Pre-processing
Handling missing values: Address missing data points using techniques like
imputation (filling in missing values with estimated values) or deletion.
Dealing with outliers: Identify and handle outliers (data points that deviate
significantly from the norm) to prevent them from skewing the analysis.
Correcting errors: Identify and correct errors or inconsistencies in the data.
This might involve data validation, data cleansing, or data standardization.
Data normalization: Scale or transform data to a common range or
distribution to ensure that features contribute equally to the analysis.
Feature engineering: Create new features or transform existing ones to
improve the model's performance. This might involve combining features,
creating derived features, or discretizing continuous variables.
Data Transformation
Data conversion: Convert data from one format to another (e.g., text to
numerical, categorical to numerical).
Data aggregation: Combine data into summary statistics or aggregates (e.g.,
calculating averages, sums, or counts).
Data sampling: Select a subset of data for analysis if the full dataset is too
large or computationally expensive to process.
Quality Assurance
Data validation: Ensure that the data adheres to specific constraints or rules.
Data quality checks: Verify the accuracy, completeness, consistency, and
reliability of the data.
By carefully collecting, cleaning, and preparing the data, you can ensure that your
data mining efforts yield accurate and meaningful results. A well-prepared dataset is
the foundation for successful data analysis
2. Data Exploration and Understanding
Descriptive statistics: Calculate summary statistics (e.g., mean, median,
mode, standard deviation) to understand the data distribution.
Data visualization: Create charts, graphs, and visualizations to explore
relationships and patterns within the data.
Feature engineering: Create new features or transform existing ones to
improve model performance.
Data Exploration and Understanding: Discovering Insights
Data exploration and understanding is a critical phase in the data mining process. It
involves delving into the data to uncover patterns, relationships, and insights that can
guide further analysis. Here's a detailed breakdown of this step:
4
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller bhupinderkumar. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $8.07. You're not tied to anything after your purchase.
