Exploratory Data Analysis, commonly called EDA, is the process of examining, cleaning, summarizing, and visualizing a dataset before building machine learning models or making important conclusions.

When you receive a new dataset, you usually do not know everything about it.

You may not know:

• How many rows it contains
• Which columns are numerical
• Whether values are missing
• Whether duplicates exist
• Whether some values are incorrect
• Whether outliers are present
• Which variables are related
• Whether categories are balanced
• Whether the data is suitable for machine learning

EDA helps answer these questions.

For example, suppose you receive a customer dataset:

name    age    city      income
Aman    25     Jaipur    45000
Riya    28     Delhi     52000
Rahul   23     Mumbai    40000
Neha    30     Delhi     65000

Instead of immediately building a model, you should first understand the structure and quality of this data.

This guide explains Exploratory Data Analysis with Python step by step using Pandas and Matplotlib.

What Is Exploratory Data Analysis?

Exploratory Data Analysis is the process of investigating a dataset to understand its structure, quality, patterns, relationships, and unusual values.

EDA is usually performed before statistical modeling or machine learning.

A simple workflow looks like:

Raw Dataset
   ↓
Load Data
   ↓
Inspect Structure
   ↓
Check Missing Values
   ↓
Check Duplicates
   ↓
Analyze Data Types
   ↓
Study Distributions
   ↓
Find Outliers
   ↓
Explore Relationships
   ↓
Visualize Patterns
   ↓
Prepare Data for Modeling

EDA is not only about creating charts.

It combines:

• Inspection
• Cleaning
• Statistics
• Visualization
• Reasoning

Why Is EDA Important?

A machine learning model learns from the data you provide.

If the data contains serious problems, the model may learn misleading patterns.

EDA helps you detect issues before they become larger problems.

For example, a dataset may contain:

age = -5
salary = 9000000
city = missing
purchase = unknown

Some of these values may be mistakes.

Others may be valid but unusual.

EDA helps you investigate them.

Main Goals of EDA

The main goals of Exploratory Data Analysis are to understand:

• Dataset size
• Column types
• Missing values
• Duplicate records
• Numerical distributions
• Categorical distributions
• Outliers
• Relationships between variables
• Correlations
• Data quality
• Potential machine learning features

The goal is not to force the data into a particular conclusion.

The goal is to understand what the dataset actually contains.

Tools Used for EDA in Python

Several Python libraries are commonly used.

Pandas

Pandas is used for:

• Loading data
• Inspecting rows and columns
• Cleaning data
• Filtering records
• Calculating statistics
• Grouping values

Import it using:

import pandas as pd

NumPy

NumPy is useful for numerical operations.

import numpy as np

Matplotlib

Matplotlib is used for charts and visualizations.

import matplotlib.pyplot as plt

Seaborn

Seaborn is useful for statistical visualization.

import seaborn as sns

For beginners, you can perform a lot of EDA using only Pandas and Matplotlib.

Installing the Required Libraries

Install Pandas and Matplotlib:

pip install pandas matplotlib

If you also want Seaborn:

pip install seaborn

Then import:

import pandas as pd
import matplotlib.pyplot as plt

Create a Sample Dataset

Let’s create a simple dataset:

import pandas as pd

data = {
    "name": [
        "Aman",
        "Riya",
        "Rahul",
        "Neha",
        "Kabir",
        "Pooja"
    ],
    "age": [
        25,
        28,
        23,
        30,
        35,
        None
    ],
    "city": [
        "Jaipur",
        "Delhi",
        "Mumbai",
        "Delhi",
        "Jaipur",
        "Mumbai"
    ],
    "salary": [
        45000,
        52000,
        40000,
        65000,
        90000,
        48000
    ],
    "experience": [
        2,
        4,
        1,
        6,
        10,
        3
    ]
}

df = pd.DataFrame(data)

print(df)

We will use this DataFrame throughout the guide.

Step 1: View the First Rows

Use:

print(df.head())

By default, head() shows the first five rows.

You can specify another number:

print(df.head(10))

This gives you a quick idea of what the dataset looks like.

Step 2: View the Last Rows

Use:

print(df.tail())

This is useful for checking the end of the dataset.

For example:

print(df.tail(3))

shows the last three rows.

Step 3: Check Dataset Shape

Use:

print(df.shape)

Suppose the result is:

(6, 5)

This means:

• 6 rows
• 5 columns

Dataset shape is one of the first things you should check.

Step 4: Check Column Names

Use:

print(df.columns)

You may get:

Index(['name', 'age', 'city', 'salary', 'experience'], dtype='object')

Check whether column names are:

• Clear
• Consistent
• Free from unexpected spaces
• Easy to use

Step 5: Inspect Data Types

Use:

print(df.dtypes)

You might see:

name           object
age           float64
city           object
salary          int64
experience      int64

Data types matter because numerical and categorical columns are analyzed differently.

For example:

salary

should usually be numerical.

If salary is stored as text, calculations may fail.

Step 6: Use df.info()

Run:

df.info()

This shows:

• Number of rows
• Column names
• Non-null values
• Data types
• Approximate memory usage

This is one of the most useful EDA commands.

Step 7: Generate Summary Statistics

Use:

print(df.describe())

For numerical columns, Pandas shows:

• Count
• Mean
• Standard deviation
• Minimum
• 25th percentile
• Median
• 75th percentile
• Maximum

Example:

age
mean = 28.2
min = 23
max = 35

This helps you quickly identify unusual values.

Include Categorical Columns

Use:

print(
    df.describe(
        include="all"
    )
)

This can include information such as:

• Unique categories
• Most frequent value
• Frequency

Step 8: Check Missing Values

Use:

print(
    df.isnull().sum()
)

Example:

name          0
age           1
city          0
salary        0
experience    0

This tells you that age contains one missing value.

Calculate Missing Value Percentage

Use:

missing_percentage = (
    df.isnull().mean() * 100
)

print(
    missing_percentage
)

This helps when working with larger datasets.

Step 9: Inspect Rows with Missing Values

Use:

missing_rows = df[
    df.isnull().any(axis=1)
]

print(
    missing_rows
)

This lets you inspect affected records before deciding what to do.

Step 10: Handle Missing Values

For example, fill age using the median:

df["age"] = df["age"].fillna(
    df["age"].median()
)

Or remove missing rows:

df = df.dropna()

Do not automatically remove or replace missing values.

The correct decision depends on why the values are missing.

Step 11: Check Duplicate Rows

Count duplicates:

print(
    df.duplicated().sum()
)

Remove duplicates if they are truly unnecessary:

df = df.drop_duplicates()

However, repeated rows may sometimes represent valid repeated events.

Always understand the meaning of each record.

Step 12: Check Unique Values

For categorical columns:

print(
    df["city"].unique()
)

Output:

['Jaipur' 'Delhi' 'Mumbai']

Count unique values:

print(
    df["city"].nunique()
)

This is useful for identifying inconsistent categories.

For example:

Delhi
delhi
DELHI

may need to be standardized.

Step 13: Count Category Frequencies

Use:

print(
    df["city"].value_counts()
)

Example:

Jaipur    2
Delhi     2
Mumbai    2

This helps you understand category distribution.

Step 14: Analyze Numerical Columns

For salary:

print(
    df["salary"].describe()
)

You can also calculate individual statistics.

Mean:

print(
    df["salary"].mean()
)

Median:

print(
    df["salary"].median()
)

Minimum:

print(
    df["salary"].min()
)

Maximum:

print(
    df["salary"].max()
)

Standard deviation:

print(
    df["salary"].std()
)

These values help summarize the distribution.

Step 15: Visualize Numerical Distributions

A histogram is useful for numerical data.

Example:

import matplotlib.pyplot as plt

plt.hist(
    df["salary"],
    bins=5
)

plt.title(
    "Salary Distribution"
)

plt.xlabel(
    "Salary"
)

plt.ylabel(
    "Frequency"
)

plt.show()

This can show whether salaries are:

• Concentrated
• Spread out
• Skewed
• Potentially affected by outliers

Step 16: Visualize Categorical Data

First count categories:

city_counts = (
    df["city"]
    .value_counts()
)

Then:

city_counts.plot(
    kind="bar"
)

plt.title(
    "Employees by City"
)

plt.xlabel(
    "City"
)

plt.ylabel(
    "Number of Employees"
)

plt.show()

This gives a quick view of category balance.

Step 17: Analyze Relationships Between Variables

Suppose you want to examine whether experience and salary are related.

Create a scatter plot:

plt.scatter(
    df["experience"],
    df["salary"]
)

plt.title(
    "Experience vs Salary"
)

plt.xlabel(
    "Years of Experience"
)

plt.ylabel(
    "Salary"
)

plt.show()

You may observe that higher experience tends to be associated with higher salary.

However, a visual relationship does not prove causation.

Step 18: Calculate Correlation

Correlation measures the strength and direction of a relationship between numerical variables.

Use:

correlation = df.corr(
    numeric_only=True
)

print(
    correlation
)

You might see something like:

            age   salary   experience
age         1.00    0.75        0.80
salary      0.75    1.00        0.92
experience  0.80    0.92        1.00

Correlation values usually range between:

-1 and 1

A value near:

1

suggests a strong positive relationship.

A value near:

-1

suggests a strong negative relationship.

A value near:

0

suggests little linear relationship.

Correlation does not automatically mean causation.

Step 19: Visualize Correlation

If you use Seaborn:

import seaborn as sns
import matplotlib.pyplot as plt

corr = df.corr(
    numeric_only=True
)

sns.heatmap(
    corr,
    annot=True
)

plt.title(
    "Correlation Matrix"
)

plt.show()

A correlation heatmap makes relationships easier to inspect.

Step 20: Detect Outliers

Outliers are unusually high or low values compared with the rest of the data.

One simple method is a box plot.

plt.boxplot(
    df["salary"]
)

plt.title(
    "Salary Box Plot"
)

plt.ylabel(
    "Salary"
)

plt.show()

Potential outliers may appear separately.

Do not automatically delete them.

An outlier may be:

• A data-entry error
• A rare but valid observation
• An important business event

Detecting Outliers Using IQR

The Interquartile Range, or IQR, is a common method.

Calculate quartiles:

q1 = df["salary"].quantile(
    0.25
)

q3 = df["salary"].quantile(
    0.75
)

iqr = q3 - q1

Calculate boundaries:

lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr

Find potential outliers:

outliers = df[
    (df["salary"] < lower) |
    (df["salary"] > upper)
]

print(
    outliers
)

This method identifies potential outliers but does not prove that those values are incorrect.

Step 21: Group Data

Grouping helps compare categories.

Example:

city_salary = (
    df.groupby("city")["salary"]
      .mean()
)

print(
    city_salary
)

You might get:

Delhi     58500
Jaipur    67500
Mumbai    44000

You can visualize:

city_salary.plot(
    kind="bar"
)

plt.title(
    "Average Salary by City"
)

plt.ylabel(
    "Average Salary"
)

plt.show()

Step 22: Compare Multiple Statistics

Use:

summary = (
    df.groupby("city")["salary"]
      .agg(
          ["mean", "min", "max"]
      )
)

print(
    summary
)

This helps compare different groups more thoroughly.

Step 23: Sort Values

You can inspect the highest salaries:

highest_salary = (
    df.sort_values(
        "salary",
        ascending=False
    )
)

print(
    highest_salary
)

Or show the top three:

print(
    highest_salary.head(3)
)

This is useful for quickly identifying extreme values.

Step 24: Filter the Dataset

For example, find employees with salary above 50,000:

high_salary = df[
    df["salary"] > 50000
]

print(
    high_salary
)

Multiple conditions:

result = df[
    (df["salary"] > 50000) &
    (df["experience"] >= 4)
]

print(
    result
)

Filtering is useful for exploring specific subgroups.

Step 25: Check Numerical Relationships by Group

Suppose you want average salary by city:

print(
    df.groupby("city")["salary"]
      .mean()
)

Or average experience:

print(
    df.groupby("city")["experience"]
      .mean()
)

Group-level analysis can reveal patterns hidden in the full dataset.

Practical EDA Example with CSV Data

Suppose you have:

customers.csv

Load it:

import pandas as pd

df = pd.read_csv(
    "customers.csv"
)

Start inspection:

print(
    df.head()
)

print(
    df.shape
)

df.info()

Check missing values:

print(
    df.isnull().sum()
)

Check duplicates:

print(
    df.duplicated().sum()
)

Generate statistics:

print(
    df.describe()
)

Inspect category counts:

print(
    df["city"].value_counts()
)

Then visualize important variables.

For example:

import matplotlib.pyplot as plt

plt.hist(
    df["age"].dropna()
)

plt.title(
    "Customer Age Distribution"
)

plt.xlabel(
    "Age"
)

plt.ylabel(
    "Frequency"
)

plt.show()

This is a basic but realistic EDA workflow.

Complete EDA Example

Consider this dataset:

import pandas as pd
import matplotlib.pyplot as plt

data = {
    "age": [
        22,
        25,
        28,
        32,
        35,
        40,
        None
    ],
    "income": [
        30000,
        42000,
        50000,
        62000,
        70000,
        85000,
        55000
    ],
    "purchased": [
        "No",
        "No",
        "Yes",
        "Yes",
        "Yes",
        "Yes",
        "No"
    ]
}

df = pd.DataFrame(data)

Inspect:

print(
    df.head()
)

print(
    df.shape
)

df.info()

Check missing values:

print(
    df.isnull().sum()
)

Fill age:

df["age"] = df["age"].fillna(
    df["age"].median()
)

Check target distribution:

print(
    df["purchased"].value_counts()
)

Visualize income:

plt.hist(
    df["income"],
    bins=5
)

plt.title(
    "Income Distribution"
)

plt.xlabel(
    "Income"
)

plt.ylabel(
    "Frequency"
)

plt.show()

Visualize age and income:

plt.scatter(
    df["age"],
    df["income"]
)

plt.title(
    "Age vs Income"
)

plt.xlabel(
    "Age"
)

plt.ylabel(
    "Income"
)

plt.show()

This gives you a better understanding of the dataset before modeling.

Univariate Analysis

Univariate analysis examines one variable at a time.

Examples include:

df["age"].describe()

df["city"].value_counts()

plt.hist(
    df["salary"]
)

Typical questions include:

• What is the average?
• What is the distribution?
• What values are most common?
• Are there outliers?

Bivariate Analysis

Bivariate analysis examines two variables together.

Example:

plt.scatter(
    df["experience"],
    df["salary"]
)

Or:

df.groupby(
    "city"
)["salary"].mean()

Typical questions include:

• Does experience relate to salary?
• Does salary differ by city?
• Do two numerical variables move together?

Multivariate Analysis

Multivariate analysis examines three or more variables.

For example:

age
income
experience
salary

You might calculate:

df.corr(
    numeric_only=True
)

or compare grouped statistics across several columns.

Multivariate analysis becomes increasingly important as datasets grow more complex.

EDA for Categorical Variables

Categorical variables represent groups.

Examples include:

city
gender
department
product_category
status

Useful commands include:

df["city"].unique()

df["city"].nunique()

df["city"].value_counts()

Visualization:

df["city"].value_counts().plot(
    kind="bar"
)

plt.show()

EDA for Numerical Variables

Numerical variables include:

age
salary
price
temperature
experience

Useful commands include:

df["salary"].mean()

df["salary"].median()

df["salary"].std()

df["salary"].describe()

Useful charts include:

• Histograms
• Box plots
• Scatter plots
• Line charts

Checking Class Balance

In classification machine learning problems, the target classes may be unbalanced.

Suppose:

print(
    df["purchased"].value_counts()
)

Output:

No     950
Yes     50

This is a highly imbalanced target.

You can calculate proportions:

print(
    df["purchased"]
    .value_counts(
        normalize=True
    )
)

Class imbalance can affect model evaluation and training decisions.

Visualizing Class Balance

Use:

df["purchased"].value_counts().plot(
    kind="bar"
)

plt.title(
    "Target Class Distribution"
)

plt.xlabel(
    "Purchased"
)

plt.ylabel(
    "Count"
)

plt.show()

This makes imbalance easier to see.

EDA Before Machine Learning

Before training a model, EDA helps you understand:

• Which columns can be used as features
• Which column is the target
• Whether missing values exist
• Whether categories need encoding
• Whether features have extreme values
• Whether classes are imbalanced
• Whether variables may be redundant
• Whether suspicious leakage exists

A simplified workflow is:

Dataset
   ↓
EDA
   ↓
Data Cleaning
   ↓
Feature Engineering
   ↓
Train/Test Split
   ↓
Preprocessing
   ↓
Model Training
   ↓
Evaluation

In practice, EDA and cleaning often happen iteratively rather than in a perfectly fixed order.

EDA and Data Leakage

When exploring machine learning datasets, be careful not to use information that would not be available during real predictions.

For example, a column such as:

final_result

might directly reveal the target.

Using it as a feature would create leakage.

EDA can help detect suspicious columns before training.

Common EDA Mistakes to Avoid

1. Building a Model Before Understanding the Data

Do not immediately call a machine learning algorithm after loading the dataset.

Inspect it first.

2. Looking Only at df.head()

The first five rows are not enough to understand an entire dataset.

Also inspect:

df.info()
df.describe()
df.isnull().sum()

3. Ignoring Missing Values

Missing values can affect statistics, charts, and models.

Always measure them.

4. Automatically Removing Outliers

Outliers may contain valuable information.

Investigate before deleting them.

5. Ignoring Categorical Variables

EDA is not only about numbers.

Categories such as city, product, and department can reveal important patterns.

6. Treating Correlation as Causation

If income and spending are correlated, this does not automatically prove that one causes the other.

7. Creating Too Many Charts Without a Question

Every chart should help answer something.

Do not generate visualizations only because a library makes them easy to create.

8. Ignoring Duplicates

Duplicate records can distort counts and averages.

Check:

df.duplicated().sum()

9. Ignoring Data Types

A numerical column stored as text may produce incorrect analysis.

Always inspect:

df.dtypes

10. Using the Entire Dataset for Preprocessing Decisions

For machine learning, transformations that learn parameters from data should generally be fitted using training data rather than the entire dataset.

This helps prevent data leakage.

Best Practices for EDA

Start with simple questions.

Ask:

What does each row represent?
What does each column represent?
Which variable is the target?
What data is missing?
Which values look unusual?

Inspect before modifying.

Useful first commands are:

df.head()
df.shape
df.info()
df.describe()
df.isnull().sum()
df.duplicated().sum()

For categorical columns:

df["column"].value_counts()

For numerical columns:

df["column"].describe()

Use visualizations to support your analysis.

Keep charts simple and clearly labeled.

Document important assumptions and cleaning decisions.

Preserve the raw dataset so that your analysis remains reproducible.

How EDA Connects to Becoming an AI Developer

EDA is one of the most practical skills in AI and machine learning.

A model is only one part of the workflow.

Before training, you need to understand:

• Where data comes from
• Whether it is reliable
• Whether it contains missing values
• Whether features are useful
• Whether classes are balanced
• Whether relationships make sense
• Whether suspicious leakage exists

A useful AI learning path is:

Python
   ↓
NumPy
   ↓
Pandas
   ↓
Data Cleaning
   ↓
Data Visualization
   ↓
Exploratory Data Analysis
   ↓
Statistics
   ↓
scikit-learn
   ↓
Machine Learning
   ↓
Deep Learning

EDA teaches you to understand data before asking an algorithm to learn from it.

What to Learn Next

After learning Exploratory Data Analysis with Python, continue with:

1. Statistics for data science
2. Mean, median, mode, and variance
3. Probability basics
4. Correlation and covariance
5. Outlier detection
6. Feature engineering
7. Categorical data encoding
8. Feature scaling
9. Train/test splitting
10. scikit-learn
11. Regression
12. Classification
13. Model evaluation
14. Machine learning pipelines

A strong progression is:

Python → NumPy → Pandas → Matplotlib → EDA → Statistics → scikit-learn → Machine Learning → Deep Learning

Frequently Asked Questions

1. What is Exploratory Data Analysis in Python?

Exploratory Data Analysis is the process of inspecting, summarizing, cleaning, and visualizing a dataset to understand its structure, patterns, relationships, and problems.

2. Which Python libraries are used for EDA?

Common libraries include Pandas, NumPy, Matplotlib, and Seaborn.

3. What should I check first during EDA?

A good starting point is:

df.head()
df.shape
df.info()
df.describe()
df.isnull().sum()

These commands provide a quick overview of the dataset.

4. Is data cleaning part of EDA?

Yes. EDA and data cleaning are closely related. During exploration, you often discover missing values, duplicates, incorrect data types, and unusual records that require cleaning.

5. What is the difference between EDA and data visualization?

Data visualization is one part of EDA. EDA also includes statistics, data inspection, missing-value analysis, duplicate checks, filtering, and relationship analysis.

6. Is EDA necessary before machine learning?

Yes. EDA helps you understand the dataset and identify issues before model training. Skipping it can lead to poor feature choices, leakage, incorrect preprocessing, or misleading model results.

7. What is correlation in EDA?

Correlation measures how strongly two numerical variables move together. It is useful for exploring relationships, but correlation does not prove causation.

8. Can EDA detect outliers?

Yes. Outliers can be investigated using summary statistics, histograms, box plots, IQR methods, and other techniques.

Conclusion

Exploratory Data Analysis is the process of understanding a dataset before making conclusions or training machine learning models.

With Python, you can use Pandas to inspect and summarize data and Matplotlib or Seaborn to visualize important patterns.

A good EDA process checks dataset structure, missing values, duplicates, data types, numerical distributions, categorical values, correlations, outliers, and relationships between variables.

The most important goal is not to create as many charts as possible. It is to understand what the data represents and whether it is suitable for further analysis.

Once you are comfortable with EDA, the next major topic to learn is statistics for data science and machine learning.

---