Data cleaning is the process of finding and fixing problems in a dataset before you analyze it or use it for machine learning.

Real-world data is rarely perfect. A CSV or Excel file may contain missing values, duplicate rows, incorrect data types, inconsistent text, invalid numbers, extra spaces, unusual date formats, or unnecessary columns.

Pandas makes these problems much easier to handle.

For example, suppose you load customer data like this:

import pandas as pd

df = pd.read_csv("customers.csv")

The file may look normal at first, but after inspection you could discover:

• Missing ages
• Duplicate customers
• Salaries stored as text
• City names written in different formats
• Extra spaces around names
• Invalid dates

Before performing analysis or training a machine learning model, you should clean these issues carefully.

This guide explains data cleaning with Pandas step by step, using simple examples that beginners can follow and reuse.

What Is Data Cleaning?

Data cleaning means identifying and correcting problems in raw data so that it becomes more accurate, consistent, and useful.

For example, consider this dataset:

name       age    city       salary
Aman       25     Jaipur     45000
Riya       NaN    Delhi      52000
 aman      25     jaipur     45000
Rahul      23     Mumbai     unknown

There are several issues:

• One age is missing.
• Aman appears twice with inconsistent spacing.
• Jaipur and jaipur use different capitalization.
• One salary contains text instead of a number.

A cleaned version might look like:

name     age    city      salary
Aman     25     Jaipur    45000
Riya     25     Delhi     52000
Rahul    23     Mumbai    40000

The exact cleaning decisions depend on the dataset and your goals.

Why Is Data Cleaning Important?

Poor-quality data can lead to poor analysis and unreliable machine learning results.

For example, imagine calculating an average salary when some salary values are stored as:

45000
52000
"unknown"
40000

You cannot treat "unknown" as a normal number.

Similarly, duplicate rows may make one customer or transaction appear more frequently than it really does.

Data cleaning helps you:

• Improve data consistency
• Reduce errors
• Prepare datasets for analysis
• Detect missing information
• Correct data types
• Remove or investigate duplicates
• Standardize text
• Prepare machine learning features
• Avoid misleading results

For AI development, data cleaning is often one of the most important parts of the entire workflow.

Prerequisites

Before starting, you should understand:

• Basic Python
• Pandas DataFrames
• How to read CSV or Excel files

Install Pandas if necessary:

pip install pandas

Then import it:

import pandas as pd

Create a Sample Dirty Dataset

We will use this dataset throughout several examples:

import pandas as pd

data = {
    "name": [
        " Aman ",
        "Riya",
        "Rahul",
        "aman",
        "Neha"
    ],
    "age": [
        25,
        None,
        23,
        25,
        30
    ],
    "city": [
        "Jaipur",
        "Delhi",
        "Mumbai",
        "jaipur",
        " Delhi "
    ],
    "salary": [
        "45000",
        "52000",
        "unknown",
        "45000",
        "65000"
    ]
}

df = pd.DataFrame(data)

print(df)

This dataset contains several common problems:

• Missing age
• Extra spaces
• Inconsistent capitalization
• Duplicate-like records
• Salary stored as text
• Invalid salary value

We will clean these issues step by step.

Step 1: Inspect the Dataset

Before changing anything, understand the data.

Use:

print(df.head())

Check the shape:

print(df.shape)

Check column names:

print(df.columns)

Check data types:

print(df.dtypes)

And inspect overall information:

df.info()

These commands help you understand the current structure before making decisions.

Check Summary Statistics

For numerical columns:

print(df.describe())

This can reveal unusual minimums, maximums, or missing values.

However, if a column that should be numerical is stored as text, describe() may not treat it as a normal numeric column yet.

Step 2: Find Missing Values

Missing values are common in real datasets.

Pandas usually represents missing values with values such as NaN.

Check missing values:

print(df.isnull())

A more useful version is:

print(df.isnull().sum())

Example:

name      0
age       1
city      0
salary    0
dtype: int64

This tells you that the age column contains one missing value.

You can also use:

print(df.isna().sum())

isna() and isnull() are equivalent in Pandas.

Step 3: Decide How to Handle Missing Values

There is no single correct way to handle missing data.

Your options usually include:

• Remove rows
• Remove columns
• Fill values
• Keep missing values
• Use a statistical or domain-specific replacement

The correct method depends on why the data is missing.

Remove Rows with Missing Values

Use:

clean_df = df.dropna()

This removes rows containing missing values.

You can also modify the DataFrame:

df = df.dropna()

Be careful. Removing rows may discard useful data.

Remove Rows Only When Specific Columns Are Missing

For example:

df = df.dropna(
    subset=["age"]
)

This removes only rows where age is missing.

Fill Missing Values with a Fixed Value

Example:

df["age"] = df["age"].fillna(0)

This may make sense in some situations, but using zero for age would usually be unrealistic.

Fill with the Mean

You could use:

df["age"] = df["age"].fillna(
    df["age"].mean()
)

If the available ages are:

25, 23, 25, 30

Pandas calculates the mean and uses it for the missing value.

Fill with the Median

The median can be useful when a numerical column contains extreme values.

df["age"] = df["age"].fillna(
    df["age"].median()
)

Fill with the Mode

The mode is the most frequent value.

For categorical data:

df["city"] = df["city"].fillna(
    df["city"].mode()[0]
)

Use this only when it makes sense for your data.

Step 4: Remove Duplicate Rows

Check duplicate rows:

print(df.duplicated())

Count them:

print(df.duplicated().sum())

Remove exact duplicates:

df = df.drop_duplicates()

However, our sample dataset contains:

" Aman "

and:

"aman"

These are not technically identical until the text is standardized.

That is why the order of cleaning operations can matter.

Step 5: Clean Extra Spaces from Text

Extra spaces are common when data comes from forms, spreadsheets, or manually entered records.

Suppose:

" Aman "
" Delhi "

Use:

df["name"] = df["name"].str.strip()
df["city"] = df["city"].str.strip()

str.strip() removes whitespace from the beginning and end of strings.

You can also use:

.str.lstrip()

to remove spaces from the left, or:

.str.rstrip()

to remove spaces from the right.

Step 6: Standardize Text Capitalization

Text values may appear in inconsistent forms.

For example:

Jaipur
jaipur
JAIPUR

These may refer to the same city.

You can standardize them:

df["city"] = df["city"].str.title()

Now:

jaipur

becomes:

Jaipur

Similarly, names can be standardized:

df["name"] = df["name"].str.title()

Other useful options include:

df["city"].str.lower()
df["city"].str.upper()

Step 7: Detect Duplicate Records After Text Cleaning

After standardizing spaces and capitalization:

df["name"] = df["name"].str.strip().str.title()
df["city"] = df["city"].str.strip().str.title()

you may discover new duplicates.

Now check again:

print(df.duplicated())

Then remove them:

df = df.drop_duplicates()

This is often more accurate than checking duplicates before normalizing text.

Step 8: Convert Data Types

Incorrect data types are a major cleaning problem.

Check:

print(df.dtypes)

Our salary values look numerical but are stored as strings because one value contains:

unknown

You could try:

df["salary"] = pd.to_numeric(
    df["salary"]
)

But this raises an error when it reaches "unknown".

A better approach is:

df["salary"] = pd.to_numeric(
    df["salary"],
    errors="coerce"
)

errors="coerce" converts invalid values into missing values.

Now:

unknown

becomes:

NaN

You can then handle that missing salary separately.

For example:

df["salary"] = df["salary"].fillna(
    df["salary"].median()
)

Converting Columns to Integers

After cleaning, you may want:

df["age"] = df["age"].astype(int)

However, regular integer conversion will fail if missing values remain.

Pandas also supports nullable integer types:

df["age"] = df["age"].astype("Int64")

This can represent both integers and missing values.

Converting Text Columns

You can explicitly use:

df["name"] = df["name"].astype("string")

This can make the intended type clearer.

Step 9: Clean Date Columns

Dates are another common source of messy data.

Suppose:

df = pd.DataFrame({
    "join_date": [
        "2025-01-15",
        "2025/02/10",
        "invalid",
        "2025-04-20"
    ]
})

Convert to datetime:

df["join_date"] = pd.to_datetime(
    df["join_date"],
    errors="coerce"
)

Invalid dates become missing datetime values.

You can then check:

print(df["join_date"])

And handle missing dates according to your use case.

Step 10: Rename Columns

Datasets often contain poor column names.

For example:

Employee Name
Annual Salary
Joining Date

You may prefer:

employee_name
annual_salary
joining_date

Rename columns:

df = df.rename(
    columns={
        "Employee Name": "employee_name",
        "Annual Salary": "annual_salary"
    }
)

You can also standardize every column name:

df.columns = (
    df.columns
      .str.strip()
      .str.lower()
      .str.replace(" ", "_")
)

This turns:

Employee Name

into:

employee_name

Step 11: Remove Unnecessary Columns

Real datasets sometimes contain columns that are irrelevant to your analysis.

Remove one:

df = df.drop(
    columns=["temporary_column"]
)

Remove several:

df = df.drop(
    columns=[
        "unused_column",
        "notes"
    ]
)

For machine learning, removing irrelevant features can simplify preprocessing.

Step 12: Replace Incorrect or Inconsistent Values

Sometimes values are not missing, but they are inconsistent.

Example:

Male
male
M
Female
F
female

You can standardize them:

df["gender"] = df["gender"].replace({
    "M": "Male",
    "male": "Male",
    "F": "Female",
    "female": "Female"
})

This creates a consistent set of categories.

Replacing Specific Values

Example:

df["status"] = df["status"].replace(
    "N/A",
    pd.NA
)

You can then handle that value as missing data.

Step 13: Detect Invalid Numerical Values

A numerical value can be valid in Python but unrealistic in the real world.

Suppose:

age
25
30
-4
250

Technically, all are numbers.

But -4 and 250 are likely invalid ages.

Filter them:

invalid_age = df[
    (df["age"] < 0) |
    (df["age"] > 120)
]

print(invalid_age)

Then decide whether to:

• Correct them
• Remove them
• Mark them missing
• Investigate the original source

For example:

df.loc[
    (df["age"] < 0) |
    (df["age"] > 120),
    "age"
] = pd.NA

Step 14: Detect Outliers

An outlier is a value that is unusually far from most other values.

For example:

salary
40000
45000
48000
50000
900000

The 900000 value might be legitimate or might be an error.

Do not automatically remove outliers.

First investigate whether they make sense.

You can inspect basic statistics:

print(df["salary"].describe())

You can also calculate quartiles and the interquartile range.

q1 = df["salary"].quantile(0.25)
q3 = df["salary"].quantile(0.75)

iqr = q3 - q1

Then define potential outlier boundaries:

lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr

outliers = df[
    (df["salary"] < lower) |
    (df["salary"] > upper)
]

print(outliers)

This is a useful detection method, but it does not automatically prove that those values are wrong.

Step 15: Clean Boolean Values

Suppose a column contains:

Yes
No
yes
Y
N
1
0

You can standardize values:

df["active"] = df["active"].replace({
    "Yes": True,
    "yes": True,
    "Y": True,
    "1": True,
    "No": False,
    "N": False,
    "0": False
})

Consistency makes later analysis easier.

Step 16: Clean Category Values

Suppose city values include:

New Delhi
Delhi
delhi
Delhi NCR

You first need to decide whether they should represent the same category.

Then standardize them:

df["city"] = df["city"].replace({
    "New Delhi": "Delhi",
    "delhi": "Delhi",
    "Delhi NCR": "Delhi"
})

Domain knowledge is important here.

Pandas can perform the replacement, but it cannot always know whether two values should mean the same thing.

Step 17: Check Unique Values

A useful way to discover inconsistent categories is:

print(df["city"].unique())

Count unique values:

print(df["city"].nunique())

See how often each value appears:

print(df["city"].value_counts())

This is especially useful for categorical columns such as:

• City
• Gender
• Department
• Product category
• Status

Step 18: Reset the Index

After removing rows, your DataFrame index may look like:

0
2
4
7

Reset it:

df = df.reset_index(
    drop=True
)

Now the index becomes:

0
1
2
3

Complete Data Cleaning Example

Let’s clean our original sample dataset.

import pandas as pd

data = {
    "name": [
        " Aman ",
        "Riya",
        "Rahul",
        "aman",
        "Neha"
    ],
    "age": [
        25,
        None,
        23,
        25,
        30
    ],
    "city": [
        "Jaipur",
        "Delhi",
        "Mumbai",
        "jaipur",
        " Delhi "
    ],
    "salary": [
        "45000",
        "52000",
        "unknown",
        "45000",
        "65000"
    ]
}

df = pd.DataFrame(data)

First clean text:

df["name"] = (
    df["name"]
    .str.strip()
    .str.title()
)

df["city"] = (
    df["city"]
    .str.strip()
    .str.title()
)

Convert salary:

df["salary"] = pd.to_numeric(
    df["salary"],
    errors="coerce"
)

Fill missing age:

df["age"] = df["age"].fillna(
    df["age"].median()
)

Fill missing salary:

df["salary"] = df["salary"].fillna(
    df["salary"].median()
)

Remove duplicates:

df = df.drop_duplicates()

Reset index:

df = df.reset_index(
    drop=True
)

Inspect the result:

print(df)

A cleaned result may look like:

    name   age    city   salary
0   Aman  25.0  Jaipur  45000.0
1   Riya  25.0   Delhi  52000.0
2  Rahul  23.0  Mumbai  48500.0
3   Neha  30.0   Delhi  65000.0

The exact replacement values depend on the data.

Cleaning Data from a CSV File

A typical workflow might look like:

import pandas as pd

df = pd.read_csv("customers.csv")

print(df.head())
df.info()

print(df.isnull().sum())

df["name"] = (
    df["name"]
    .str.strip()
    .str.title()
)

df["age"] = pd.to_numeric(
    df["age"],
    errors="coerce"
)

df["age"] = df["age"].fillna(
    df["age"].median()
)

df = df.drop_duplicates()

df.to_csv(
    "customers_clean.csv",
    index=False
)

This creates a cleaned version while preserving the original file.

Data Cleaning Workflow for Beginners

A practical workflow is:

Load Data
   ↓
Inspect Structure
   ↓
Check Missing Values
   ↓
Check Data Types
   ↓
Clean Text
   ↓
Fix Invalid Values
   ↓
Handle Missing Data
   ↓
Check Duplicates
   ↓
Validate Ranges
   ↓
Standardize Categories
   ↓
Review Results
   ↓
Save Clean Data

Do not treat this as a rigid order for every dataset.

Some tasks need to happen earlier or later depending on the data.

Data Cleaning for Machine Learning

Data cleaning is especially important in machine learning.

Suppose your dataset contains:

df = pd.DataFrame({
    "age": [22, 25, None, 30],
    "income": [
        "30000",
        "45000",
        "unknown",
        "60000"
    ],
    "purchased": [
        "Yes",
        "No",
        "Yes",
        "No"
    ]
})

Before training a model, you may need to:

1. Fill missing age values.
2. Convert income to numeric values.
3. Handle invalid income entries.
4. Encode Yes and No.
5. Separate features and target.

For example:

df["income"] = pd.to_numeric(
    df["income"],
    errors="coerce"
)

Then:

df["income"] = df["income"].fillna(
    df["income"].median()
)

Fill age:

df["age"] = df["age"].fillna(
    df["age"].median()
)

Convert the target:

df["purchased"] = df["purchased"].map({
    "Yes": 1,
    "No": 0
})

Select features:

X = df[
    ["age", "income"]
]

Select target:

y = df["purchased"]

Now the dataset is much closer to being ready for machine learning.

Data Cleaning vs Data Preprocessing

These terms are related but not identical.

Data cleaning focuses on fixing problems such as:

• Missing values
• Invalid records
• Duplicates
• Wrong data types
• Inconsistent text

Data preprocessing is broader and may also include:

• Scaling
• Normalization
• Encoding categories
• Feature engineering
• Train/test splitting
• Feature selection

Data cleaning is often an early part of data preprocessing.

Advantages of Data Cleaning with Pandas

Pandas provides several benefits for cleaning data.

Easy Missing-Value Handling

Functions such as:

isna()
dropna()
fillna()

make missing-data operations convenient.

Strong Text Processing

Pandas provides:

.str.strip()
.str.lower()
.str.upper()
.str.replace()

for cleaning text columns.

Data Type Conversion

Functions such as:

pd.to_numeric()
pd.to_datetime()
astype()

help standardize column types.

Duplicate Detection

Use:

duplicated()
drop_duplicates()

to identify and handle repeated rows.

Easy Filtering

You can find invalid records using conditions.

Example:

df[df["age"] < 0]

Good Integration with Machine Learning

Clean DataFrames can be passed into tools such as scikit-learn after appropriate preprocessing.

Limitations of Pandas for Data Cleaning

Pandas is powerful, but it cannot automatically understand every data problem.

For example, it cannot know whether:

Delhi
New Delhi
Delhi NCR

should always be treated as the same category.

That requires domain knowledge.

Other limitations include:

• Large datasets may use significant memory.
• Automated replacements can introduce errors.
• Outliers are not always mistakes.
• Missing values should not always be filled.
• Duplicate rows may sometimes be legitimate.
• Cleaning rules need to be validated against the real meaning of the data.

Common Mistakes to Avoid

1. Dropping Every Missing Row

Avoid immediately doing:

df = df.dropna()

without understanding how much information will be lost.

2. Filling Missing Values with Zero Automatically

Zero may have a real meaning.

For example, an income of zero is different from an unknown income.

3. Removing Every Duplicate

Some repeated rows may represent legitimate repeated events.

Understand what each row represents first.

4. Changing Data Types Without Checking Values

This:

df["salary"].astype(int)

may fail if values such as "unknown" exist.

Use inspection or safer conversion first.

5. Removing Outliers Automatically

An unusually large salary, transaction, or purchase might be completely valid.

Investigate outliers before deleting them.

6. Cleaning the Original File Directly

Preserve your raw data whenever possible.

Instead of overwriting:

customers.csv

save:

customers_clean.csv

7. Ignoring Category Inconsistency

Values such as:

IT
it
I.T.
Information Technology

may need standardization before grouping or analysis.

8. Cleaning Test Data Using Information from the Full Dataset

This is particularly important in machine learning.

For example, if you calculate a median using the entire dataset before splitting training and test data, information from the test set may influence training preparation.

This is called data leakage.

Preprocessing values that are learned from data should generally be fitted using training data only.

Best Practices for Data Cleaning

Start with inspection before modification.

Useful commands include:

df.head()
df.shape
df.info()
df.describe()
df.isnull().sum()
df.duplicated().sum()

Check categorical values:

df["column"].value_counts()

Preserve the raw dataset.

Document the cleaning decisions you make.

For example:

# Convert invalid salary entries to missing values
df["salary"] = pd.to_numeric(
    df["salary"],
    errors="coerce"
)

Use reproducible code instead of manually editing cells in a spreadsheet.

Validate the DataFrame after cleaning.

For example:

print(df.isnull().sum())
print(df.dtypes)
print(df.shape)

For machine learning, avoid data leakage by separating training and test data at the correct stage and learning preprocessing parameters only from training data.

How Data Cleaning Connects to Becoming an AI Developer

AI development is not only about building models.

A large part of practical machine learning involves understanding and preparing data.

A typical workflow looks like:

Raw Data
   ↓
Pandas
   ↓
Data Cleaning
   ↓
Exploratory Data Analysis
   ↓
Feature Engineering
   ↓
Preprocessing
   ↓
Train/Test Split
   ↓
Machine Learning Model
   ↓
Evaluation

If your dataset contains incorrect values, missing information, or inconsistent formats, even a powerful model may learn from poor-quality inputs.

Learning data cleaning teaches you how to understand the information that goes into AI systems instead of simply passing datasets into algorithms.

What to Learn Next

After learning basic data cleaning with Pandas, continue with:

1. Pandas indexing and filtering
2. Missing value handling in depth
3. Pandas groupby()
4. Merging and joining DataFrames
5. Exploratory data analysis
6. Matplotlib
7. Seaborn concepts for data exploration
8. Statistics for machine learning
9. Feature engineering
10. Categorical encoding
11. Feature scaling
12. Train/test splitting
13. scikit-learn pipelines
14. Machine learning fundamentals

A useful path is:

Python → NumPy → Pandas → Data Cleaning → EDA → Statistics → scikit-learn → Machine Learning → Deep Learning

Frequently Asked Questions

1. What is data cleaning in Pandas?

Data cleaning in Pandas means using DataFrame operations to detect and fix problems such as missing values, duplicates, inconsistent text, invalid data types, and incorrect records.

2. How do I find missing values in Pandas?

Use:

df.isnull().sum()

or:

df.isna().sum()

These show the number of missing values in each column.

3. How do I remove duplicate rows?

Use:

df = df.drop_duplicates()

Check duplicates first with:

df.duplicated().sum()

4. How do I fill missing values?

Use fillna().

For example:

df["age"] = df["age"].fillna(
    df["age"].median()
)

Choose the replacement method based on what makes sense for the data.

5. How do I convert text to numbers in Pandas?

Use:

df["salary"] = pd.to_numeric(
    df["salary"],
    errors="coerce"
)

Invalid values become missing values when errors="coerce" is used.

6. How do I remove spaces from Pandas text columns?

Use:

df["name"] = df["name"].str.strip()

This removes spaces from both ends of each string.

7. Should I remove all outliers from a dataset?

No. Outliers can be errors, but they can also represent valid and important observations. Investigate them before deciding whether to remove or transform them.

8. Why is data cleaning important for machine learning?

Machine learning models learn patterns from the data you provide. Incorrect, inconsistent, or poorly prepared data can produce misleading patterns and weaker model performance.

Conclusion

Data cleaning with Pandas is one of the most practical skills you can learn for data analysis and machine learning.

Start by inspecting your dataset, checking missing values, reviewing data types, cleaning text, detecting duplicates, validating numerical ranges, standardizing categories, and investigating unusual values.

The most important rule is not to clean data blindly. Every change should have a clear reason based on what the data represents.

Once you are comfortable cleaning datasets with Pandas, your next step should be exploratory data analysis, visualization, and machine learning preprocessing.

---