Missing data is one of the most common problems you will face when working with real-world datasets in Python.

A dataset may contain empty cells, None, NaN, blank strings, "unknown", "N/A", or other values that represent missing information.

For example:

name     age    salary
Aman     25     45000
Riya     NaN    52000
Rahul    23     NaN

Before analyzing this data or using it for machine learning, you need to decide how to handle those missing values.

Python provides several ways to work with missing data. For small programs, you can use built-in Python checks. For structured datasets, Pandas is usually the most practical tool.

This guide explains how to detect, remove, replace, and manage missing data in Python with beginner-friendly examples.

What Is Missing Data?

Missing data means a value that should exist in a dataset is unavailable or unknown.

For example:

Name: Riya
Age: missing
City: Delhi

The missing age may be represented in different ways depending on the source.

Common representations include:

None

float("nan")

And inside datasets:

NaN
N/A
NULL
unknown
-
blank cell

These values should not always be treated the same way.

Understanding what the missing value means is an important part of data cleaning.

Why Does Missing Data Occur?

Missing data can appear for many reasons.

Common examples include:

• A user skips a form field
• A sensor fails to record a value
• A database field is optional
• Data is lost during import
• A CSV cell is empty
• Information is unavailable
• Different systems use different formats
• A calculation fails
• Historical records are incomplete

Before deciding how to handle missing values, try to understand why they are missing.

Why Is Missing Data Important?

Missing data can cause problems in:

• Data analysis
• Statistical calculations
• Charts and reports
• Machine learning models
• Business decisions
• Data pipelines

For example, suppose you calculate an average age:

20
25
missing
30

You need to decide whether the missing age should be ignored, estimated, removed, or handled another way.

There is no single correct method for every dataset.

Missing Data in Plain Python

Before learning Pandas methods, it helps to understand missing values in normal Python.

Using None

Python commonly uses None to represent the absence of a value.

Example:

age = None

print(age)

Output:

None

Check whether a value is None:

if age is None:
    print("Age is missing")

Output:

Age is missing

Use:

is None

rather than:

== None

for clear Python code.

Missing Values Inside Python Lists

Suppose:

ages = [25, None, 30, None, 22]

You can find missing values:

for age in ages:
    if age is None:
        print("Missing value found")

Or remove them:

clean_ages = [
    age for age in ages
    if age is not None
]

print(clean_ages)

Output:

[25, 30, 22]

This works well for small Python programs.

For large tabular datasets, Pandas provides more convenient tools.

Understanding NaN in Python

NaN means Not a Number.

It is commonly used to represent missing numerical values.

Create one:

value = float("nan")

print(value)

Output:

nan

A surprising property of NaN is:

value == value

returns:

False

So you should not normally check NaN using equality.

Instead, use tools designed for missing-value detection.

Using math.isnan()

For a floating-point NaN:

import math

value = float("nan")

print(math.isnan(value))

Output:

True

However, math.isnan() expects a numerical value and is not convenient for mixed datasets.

For data analysis, Pandas provides better missing-value handling.

How Pandas Represents Missing Data

Pandas commonly uses:

NaN

for missing numerical values.

It can also work with:

None

and:

pd.NA

Example:

import pandas as pd

data = {
    "name": ["Aman", "Riya", "Rahul"],
    "age": [25, None, 23]
}

df = pd.DataFrame(data)

print(df)

Output may look like:

    name   age
0   Aman  25.0
1   Riya   NaN
2  Rahul  23.0

Pandas recognizes the missing age automatically.

How to Find Missing Values with Pandas

Use:

df.isnull()

Example:

import pandas as pd

df = pd.DataFrame({
    "name": ["Aman", "Riya", "Rahul"],
    "age": [25, None, 23],
    "salary": [45000, 52000, None]
})

print(df.isnull())

Output:

    name    age  salary
0  False  False   False
1  False   True   False
2  False  False    True

True means the value is missing.

Counting Missing Values

A more useful command is:

print(df.isnull().sum())

Example output:

name      0
age       1
salary    1
dtype: int64

This tells you how many missing values exist in each column.

You can also use:

df.isna().sum()

isna() and isnull() are equivalent in Pandas.

Finding the Total Number of Missing Values

Use:

total_missing = df.isnull().sum().sum()

print(total_missing)

If the output is:

2

the entire DataFrame contains two missing cells.

Finding Rows with Missing Values

You can select rows containing at least one missing value:

missing_rows = df[
    df.isnull().any(axis=1)
]

print(missing_rows)

This is useful when you want to inspect problematic records before deciding what to do.

Finding Rows Without Missing Values

Use:

complete_rows = df[
    df.notnull().all(axis=1)
]

print(complete_rows)

These rows contain values in every column.

Removing Missing Data with dropna()

One option is to remove rows containing missing values.

Use:

clean_df = df.dropna()

Example:

import pandas as pd

df = pd.DataFrame({
    "name": ["Aman", "Riya", "Rahul"],
    "age": [25, None, 23]
})

clean_df = df.dropna()

print(clean_df)

Output:

    name   age
0   Aman  25.0
2  Rahul  23.0

The row containing the missing age is removed.

Should You Always Remove Missing Rows?

No.

Removing missing rows can result in valuable information being lost.

For example, a customer record may have:

name       = Riya
age        = missing
city       = Delhi
purchases  = 12

Deleting the entire row because age is missing may remove useful purchase information.

Always evaluate how important the missing field is.

Remove Rows Based on Specific Columns

You can remove rows only when a particular column is missing.

df = df.dropna(
    subset=["age"]
)

This removes rows where age is missing but keeps rows with missing values in other columns.

For multiple columns:

df = df.dropna(
    subset=["age", "salary"]
)

Removing Columns with Missing Values

You can remove columns instead:

df = df.dropna(
    axis=1
)

This removes any column containing at least one missing value.

Be careful because this can remove important features.

Requiring a Minimum Number of Valid Values

Suppose a row should contain at least three non-missing values.

Use:

df = df.dropna(
    thresh=3
)

Rows with fewer than three non-missing values are removed.

Filling Missing Values with fillna()

Instead of deleting missing values, you can replace them.

Use:

df["age"] = df["age"].fillna(0)

This replaces missing ages with zero.

However, zero is not always meaningful.

A missing age and an age of zero are not the same thing.

Filling Missing Values with the Mean

For numerical columns:

mean_age = df["age"].mean()

df["age"] = df["age"].fillna(
    mean_age
)

Shorter version:

df["age"] = df["age"].fillna(
    df["age"].mean()
)

Suppose ages are:

20, 25, missing, 30

The mean is:

25

The missing value can therefore be replaced with 25.

Filling Missing Values with the Median

Use:

df["salary"] = df["salary"].fillna(
    df["salary"].median()
)

The median is the middle value after sorting.

It can be more useful than the mean when the data contains extreme values.

For example:

30000
35000
40000
45000
500000

The unusually high value can strongly affect the mean.

Filling Missing Values with the Mode

The mode is the most frequently occurring value.

It is often useful for categorical columns.

Example:

df["city"] = df["city"].fillna(
    df["city"].mode()[0]
)

If the most common city is Jaipur, missing city values will be replaced with Jaipur.

Use this only if it makes sense for your dataset.

Filling Missing Text with a Label

Sometimes it is useful to keep the missing information explicit.

Example:

df["city"] = df["city"].fillna(
    "Unknown"
)

Now the missing value is clearly identified rather than silently replaced with another city.

Forward Fill

Forward fill uses the previous valid value.

Example:

df["price"] = df["price"].ffill()

Suppose:

100
NaN
NaN
120

After forward fill:

100
100
100
120

This can be useful for certain ordered or time-series datasets.

Do not use it automatically for unrelated rows.

Backward Fill

Backward fill uses the next valid value.

df["price"] = df["price"].bfill()

Example:

100
NaN
NaN
120

may become:

100
120
120
120

Whether this makes sense depends on your data.

Interpolating Missing Values

Interpolation estimates values between known observations.

Example:

df["temperature"] = (
    df["temperature"]
    .interpolate()
)

Suppose:

20
NaN
24

A simple linear interpolation may estimate:

22

Interpolation can be useful for time series and continuous measurements.

However, estimated values should not be treated as actual observations without understanding the implications.

Handling Custom Missing Values in CSV Files

Real datasets may use special text such as:

N/A
unknown
missing
-
?

When reading the CSV, you can tell Pandas to treat these as missing:

import pandas as pd

df = pd.read_csv(
    "data.csv",
    na_values=[
        "unknown",
        "missing",
        "-",
        "?"
    ]
)

Pandas can then process them as missing values.

Example CSV with Missing Data

Suppose employees.csv contains:

name,age,salary
Aman,25,45000
Riya,,52000
Rahul,23,unknown
Neha,30,65000

Read it:

import pandas as pd

df = pd.read_csv(
    "employees.csv",
    na_values=["unknown"]
)

Check missing values:

print(df.isnull().sum())

You may get:

name      0
age       1
salary    1
dtype: int64

Converting Invalid Values to Missing Data

Suppose salary contains:

45000
52000
unknown
65000

Convert it:

df["salary"] = pd.to_numeric(
    df["salary"],
    errors="coerce"
)

errors="coerce" changes invalid numeric values into missing values.

Then:

print(df["salary"])

might show:

45000.0
52000.0
NaN
65000.0

Now you can handle the missing salary consistently.

Handling Missing Dates

Suppose:

df = pd.DataFrame({
    "join_date": [
        "2025-01-10",
        "invalid",
        None,
        "2025-04-20"
    ]
})

Convert dates:

df["join_date"] = pd.to_datetime(
    df["join_date"],
    errors="coerce"
)

Invalid or missing dates become:

NaT

NaT means Not a Time and is Pandas’ missing datetime representation.

Check:

print(
    df["join_date"].isna()
)

Handling Missing Values in Multiple Columns

Suppose:

df = pd.DataFrame({
    "age": [20, None, 30],
    "salary": [40000, 50000, None]
})

You can fill each column differently:

df["age"] = df["age"].fillna(
    df["age"].median()
)

df["salary"] = df["salary"].fillna(
    df["salary"].mean()
)

This is usually better than applying one arbitrary replacement to every column.

Filling Several Columns with Different Values

You can also use a dictionary:

df = df.fillna({
    "age": 25,
    "city": "Unknown",
    "salary": 0
})

Again, choose replacements based on the meaning of each variable.

Using pd.NA

Pandas provides:

pd.NA

as a general missing-value marker.

Example:

import pandas as pd

df = pd.DataFrame({
    "name": ["Aman", "Riya"],
    "age": [25, pd.NA]
})

print(df)

pd.NA is especially useful with Pandas nullable data types.

Nullable Integer Data Types

Traditional integer columns cannot always represent standard floating-point NaN without being converted to floating-point values.

Pandas provides nullable integers:

df["age"] = df["age"].astype(
    "Int64"
)

Notice the capital I.

This allows values such as:

25
<NA>
30

while keeping the column logically integer-based.

Nullable Boolean Data

You can also use:

df["active"] = df["active"].astype(
    "boolean"
)

This supports:

True
False
<NA>

Complete Missing Data Example

Consider:

import pandas as pd

data = {
    "name": [
        "Aman",
        "Riya",
        "Rahul",
        "Neha"
    ],
    "age": [
        25,
        None,
        23,
        30
    ],
    "salary": [
        "45000",
        "52000",
        "unknown",
        "65000"
    ],
    "city": [
        "Jaipur",
        None,
        "Mumbai",
        "Delhi"
    ]
}

df = pd.DataFrame(data)

print(df)

First convert salary:

df["salary"] = pd.to_numeric(
    df["salary"],
    errors="coerce"
)

Check missing data:

print(df.isnull().sum())

Fill age using the median:

df["age"] = df["age"].fillna(
    df["age"].median()
)

Fill salary:

df["salary"] = df["salary"].fillna(
    df["salary"].median()
)

Fill city:

df["city"] = df["city"].fillna(
    "Unknown"
)

Check again:

print(df.isnull().sum())

Now you should see:

name      0
age       0
salary    0
city      0
dtype: int64

Inspect the cleaned DataFrame:

print(df)

This demonstrates an important point: different columns may require different missing-value strategies.

How to Calculate Missing Data Percentage

Sometimes raw counts are not enough.

You may want to know the percentage missing in each column.

Use:

missing_percentage = (
    df.isnull().mean() * 100
)

print(missing_percentage)

Suppose the output is:

name       0.0
age       10.0
salary    35.0
city       5.0
dtype: float64

This can help you decide which columns require closer attention.

Should You Remove a Column with Many Missing Values?

Sometimes a column contains very little usable data.

For example:

phone_number → 85% missing

You might consider removing it:

df = df.drop(
    columns=["phone_number"]
)

But do not use a fixed percentage rule blindly.

A mostly missing column may still be extremely important.

The decision should depend on:

• Business meaning
• Machine learning relevance
• Whether the values can be recovered
• Why values are missing
• Amount of available data

Understanding Types of Missing Data

When you move into statistics and machine learning, it helps to understand that missingness can happen in different ways.

Missing Completely at Random

The missing value has no meaningful relationship with other variables.

For example, a random technical failure loses a small number of measurements.

Missing at Random

Missingness may depend on other observed information.

For example, one customer group may be less likely to provide a particular field.

Missing Not at Random

The missing value may be related to the missing information itself.

For example, people with very high incomes may be less likely to report income.

These ideas matter because blindly filling missing values can introduce bias.

Beginners do not need to master the statistical theory immediately, but they should understand that missing data is not always random.

Missing Data in Machine Learning

Missing values are especially important when preparing data for machine learning.

Suppose:

df = pd.DataFrame({
    "age": [
        22,
        25,
        None,
        35
    ],
    "income": [
        30000,
        None,
        60000,
        80000
    ],
    "purchased": [
        0,
        0,
        1,
        1
    ]
})

You might initially write:

df["age"] = df["age"].fillna(
    df["age"].median()
)

df["income"] = df["income"].fillna(
    df["income"].median()
)

This can be fine for a simple learning example.

However, real machine learning projects need extra care.

Avoiding Data Leakage

A major mistake is calculating replacement values using the entire dataset before model evaluation.

Suppose you calculate:

df["age"].median()

using training and test data together.

Information from your test dataset has now influenced preprocessing.

This is known as data leakage.

A safer machine learning workflow is:

Dataset
   ↓
Train/Test Split
   ↓
Learn Imputation Values from Training Data
   ↓
Apply Same Transformation to Training Data
   ↓
Apply Same Learned Transformation to Test Data
   ↓
Train Model

Using SimpleImputer with scikit-learn

For machine learning, scikit-learn provides tools such as SimpleImputer.

Example:

from sklearn.impute import SimpleImputer

imputer = SimpleImputer(
    strategy="median"
)

Fit the imputer using training data:

imputer.fit(X_train)

Transform training data:

X_train_clean = imputer.transform(
    X_train
)

Transform test data using the same learned values:

X_test_clean = imputer.transform(
    X_test
)

This avoids calculating a separate median from the test data.

Common SimpleImputer Strategies

You can use:

strategy="mean"

for the mean.

strategy="median"

for the median.

strategy="most_frequent"

for the mode-like most frequent value.

You can also use a constant replacement:

SimpleImputer(
    strategy="constant",
    fill_value="Unknown"
)

The appropriate strategy depends on your data.

Missing Values Can Be Information

Sometimes the fact that a value is missing is useful.

For example, a customer not providing a phone number may itself correlate with another behavior.

You can create a missing indicator:

df["salary_missing"] = (
    df["salary"].isna()
)

Now the model or analysis can retain information about whether salary was originally missing.

Do this only when it is logically useful.

Practical Example: Cleaning a Customer Dataset

Suppose:

import pandas as pd

df = pd.DataFrame({
    "customer": [
        "Aman",
        "Riya",
        "Rahul",
        "Neha"
    ],
    "age": [
        25,
        None,
        31,
        None
    ],
    "city": [
        "Jaipur",
        "Delhi",
        None,
        "Mumbai"
    ],
    "spending": [
        5000,
        7000,
        None,
        6000
    ]
})

Inspect missing values:

print(df.isnull().sum())

Fill age using the median:

df["age"] = df["age"].fillna(
    df["age"].median()
)

Fill city with a clear category:

df["city"] = df["city"].fillna(
    "Unknown"
)

Fill spending using the median:

df["spending"] = (
    df["spending"]
    .fillna(
        df["spending"].median()
    )
)

Verify:

print(df.isnull().sum())

Then inspect:

print(df)

This is a simple but realistic missing-data workflow.

Remove or Fill Missing Data?

Beginners often ask whether missing values should be removed or filled.

The answer depends on the situation.

Consider removing rows when:

• Only a small number are affected
• Missing records are unusable
• Removing them will not create serious bias
• You have enough remaining data

Consider filling values when:

• Removing rows would lose too much information
• A reasonable replacement exists
• The feature is important
• Your analysis or model requires complete values

Consider leaving values missing when:

• Your tool supports missing values
• Missingness itself has meaning
• Estimating a replacement would be misleading

Mean vs Median vs Mode

Mean

Example:

df["age"].mean()

Useful for numerical data without severe outliers.

Median

Example:

df["salary"].median()

Often useful for skewed numerical data or data containing extreme values.

Mode

Example:

df["city"].mode()[0]

Useful for categorical information when the most frequent category is a sensible replacement.

None of these methods is automatically correct.

Common Mistakes to Avoid

1. Replacing Every Missing Value with Zero

This can change the meaning of the data.

For example:

salary = 0

means something different from:

salary = unknown

2. Dropping Every Missing Row

This can dramatically reduce your dataset and may introduce bias.

Always inspect missing-data patterns first.

3. Using the Mean for Every Numeric Column

The mean can be strongly influenced by outliers.

Consider the distribution before selecting a method.

4. Ignoring Missing Categorical Values

Missing text data needs attention too.

For example:

df["city"].isna().sum()

5. Treating “Unknown” as a Real Category Accidentally

If "unknown" actually means missing, convert it appropriately during import or cleaning.

6. Comparing NaN with Equality

Avoid:

value == float("nan")

Use appropriate missing-value functions instead.

7. Filling Test Data Using Test Statistics

In machine learning, do not calculate test-set means, medians, or other learned preprocessing values independently.

Learn preprocessing from training data and apply it consistently.

8. Ignoring Why Data Is Missing

The reason for missing data can affect which treatment is appropriate.

Best Practices for Handling Missing Data

Start by measuring the problem.

Use:

df.isnull().sum()

and:

df.isnull().mean() * 100

Inspect affected rows before changing them.

df[
    df.isnull().any(axis=1)
]

Choose a strategy separately for different columns.

Keep raw data unchanged whenever possible.

Document why you removed or replaced values.

For machine learning, split data appropriately and use preprocessing pipelines to reduce leakage risk.

Validate the dataset after handling missing values:

print(df.isnull().sum())
print(df.dtypes)
print(df.shape)

How Missing Data Handling Connects to AI Development

Handling missing data is a core data preprocessing skill.

A typical AI workflow looks like:

Raw Dataset
   ↓
Load with Pandas
   ↓
Detect Missing Data
   ↓
Understand Missingness
   ↓
Clean / Impute Values
   ↓
Feature Engineering
   ↓
Train/Test Split
   ↓
Model Preprocessing
   ↓
Machine Learning Model

As an AI Developer, you need to understand the data before training a model.

A powerful algorithm cannot automatically fix poor decisions about missing values.

Learning how to inspect, remove, fill, and correctly preprocess missing data gives you a much stronger foundation for machine learning.

What to Learn Next

After learning how to handle missing data in Python, continue with:

1. Data cleaning with Pandas
2. Duplicate data handling
3. Outlier detection
4. Pandas data types
5. Categorical data encoding
6. Feature scaling
7. Exploratory data analysis
8. Statistics for machine learning
9. Train/test splitting
10. scikit-learn preprocessing
11. Machine learning pipelines
12. Feature engineering

A useful learning path is:

Python → NumPy → Pandas → Data Cleaning → Missing Data → EDA → Statistics → scikit-learn → Machine Learning → Deep Learning

Frequently Asked Questions

1. What is missing data in Python?

Missing data means an expected value is unavailable. In Python and Pandas, it may be represented using None, NaN, pd.NA, NaT, empty cells, or custom markers such as "unknown".

2. How do I find missing values in Pandas?

Use:

df.isnull().sum()

or:

df.isna().sum()

These show the number of missing values in each column.

3. How do I remove missing values?

Use:

df = df.dropna()

Be careful because this removes rows and may discard useful information.

4. How do I replace missing values?

Use:

df["column"] = df["column"].fillna(
    replacement
)

The replacement can be a fixed value, mean, median, mode, or another suitable value.

5. Should I use mean or median for missing values?

The mean can work for reasonably balanced numerical data, while the median is often more robust when the data is skewed or contains extreme values. The best choice depends on the variable.

6. What is the difference between None and NaN?

None is Python’s general representation for no value. NaN is a special floating-point value meaning Not a Number and is commonly used for missing numerical data.

7. Can machine learning models handle missing data?

It depends on the algorithm and library. Some models support missing values directly, while many workflows require missing data to be handled before training.

8. What is data imputation?

Imputation means replacing missing values with estimated or predefined values, such as a mean, median, mode, constant, or values learned through a more advanced method.

Conclusion

Handling missing data is an essential skill for anyone working with Python, Pandas, data science, or machine learning.

The first step is always to identify how much data is missing and understand why. You can then decide whether to remove rows, fill values, interpolate observations, preserve missingness, or use a machine learning imputation strategy.

Avoid blindly replacing every missing value with zero or deleting every incomplete record.

For AI development, pay particular attention to data leakage. Replacement values learned from data should generally be calculated from the training set and then applied consistently to other datasets.

Once you understand missing data, the next logical step is learning outlier detection, categorical data preprocessing, feature scaling, and exploratory data analysis.

---