First hour with a new dataset: a repeatable EDA checklist in pandas

A new dataset is not a blank canvas. It is the result of a pipeline: databases, APIs, spreadsheets, sensors, human input, and business processes. Every one of those sources can introduce problems.

Before building a model, publishing a dashboard, or calculating business metrics, spend the first hour on exploratory data analysis (EDA).

The goal is not to create dozens of charts. The goal is to answer one question:

Can I trust this data enough to make decisions with it?

This repeatable EDA checklist uses current pandas practices and focuses on practical DataQuality checks:

• Dataset structure

• Data types

• Missing values

• Duplicate records

• Numerical distributions

• Categorical inconsistencies

• Potential data leakage

The examples use pandas APIs that work with modern pandas 3.x releases and the compatible pandas 2.3+ line.

By the end, you should have a documented first impression of a messy CSV: what is reliable, what needs cleaning, and what requires investigation.

Step 0: Load the dataset reproducibly

The first EDA mistake often happens before analysis begins.

CSV files commonly contain:

• Dates imported as plain text

• Numbers stored as strings

• Extra index columns from exports

• Mixed data types

• Unexpected missing-value markers

• Encoding problems

Keep the original file unchanged and make the loading step reproducible.

import pandas as pd

df = pd.read_csv(
    "messy_customer_data.csv",
    low_memory=False
)

print(df.head())

For a quick first look at a large file, read a sample.

import pandas as pd

sample = pd.read_csv(
    "messy_customer_data.csv",
    nrows=1000,
    low_memory=False
)

print(sample.head())
print(sample.shape)

Sampling is useful for exploration, but final checks should run on the full dataset.

Step 1: Check the dataset shape and columns

The first question is simple:

What exactly did I receive?

rows, columns = df.shape

print(f"Rows: {rows}")
print(f"Columns: {columns}")

Dataset size changes your strategy:

• Small datasets may need careful manual inspection.

• Wide datasets may need feature selection and column review.

• Very large datasets may require chunk processing.

Inspect column names immediately.

print(df.columns.tolist())

Look for:

• Export artifacts such as Unnamed: 0

• Duplicate column names

• Unexpected fields

• Personally identifiable information

• Columns that should not enter a machine-learning pipeline

A column list often reveals mistakes before any analysis does.

Step 2: Understand pandas data types

A column name does not guarantee a usable feature.

A column called customer_age containing "42" is text until converted. A date column may look correct while still being unusable for time-based analysis.

Start with the built-in overview:

df.info()

Then inspect individual types:

print(df.dtypes)

Common problems include:

Problem	Example
Numbers stored as text	"42" instead of 42
Dates stored as strings	"2026-07-18"
Mixed values	"unknown" inside numeric columns
Category differences	"Germany" and "germany"

Make conversion problems visible.

df["customer_age"] = pd.to_numeric(
    df["customer_age"],
    errors="coerce"
)

print(df["customer_age"].dtype)

Using errors="coerce" converts invalid values into missing values. That turns hidden problems into measurable problems.

Step 3: Measure missing values

Missing data is not automatically a failure.

A missing value can mean:

• A customer skipped a question

• A sensor was unavailable

• A system did not collect a field

• A process changed over time

Measure the size of the problem first.

missing_report = pd.DataFrame(
    {
        "missing_count": df.isna().sum(),
        "missing_percent": df.isna().mean() * 100,
    }
)

print(
    missing_report.sort_values(
        "missing_percent",
        ascending=False
    )
)

Ask:

• Is a column almost empty?

• Did missing values appear after a migration?

• Are missing values concentrated in one customer group?

• Should they be removed, imputed, or treated as information?

Avoid replacing every missing value with zero.

Zero and unknown usually mean different things.

Step 4: Find duplicate records

Duplicates can distort statistics and machine-learning results.

Check exact duplicate rows:

duplicate_count = df.duplicated().sum()

print(
    f"Duplicate rows: {duplicate_count}"
)

Inspect examples:

duplicates = df[
    df.duplicated(keep=False)
]

print(duplicates.head())

Context matters.

A duplicated transaction might be correct. A duplicated customer profile might indicate a pipeline problem.

Also check identifiers that should be unique.

repeated_ids = df["customer_id"].duplicated().sum()

print(
    f"Repeated customer IDs: {repeated_ids}"
)

Step 5: Review distributions

Summary statistics reveal whether values behave as expected.

Start with numerical columns:

print(df.describe())

For mixed datasets:

print(
    df.describe(include="all")
)

Look for:

• Impossible minimum or maximum values

• Extreme outliers

• Constant columns

• Unexpected categories

For example:

print(
    df["customer_age"].describe()
)

An age column containing negative values is not an interesting pattern. It is a data-quality issue.

Step 6: Inspect categorical columns

Text columns often reveal the history of a dataset.

Check the most common values:

for column in df.select_dtypes(include="object"):
    print(f"\nColumn: {column}")
    print(
        df[column]
        .value_counts()
        .head(10)
    )

Common discoveries:

• "USA" versus "United States"

• "Premium" versus "premium "

• Misspelled categories

• Unexpected values from external systems

Clean only after understanding the meaning.

Example:

df["country"] = (
    df["country"]
    .str.strip()
    .str.lower()
)

Every transformation should have a reason. A clean-looking dataset can still contain incorrect assumptions.

Step 7: Search for suspicious columns and leakage

Some columns are useful for analysis but dangerous for machine learning.

Common risks:

• Database identifiers

• Future information

• Manual labels

• Columns created after the prediction event

Check uniqueness:

for column in df.columns:
    unique_values = df[column].nunique()

    print(
        column,
        unique_values
    )

A nearly unique column could be:

• A legitimate identifier

• A timestamp

• Random noise

• A leakage source

Do not automatically remove it. Investigate it.

Step 8: Create a reusable EDA function

EDA works best as a habit, not a one-time notebook exercise.

A small helper creates a consistent starting point.

def quick_eda(dataframe):
    print("Shape:")
    print(dataframe.shape)

    print("\nData types:")
    print(dataframe.dtypes)

    print("\nMissing values:")
    print(
        dataframe.isna()
        .sum()
        .sort_values(ascending=False)
        .head(20)
    )

    print("\nDuplicate rows:")
    print(
        dataframe.duplicated().sum()
    )

    print("\nNumeric summary:")
    print(
        dataframe.describe()
    )


quick_eda(df)

This is not the final analysis. It is a map showing where deeper investigation is needed.

Step 9: Visualize after the checks

Charts are useful when they answer a question.

A common mistake is creating many visualizations before checking whether the data itself is reliable.

Useful first visual checks:

• Numeric feature distributions

• Category frequencies

• Missing-value patterns

• Relationships between important variables

Example:

import matplotlib.pyplot as plt

df["customer_age"].hist(
    bins=30
)

plt.xlabel("Age")
plt.ylabel("Count")
plt.title("Customer age distribution")

plt.show()

A simple histogram can reveal:

• Unexpected spikes

• Impossible values

• Skewed distributions

• Data-entry mistakes

Cherry on the cake: the cost of skipping basic data checks

A famous example of data understanding going wrong is the original Google Flu Trends project.

The system attempted to estimate influenza activity from search behavior. Early results looked impressive because search patterns correlated with reported flu activity. Later analysis showed that the relationship was unstable: user behavior changed, the data signal drifted, and the model did not generalize reliably over time.

The lesson is not that search data was useless. The lesson is that a model can learn patterns from data that look meaningful today but fail when the world changes.

A five-minute EDA habit could have raised early warning questions:

• Did input distributions change?

• Were important relationships stable over time?

• Were predictions checked against independent measurements?

• Was the model learning flu activity or temporary search behavior?

EDA is not a replacement for proper validation, but it is often the fastest way to discover that validation is needed.

A practical first-hour EDA checklist

When receiving a new CSV, follow this order:

1. Preserve the original file.

2. Load it with a reproducible script.

3. Check rows, columns, and column names.

4. Inspect pandas data types.

5. Measure missing values.

6. Find duplicate records.

7. Review numerical summaries.

8. Inspect categorical values.

9. Search for leakage risks.

10. Create only targeted visualizations.

11. Document findings before cleaning.

A good EDA process turns an unknown dataset into a known engineering problem.

Before your next model, run this checklist. Save your findings, turn your EDA workflow into a repeatable DataQuality gate, and continue to the next lesson by applying the checklist to a real messy dataset of your own.