Data Wrangling Basics

Published

Aug 2026

  • ID: DS-L04
  • Type: Lesson
  • Audience: Beginner / Intermediate
  • Theme: Transforming and reshaping data using reproducible workflows

Data rarely arrives in the exact format required for analysis.

After a dataset has been cleaned, the next step is often to reshape, filter, summarize, and create analysis-ready outputs.

In CDI systems, wrangling is the bridge between:

cleaned table
        ↓
analysis-ready tables
        ↓
visualization, summaries, modeling, and reporting

This lesson continues from Chapter 03, where we created:

data/iris_clean.csv

In this chapter, we transform that cleaned table and create reusable downstream outputs.

Lesson overview

By the end of this lesson, you will be able to:

  • select and filter rows and columns
  • create derived features
  • sort and reorder data
  • group and aggregate values
  • apply conditional logic
  • reshape data into long format
  • save structured outputs for downstream analysis
  • run a reusable wrangling script from the command line

Chapter workflow

This chapter introduces the third reusable Python script in the system:

04-data-wrangling-basics.qmd
        ↓
scripts/python/wrangle_example_data.py
        ↓
data/iris_wrangled.csv
results/wrangling/species-summary.tsv
results/wrangling/iris-long.tsv
results/wrangling/wrangling-report.txt

The wrangled outputs support the next chapters on visualization and summary statistics.

Load the cleaned dataset

We use the cleaned dataset created in Chapter 03.

import numpy as np
import pandas as pd

df = pd.read_csv("data/iris_clean.csv")
rows_before = len(df)
df.insert(0, "observation_id", range(1, len(df) + 1))
df.head()

observation_id preserves the identity of each original flower as the table is sorted and reshaped later in the workflow.

Select columns

Selecting columns helps focus the analysis on variables of interest.

df[["sepal_length", "petal_length"]].head()

Column selection is often used to simplify a table before plotting, summarizing, or exporting results.

Filter rows

Filtering keeps rows that meet a condition.

Example: select rows where petal_length is greater than 4.

df_filtered = df[df["petal_length"] > 4]
df_filtered.head()

Filtering should always be tied to an analysis question or quality-control rule.

Create derived features

Feature creation extends the dataset with new variables.

Here, we define a new feature, petal_area.

df["petal_area"] = df["petal_length"] * df["petal_width"]
df.head()

For this lesson, petal_area is a simple rectangular approximation calculated as petal length × petal width. It is not the true biological area of a petal. Because both measurements are in centimetres, the approximate area is in square centimetres (cm²).

A derived feature should be easy to explain and reproducible from existing columns.

Sort data

Sorting helps examine extreme or ordered values.

df = (
    df.sort_values(
        by=["petal_area", "species"],
        ascending=[False, True]
    )
    .reset_index(drop=True)
)

df.head()

This makes it easier to inspect the largest values, smallest values, or top-ranked observations. Resetting the index produces a clean row index after sorting, while observation_id continues to identify the original flowers.

Group and aggregate values

Grouping summarizes values across categories.

Here, we compute summary measurements by species.

grouped = (
    df.groupby("species", observed=False)
      .agg(
          n=("species", "size"),
          mean_petal_area=("petal_area", "mean"),
          median_petal_area=("petal_area", "median"),
          mean_sepal_length=("sepal_length", "mean"),
          mean_sepal_width=("sepal_width", "mean"),
          mean_petal_length=("petal_length", "mean"),
          mean_petal_width=("petal_width", "mean")
      )
      .reset_index()
      .sort_values("mean_petal_area", ascending=False)
)

grouped

This type of grouped table is one of the most common outputs in tidy-table analysis.

Apply conditional logic

Conditional logic creates categories based on defined rules.

Here, we assign a size category based on whether petal_area is above the dataset median.

median_petal_area = df["petal_area"].median()

df["size_category"] = np.where(
    df["petal_area"] > median_petal_area,
    "large",
    "small"
)

df.head()

Values greater than the median are labelled "large"; values equal to or below the median are labelled "small". This is a relative teaching category for this dataset, not a biological classification.

Reshape into long format

Long format is especially useful for visualization and grouped summaries.

df_long = df.melt(
    id_vars=["observation_id", "species", "size_category"],
    value_vars=["sepal_length", "sepal_width", "petal_length", "petal_width"],
    var_name="measurement",
    value_name="value"
)

df_long.head()

Long format is useful because many plotting and analysis workflows expect:

one row per observation-measurement pair

Validation checks

After transformation, confirm that the dataset remains consistent and complete.

print("Shape:", df.shape)
print("Missing values:", int(df.isna().sum().sum()))
print("Columns:", df.columns.tolist())

assert int(df.isna().sum().sum()) == 0, "Missing values detected after wrangling."
assert "petal_area" in df.columns, "petal_area was not created."
assert "size_category" in df.columns, "size_category was not created."
assert len(df) == rows_before, "Row count changed during wrangling."
assert df["observation_id"].is_unique, "observation_id must be unique."
assert set(df["size_category"]) <= {"small", "large"}, "Unexpected size category detected."
assert len(df_long) == len(df) * 4, "Long table must contain four rows per observation."
assert df_long["observation_id"].value_counts().eq(4).all(), (
    "Each observation must appear four times in the long table."
)
assert len(grouped) == df["species"].nunique(), (
    "Species summary must contain one row per species."
)
assert int(grouped.isna().sum().sum()) == 0, "Missing values detected in species summary."
assert int(df_long.isna().sum().sum()) == 0, "Missing values detected in long table."

Validation checks help prevent downstream chapters from using incomplete or incorrectly transformed data.

Save wrangled outputs

Save each output with a clear purpose.

from pathlib import Path

Path("data").mkdir(exist_ok=True)
Path("results/wrangling").mkdir(parents=True, exist_ok=True)

df.to_csv("data/iris_wrangled.csv", index=False)
grouped.to_csv("results/wrangling/species-summary.tsv", sep="\t", index=False)
df_long.to_csv("results/wrangling/iris-long.tsv", sep="\t", index=False)

print("Saved wrangled table to: data/iris_wrangled.csv")
print("Saved species summary to: results/wrangling/species-summary.tsv")
print("Saved long-format table to: results/wrangling/iris-long.tsv")

These files make the workflow progressive and testable.

Run the reusable wrangling script

The manual steps above explain the logic. The reusable script applies the wrangling workflow from the command line.

Run this from the project root:

python scripts/python/wrangle_example_data.py data/iris_clean.csv data/iris_wrangled.csv results/wrangling

Expected outputs:

data/iris_wrangled.csv
results/wrangling/species-summary.tsv
results/wrangling/iris-long.tsv
results/wrangling/wrangling-report.txt

What the wrangling script does

The script:

  • reads the cleaned input table
  • creates observation_id to preserve original row identity
  • creates petal_area
  • creates size_category
  • sorts the table by petal_area
  • creates a species-level summary table
  • reshapes measurements into long format
  • validates the transformed table
  • saves downstream outputs
  • writes a wrangling report

Exercise

Try the following:

  1. Open results/wrangling/species-summary.tsv.
  2. Identify the species with the largest mean petal_area.
  3. Open results/wrangling/iris-long.tsv.
  4. Confirm that each original observation contributes four rows—one for each measurement.
  5. Rerun the Chapter 02 inspection script on the wrangled table.

Use:

python scripts/python/inspect_table.py data/iris_wrangled.csv results/inspection-wrangled

Inspect the wrangled table with:

python scripts/python/inspect_table.py data/iris_wrangled.csv results/inspection-wrangled

Expected outputs:

results/inspection-wrangled/
├── table-inspection-summary.txt
├── table-column-summary.tsv
└── table-missing-values.tsv

The species summary table should show that petal-based measurements differ strongly by species.

For the standard Iris dataset:

  • virginica has the largest mean approximate petal_area
  • 150 original observations produce 600 long-format rows
  • every observation_id appears exactly four times in the long-format table

CDI Insight

Wrangling turns a cleaned table into analysis-ready outputs.

This is where data begins to serve specific analytical purposes.

A good wrangling step should produce outputs that are:

  • clear
  • documented
  • reusable
  • easy to validate
  • ready for downstream visualization and interpretation

In CDI systems, wrangling is not hidden inside a notebook. It is captured as a repeatable script.

Summary

In this lesson, you:

  • selected and filtered data
  • created an identifier for each original observation
  • created petal_area
  • sorted rows by a derived feature
  • grouped and summarized values by species
  • created size_category
  • reshaped the table into long format
  • saved wrangled outputs
  • created a wrangling report with wrangle_example_data.py

Looking Ahead

In the next chapter, we use the wrangled outputs to create exploratory visualizations. The files produced here become inputs for plotting.