Summary Statistics and Insights

Published

Aug 2026

  • ID: DS-L06
  • Type: Lesson
  • Audience: Beginner / Intermediate
  • Theme: Using summary statistics to move from numbers to interpretation

Summary statistics turn rows of observations into evidence we can compare and explain.

No single statistic tells the whole story. The mean describes an arithmetic center, the median provides a center that is less sensitive to unusual values, and the standard deviation and interquartile range describe different aspects of variability. Counts reveal whether groups are balanced, while correlations describe how numeric variables move together.

In this lesson, we combine these summaries with visual checks and write concise insights without making claims that the data cannot support.

inspect
  ↓
clean
  ↓
wrangle
  ↓
visualize
  ↓
summarize and interpret

Lesson overview

By the end of this lesson, you will be able to:

  • summarize numeric variables using mean, median, standard deviation, quartiles, and IQR
  • summarize categorical variables using counts and percentages
  • compare groups using consistent descriptive statistics
  • rank features using a scale-independent measure of group differentiation
  • compare Pearson and Spearman correlations
  • connect statistical tables to visual evidence
  • write careful, evidence-based interpretations
  • save a reusable summary evidence package

Chapter workflow

06-summary-statistics-and-insights.qmd
        ↓
scripts/python/summarize_table.py
        ↓
data/iris_wrangled.csv
results/summary/

Expected outputs:

results/summary/
├── numeric-summary.tsv
├── species-counts.tsv
├── grouped-summary.tsv
├── feature-differentiation.tsv
├── pearson-correlation.tsv
├── spearman-correlation.tsv
└── analysis-insights.md

Load the wrangled dataset

We use the wrangled dataset created in Chapter 04.

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

df = pd.read_csv("data/iris_wrangled.csv")
df.head()

Inspect before summarizing

print("Shape:", df.shape)
print("\nColumns:", df.columns.tolist())
print("\nData types:")
print(df.dtypes)
print("\nMissing values:")
print(df.isna().sum())

Before choosing statistics, identify:

  • the numeric variables to summarize
  • the categorical variable that defines groups
  • any derived features
  • any remaining missing values
num_cols = [
    "sepal_length",
    "sepal_width",
    "petal_length",
    "petal_width",
    "petal_area"
]

petal_area is derived from petal length and width. It is useful as a combined size measure, but it is not independent of those two source variables.

Numeric summaries: center and spread

The most useful first table combines conventional and robust statistics.

numeric_summary = (
    df[num_cols]
    .agg(["count", "mean", "median", "std", "min", "max"])
    .transpose()
    .reset_index(names="feature")
)

numeric_summary["q1"] = df[num_cols].quantile(0.25).values
numeric_summary["q3"] = df[num_cols].quantile(0.75).values
numeric_summary["iqr"] = numeric_summary["q3"] - numeric_summary["q1"]

numeric_summary = numeric_summary[
    ["feature", "count", "mean", "median", "std", "q1", "q3", "iqr", "min", "max"]
]

numeric_summary

What each statistic contributes

Statistic What it tells you
count Number of non-missing observations
mean Arithmetic center
median Middle value; less sensitive to extreme values
std Typical variability around the mean
q1, q3 Boundaries of the middle 50% of values
iqr Spread of the middle 50%; calculated as Q3 − Q1
min, max Observed endpoints

Compare the mean with the median and the standard deviation with the IQR. Agreement suggests a fairly stable summary; noticeable differences are a reason to inspect the distribution.

Visual check of distributions

df_long = df.melt(
    value_vars=num_cols,
    var_name="feature",
    value_name="value"
)

g = sns.displot(
    data=df_long,
    x="value",
    col="feature",
    col_wrap=3,
    bins=12,
    kde=True,
    height=3.3,
    aspect=1.15
)

g.fig.suptitle("Iris — Distributions of Numeric Features", y=1.03)

plt.show()

Histograms add distribution shape to the numeric table. Multiple peaks in the combined data may reflect the three species rather than unusual observations.

Categorical counts and percentages

Counts show sample size; percentages make the group balance immediately interpretable.

species_counts = (
    df["species"]
    .value_counts()
    .sort_index()
    .rename_axis("species")
    .reset_index(name="n")
)

species_counts["percent"] = (
    species_counts["n"] / species_counts["n"].sum() * 100
)

species_counts
fig, ax = plt.subplots(figsize=(7, 4))

sns.countplot(
    data=df,
    x="species",
    order=sorted(df["species"].unique()),
    ax=ax
)

ax.set_title("Number of Samples by Species")
ax.set_xlabel("Species")
ax.set_ylabel("Number of samples")

plt.show()

Balanced groups receive equal influence in an overall summary. With imbalanced data, always examine group-level results because a large group can dominate the combined mean.

Grouped summaries

A long-format grouped table is compact, consistent, and easy to filter.

group_records = []

for species, group in df.groupby("species", observed=False):
    for feature in num_cols:
        values = group[feature]
        q1 = values.quantile(0.25)
        q3 = values.quantile(0.75)

        group_records.append(
            {
                "species": species,
                "feature": feature,
                "n": values.count(),
                "mean": values.mean(),
                "median": values.median(),
                "sd": values.std(),
                "q1": q1,
                "q3": q3,
                "iqr": q3 - q1,
                "min": values.min(),
                "max": values.max()
            }
        )

grouped_summary = pd.DataFrame(group_records)

grouped_summary

This table answers several questions at once:

  • Which species has the largest typical value?
  • Do the mean and median tell a similar story?
  • Which species is most variable?
  • How much do the middle 50% of observations overlap?

Visual comparison with boxplots

Boxplots are more informative here than bars because they show the median, IQR, overall spread, and potential outliers.

df_group_long = df.melt(
    id_vars="species",
    value_vars=num_cols,
    var_name="feature",
    value_name="value"
)

g = sns.catplot(
    data=df_group_long,
    x="species",
    y="value",
    col="feature",
    col_wrap=3,
    kind="box",
    sharey=False,
    height=3.4,
    aspect=1.1
)

g.set_axis_labels("Species", "Value")
g.fig.suptitle("Numeric Features by Species", y=1.03)

plt.show()

Use the second version in your own analysis because the relationship between each measurement and its species remains explicit.

Scale-independent group differentiation

Subtracting the smallest group mean from the largest is not suitable for ranking features measured on different scales. A feature with larger numeric units can appear more important simply because its values are larger.

Instead, use eta-squared (η²), a descriptive effect-size measure:

\[ \eta^2 = \frac{\text{between-group variation}}{\text{total variation}} \]

Values range from 0 to 1:

  • values near 0 indicate little differentiation among group means
  • larger values indicate that species membership accounts for more of the observed variation
feature_differentiation = []

for col in num_cols:
    overall_mean = df[col].mean()

    between_ss = sum(
        len(group) * (group[col].mean() - overall_mean) ** 2
        for _, group in df.groupby("species", observed=False)
    )

    total_ss = ((df[col] - overall_mean) ** 2).sum()

    feature_differentiation.append(
        {
            "feature": col,
            "eta_squared": between_ss / total_ss
        }
    )

feature_differentiation = (
    pd.DataFrame(feature_differentiation)
    .sort_values("eta_squared", ascending=False)
    .reset_index(drop=True)
)

feature_differentiation

Eta-squared ranks descriptive group differentiation on a common scale. It does not measure predictive accuracy and does not prove that species causes a measurement.

Because petal_area is derived from petal length and width, interpret its ranking as a useful summary of combined petal size—not as evidence of a new independent measurement.

Pearson and Spearman correlations

Pearson correlation summarizes linear association. Spearman correlation summarizes monotonic association using ranks and is less sensitive to extreme values and nonlinearity.

pearson_corr = df[num_cols].corr(method="pearson")
spearman_corr = df[num_cols].corr(method="spearman")

pearson_corr
spearman_corr
fig, axes = plt.subplots(1, 2, figsize=(14, 5.5))

sns.heatmap(
    pearson_corr,
    annot=True,
    fmt=".2f",
    vmin=-1,
    vmax=1,
    cmap="vlag",
    ax=axes[0]
)
axes[0].set_title("Pearson Correlation")

sns.heatmap(
    spearman_corr,
    annot=True,
    fmt=".2f",
    vmin=-1,
    vmax=1,
    cmap="vlag",
    ax=axes[1]
)
axes[1].set_title("Spearman Correlation")

plt.tight_layout()
plt.show()

When Pearson and Spearman values are similar, the association is likely both linear and monotonic. A noticeable difference suggests that shape or unusual observations deserve closer inspection.

Important

Correlation does not establish causation. Correlations in the full Iris dataset may also reflect differences among species. Examine scatterplots and within-species patterns before interpreting a pooled correlation biologically.

Correlations involving petal_area are expected to be strong because that feature was calculated from petal length and width. This is mathematical dependence, not independent confirmation.

Generate an insights report

A useful report includes concrete values, interpretation, and limitations.

top_feature = feature_differentiation.iloc[0]

petal_length_by_species = (
    grouped_summary
    .query("feature == 'petal_length'")
    [["species", "n", "mean", "median", "sd", "iqr"]]
)

report = f"""# Insights Report: Iris Dataset

## Dataset and group balance

- The dataset contains {df.shape[0]} observations and {df.shape[1]} columns.
- It contains {df["species"].nunique()} species groups.
- Group sizes range from {species_counts["n"].min()} to {species_counts["n"].max()} observations.

## Petal length by species

{petal_length_by_species.to_markdown(index=False)}

## Strongest descriptive differentiation

`{top_feature["feature"]}` has the largest eta-squared value
({top_feature["eta_squared"]:.3f}) among the summarized features.

## Interpretation

Petal measurements provide clearer descriptive differentiation among species
than sepal measurements in this dataset.

## Limitations

These results describe this dataset. They do not establish causation or
classification performance. Correlations involving `petal_area` partly reflect
how that derived feature was calculated.
"""

print(report)

A strong insight states:

  1. the evidence,
  2. what the evidence reasonably suggests, and
  3. what the analysis does not establish.

Validation checks

assert int(df.isna().sum().sum()) == 0, "Missing values remain."
assert "species" in df.columns, "The grouping variable is missing."
assert set(num_cols).issubset(df.columns), "A numeric feature is missing."
assert species_counts["n"].sum() == len(df), "Group counts do not match row count."
assert feature_differentiation["eta_squared"].between(0, 1).all()
assert pearson_corr.equals(pearson_corr.transpose())
assert spearman_corr.equals(spearman_corr.transpose())

print("All summary validation checks passed.")

Save summary outputs

from pathlib import Path

output_dir = Path("results/summary")
output_dir.mkdir(parents=True, exist_ok=True)

numeric_summary.to_csv(
    output_dir / "numeric-summary.tsv", sep="\t", index=False
)
species_counts.to_csv(
    output_dir / "species-counts.tsv", sep="\t", index=False
)
grouped_summary.to_csv(
    output_dir / "grouped-summary.tsv", sep="\t", index=False
)
feature_differentiation.to_csv(
    output_dir / "feature-differentiation.tsv", sep="\t", index=False
)
pearson_corr.to_csv(
    output_dir / "pearson-correlation.tsv", sep="\t"
)
spearman_corr.to_csv(
    output_dir / "spearman-correlation.tsv", sep="\t"
)
(output_dir / "analysis-insights.md").write_text(
    report, encoding="utf-8"
)

Run the reusable summary script

Run this from the project root:

python scripts/python/summarize_table.py data/iris_wrangled.csv results/summary

The script validates the input and creates the same evidence package developed manually in this lesson.

Key takeaways

  • Use both conventional and robust summaries: mean with standard deviation, and median with IQR.
  • Always report group size alongside group statistics.
  • Prefer scale-independent measures when ranking features with different units.
  • Compare Pearson and Spearman correlations and inspect the corresponding plots.
  • Treat derived variables carefully because they can create expected mathematical relationships.
  • Descriptive statistics support interpretation, but they do not establish causation or predictive performance.

Next step

In Chapter 07, these descriptive insights become the foundation for moving from analysis to machine learning.