Statistical Programming
Prof. Dr. Martin Spindler
The 80/20 Rule (The Painful Truth)
“Data cleaning and preparation are not glamorous, but they are absolutely critical to the success of any data analysis project.”
Three main approaches:
The choice depends on:
Reality Check: Most “clean” datasets look like someone played Jenga with your rows. Missing values everywhere!
# Histogram of heights
import matplotlib.pyplot as plt
import seaborn as sns
plt.figure(figsize=(10, 6))
plt.hist(starwars['height'].dropna(), bins=20, color='skyblue', edgecolor='black')
plt.xlabel('Height (cm)', fontsize=12)
plt.ylabel('Frequency', fontsize=12)
plt.title('Distribution of Character Heights', fontsize=14, fontweight='bold')
plt.grid(axis='y', alpha=0.3)
plt.show()
# Density plot of mass
plt.figure(figsize=(10, 6))
starwars['mass'].dropna().plot(kind='density', color='coral', linewidth=2)
plt.xlabel('Mass (kg)', fontsize=12)
plt.ylabel('Density', fontsize=12)
plt.title('Density Plot of Character Mass', fontsize=14, fontweight='bold')
plt.grid(alpha=0.3)
plt.show()
# Box plot: height by gender
plt.figure(figsize=(10, 6))
data_for_box = starwars[starwars['sex'].isin(['male', 'female'])].dropna(subset=['height', 'sex'])
sns.boxplot(data=data_for_box, x='sex', y='height', palette='Set2')
plt.xlabel('Gender', fontsize=12)
plt.ylabel('Height (cm)', fontsize=12)
plt.title('Height Distribution by Gender', fontsize=14, fontweight='bold')
plt.show()
# Mass by gender - notice the outlier!
plt.figure(figsize=(10, 6))
data_mass = starwars[starwars['sex'].isin(['male', 'female'])].dropna(subset=['mass', 'sex'])
sns.boxplot(data=data_mass, x='sex', y='mass', palette='Set3')
plt.xlabel('Gender', fontsize=12)
plt.ylabel('Mass (kg)', fontsize=12)
plt.title('Mass Distribution by Gender (Note: Outlier!)', fontsize=14, fontweight='bold')
plt.show()
# Violin plot combines box plot and density
plt.figure(figsize=(10, 6))
sns.violinplot(data=data_for_box, x='sex', y='height', palette='muted')
plt.xlabel('Gender', fontsize=12)
plt.ylabel('Height (cm)', fontsize=12)
plt.title('Height Distribution by Gender (Violin Plot)', fontsize=14, fontweight='bold')
plt.show()
# Top 10 species by count
plt.figure(figsize=(12, 6))
top_species = starwars['species'].value_counts().head(10)
top_species.plot(kind='bar', color='steelblue')
plt.xlabel('Species', fontsize=12)
plt.ylabel('Count', fontsize=12)
plt.title('Top 10 Most Common Species', fontsize=14, fontweight='bold')
plt.xticks(rotation=45, ha='right')
plt.grid(axis='y', alpha=0.3)
plt.tight_layout()
plt.show()
# Average height for top species
plt.figure(figsize=(12, 6))
top_species_names = starwars['species'].value_counts().head(8).index
species_height = (starwars[starwars['species'].isin(top_species_names)]
.groupby('species')['height']
.mean()
.sort_values(ascending=False))
species_height.plot(kind='bar', color='coral')
plt.xlabel('Species', fontsize=12)
plt.ylabel('Average Height (cm)', fontsize=12)
plt.title('Average Height by Species (Top 8)', fontsize=14, fontweight='bold')
plt.xticks(rotation=45, ha='right')
plt.grid(axis='y', alpha=0.3)
plt.tight_layout()
plt.show()
# Scatter plot: height vs mass
plt.figure(figsize=(10, 6))
plt.scatter(starwars['height'], starwars['mass'], alpha=0.6, s=50, color='darkblue')
plt.xlabel('Height (cm)', fontsize=12)
plt.ylabel('Mass (kg)', fontsize=12)
plt.title('Relationship: Height vs. Mass', fontsize=14, fontweight='bold')
plt.grid(alpha=0.3)
plt.show()
# Scatter plot colored by gender
plt.figure(figsize=(10, 6))
data_scatter = starwars.dropna(subset=['height', 'mass', 'sex'])
data_scatter = data_scatter[data_scatter['sex'].isin(['male', 'female'])]
for gender in ['male', 'female']:
subset = data_scatter[data_scatter['sex'] == gender]
plt.scatter(subset['height'], subset['mass'],
label=gender, alpha=0.6, s=60)
plt.xlabel('Height (cm)', fontsize=12)
plt.ylabel('Mass (kg)', fontsize=12)
plt.title('Height vs. Mass by Gender', fontsize=14, fontweight='bold')
plt.legend()
plt.grid(alpha=0.3)
plt.show()
# Scatter with regression line
plt.figure(figsize=(10, 6))
data_clean = starwars.dropna(subset=['height', 'mass'])
# Remove extreme outlier
data_clean = data_clean[data_clean['mass'] < 500]
sns.regplot(data=data_clean, x='height', y='mass',
scatter_kws={'alpha': 0.5}, line_kws={'color': 'red'})
plt.xlabel('Height (cm)', fontsize=12)
plt.ylabel('Mass (kg)', fontsize=12)
plt.title('Height vs. Mass with Regression Line', fontsize=14, fontweight='bold')
plt.show()
# Correlation heatmap for numerical variables
plt.figure(figsize=(8, 6))
numerical_cols = starwars[['height', 'mass', 'birth_year']].dropna()
correlation = numerical_cols.corr()
sns.heatmap(correlation, annot=True, cmap='coolwarm', center=0,
square=True, linewidths=1, cbar_kws={"shrink": 0.8})
plt.title('Correlation Matrix', fontsize=14, fontweight='bold')
plt.tight_layout()
plt.show()
Remember, Fellow Data Scientists!
Correlation ≠ Causation (but it does waggle its eyebrows suggestively)
Just because height and mass correlate doesn’t mean being tall causes you to be heavy. Could be: - The other way around - Both caused by genetics? - Diet? The Force? 🤔
String Operations: Not as scary as regex!
Good news: We’re just using .str.split() and .str.contains()
Bad news: Eventually you’ll need regex…
(But seriously, pandas string methods are your friend - they save you from regex 90% of the time)
# Distribution of maximum wind speeds
plt.figure(figsize=(12, 4))
# Histogram
plt.subplot(1, 2, 1)
plt.hist(storms['wind'], bins=40, color='steelblue', edgecolor='black', alpha=0.7)
plt.xlabel('Wind Speed (knots)', fontsize=11)
plt.ylabel('Frequency', fontsize=11)
plt.title('Distribution of Storm Wind Speeds', fontsize=13, fontweight='bold')
plt.grid(axis='y', alpha=0.3)
# By status
plt.subplot(1, 2, 2)
for status in storms['status'].unique():
data = storms[storms['status'] == status]['wind']
plt.hist(data, bins=20, alpha=0.5, label=status)
plt.xlabel('Wind Speed (knots)', fontsize=11)
plt.ylabel('Frequency', fontsize=11)
plt.title('Wind Speed by Storm Status', fontsize=13, fontweight='bold')
plt.legend()
plt.tight_layout()
plt.show()
# Number of storms per year
storms_per_year = storms.groupby(['year', 'name']).first().groupby('year').size()
plt.figure(figsize=(14, 6))
storms_per_year.plot(kind='line', linewidth=2, color='darkblue')
plt.xlabel('Year', fontsize=12)
plt.ylabel('Number of Named Storms', fontsize=12)
plt.title('Atlantic Named Storms (1975-2024)', fontsize=14, fontweight='bold')
plt.grid(alpha=0.3)
plt.axhline(y=storms_per_year.mean(), color='red', linestyle='--',
label=f'Average: {storms_per_year.mean():.1f}')
plt.legend()
plt.tight_layout()
plt.show()Climate Impact Note
Data suggests increasing frequency of named storms in recent decades, consistent with climate change predictions.
Important: We can responsibly discuss trends within our data range (1975-2020), but extrapolating far into the future requires climate models, not just historical data!
Hofstadter’s Law - Data Science Edition
Mistakes to Watch For
.copy() when creating subsets.loc[] instead of chained [][]“Data cleaning always takes longer than you expect, even when you take into account Hofstadter’s Law.”
