ML Data Preparation & Evaluation

This skill helps you prepare raw data for machine learning and evaluate model performance. Follow the workflow below for systematic data preparation.

Quick Start

# Clean and prepare your data
python scripts/data_cleaning.py --input data.csv --output cleaned_data.csv

# Transform features
python scripts/data_transformation.py --input cleaned_data.csv --output transformed_data.csv

# Split for training
python scripts/data_splitting.py --input transformed_data.csv --train-ratio 0.7 --val-ratio 0.15

# Evaluate model predictions
python scripts/model_evaluation.py --actual actual.csv --predicted predictions.csv

Workflow Overview

1. Data Collection → Gather from databases, APIs, files, or web scraping
2. Data Cleaning → Handle missing values, remove duplicates, filter outliers
3. Data Transformation → Normalize, encode, engineer features
4. Data Splitting → Create train/validation/test sets
5. Model Evaluation → Calculate performance metrics

1. Data Collection

Supported Sources

pandas.read_csv()

pd.read_csv('data.csv')

sqlalchemy

pd.read_sql(query, connection)

requests

requests.get(url).json()

beautifulsoup4

BeautifulSoup(html, 'html.parser')

Best Practices

• Validate data types immediately after loading
• Check for encoding issues (UTF-8 is standard)
• Log the number of records collected
• Store metadata about collection time and source

2. Data Cleaning

Missing Values

Strategies by data type:

Use the cleaning script:

python scripts/data_cleaning.py \
  --input data.csv \
  --numeric-strategy median \
  --categorical-strategy most_frequent \
  --remove-duplicates \
  --filter-outliers zscore:3

Duplicates

• Always check for exact duplicates:

  df.drop_duplicates()

• Check for near-duplicates on key columns
• Decide whether to keep first, last, or aggregate

Outliers

Detection methods:

Decision framework:

• Remove if clearly erroneous (e.g., age = 200)
• Transform if valid but extreme (log transform)
• Keep if legitimate edge cases (fraud detection)

3. Data Transformation

Normalization & Standardization

(X - min) / (max - min)

(X - μ) / σ

(X - median) / IQR

Script usage:

python scripts/data_transformation.py \
  --input cleaned_data.csv \
  --normalize zscore \
  --columns "feature1,feature2,feature3"

Encoding Categorical Variables

Text encoding:

• Bag of Words: Simple word counts
• TF-IDF: Weighted by document frequency
• Bigrams/Trigrams: Capture word sequences

Feature Engineering

Common patterns:

# Date/time features
df['hour'] = df['timestamp'].dt.hour
df['day_of_week'] = df['timestamp'].dt.dayofweek
df['is_weekend'] = df['day_of_week'].isin([5, 6])

# Ratios and combinations
df['price_per_sqft'] = df['price'] / df['sqft']
df['total_value'] = df['quantity'] * df['unit_price']

# Binning
df['age_group'] = pd.cut(df['age'], bins=[0, 18, 35, 50, 100], 
                         labels=['child', 'young', 'middle', 'senior'])

4. Data Splitting

Standard Split Ratios

Splitting Strategies

Stratified Split (classification with imbalanced classes):

python scripts/data_splitting.py \
  --input data.csv \
  --stratify target_column \
  --train-ratio 0.7 \
  --val-ratio 0.15

Time Series Split (temporal data):

• Train on earlier periods
• Test on later periods
• Never shuffle time series data

K-Fold Cross-Validation (small datasets):

• K=5 or K=10 typical
• Each fold used once as validation
• Average metrics across folds

5. Model Evaluation

Classification Metrics

(TP+TN) / Total

TP / (TP+FP)

TP / (TP+FN)

2×(P×R)/(P+R)

TN / (TN+FP)

Script usage:

python scripts/model_evaluation.py \
  --actual actual_labels.csv \
  --predicted predictions.csv \
  --metrics "accuracy,precision,recall,f1,roc_auc,mcc"

Regression Metrics

mean((y - ŷ)²)

sqrt(MSE)

1 - SS_res/SS_tot

Confusion Matrix

                Predicted
              Positive  Negative
Actual Positive    TP        FN
Actual Negative    FP        TN

Key insights:

• High FP: Model is too aggressive (lower threshold)
• High FN: Model is too conservative (raise threshold)
• Diagonal dominance: Good performance
• Off-diagonal patterns: Systematic errors to investigate

Common Patterns & Pitfalls

✅ Do

• Always split data BEFORE any feature engineering
• Use stratified splits for imbalanced classification
• Keep test set completely untouched until final evaluation
• Document all transformations for reproducibility
• Check for data leakage (future info in training)

❌ Don't

• Don't normalize using test set statistics
• Don't impute missing values after splitting (fit on train only)
• Don't use accuracy for imbalanced datasets
• Don't evaluate on training data only
• Don't shuffle time series data

⚠️ Watch Out For

• Data leakage: Target information in features
• Target imbalance: Use appropriate metrics (F1, MCC, ROC-AUC)
• Overfitting: Large gap between train and test performance
• Underfitting: Poor performance on both train and test
• Feature scaling: Always scale before distance-based algorithms

Quick Reference

When to use which metric

Script Quick Commands

# Full pipeline
python scripts/data_cleaning.py -i raw.csv -o clean.csv --remove-duplicates --filter-outliers zscore:3
python scripts/data_transformation.py -i clean.csv -o prep.csv --normalize zscore --encode onehot
python scripts/data_splitting.py -i prep.csv --stratify target --train-ratio 0.8
python scripts/model_evaluation.py -a actual.csv -p pred.csv --metrics all

Next Steps

After data preparation:

1. Train your model on the training set
2. Tune hyperparameters using validation set
3. Final evaluation on test set
4. Generate confusion matrix and detailed metrics
5. Analyze errors and iterate on features

For model training and deployment, consider using specialized ML frameworks (scikit-learn, TensorFlow, PyTorch).