Session 1 Theory: Understanding Random Forest for Earth Observation

CoPhil 4-Day Advanced Online Training
DAY 2 - Session 1: Supervised Machine Learning - Part 1

---

Learning Objectives

By the end of this notebook, you will be able to:

1. Understand Decision Trees: Explain how a single decision tree makes predictions through recursive splitting
2. Grasp Ensemble Learning: Describe how Random Forest combines multiple trees through bootstrap sampling and random feature selection
3. Interpret Feature Importance: Analyze which spectral bands or derived indices contribute most to classification
4. Evaluate Model Performance: Read and interpret confusion matrices to assess classification accuracy
5. Apply to EO Context: Connect these concepts to satellite image classification tasks

---

Why Random Forest for Earth Observation?

Random Forest is one of the most popular algorithms for land cover classification because:

• Handles high-dimensional data: Works well with many spectral bands (Sentinel-2 has 13 bands)
• Robust to overfitting: Ensemble approach reduces variance
• Feature importance: Reveals which bands are most informative
• No feature scaling required: Unlike neural networks
• Fast training: Efficient even with large datasets
• Interpretable: Can visualize decision rules

---

Estimated Time: 70 minutes

A. Introduction and Setup (5 minutes)

Let’s start by importing the necessary libraries and setting up our environment for reproducible results.

# Core scientific computing libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# Scikit-learn for machine learning
from sklearn.datasets import make_moons, make_circles, make_classification
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score

# Suppress warnings for cleaner output
import warnings
warnings.filterwarnings('ignore')

# Set random seeds for reproducibility
RANDOM_STATE = 42
np.random.seed(RANDOM_STATE)

# Configure plotting style
plt.style.use('seaborn-v0_8-darkgrid')
sns.set_palette("colorblind")  # Color-blind friendly palette
plt.rcParams['figure.figsize'] = (10, 6)
plt.rcParams['font.size'] = 11

print("✓ Libraries imported successfully!")
print(f"✓ Random state set to: {RANDOM_STATE}")
print(f"✓ NumPy version: {np.__version__}")
print(f"✓ Pandas version: {pd.__version__}")

---

B. Decision Trees Interactive Demo (15 minutes)

What is a Decision Tree?

A Decision Tree is a supervised learning algorithm that makes predictions by learning a series of if-then-else decision rules from data. Think of it like a flowchart:

Is NDVI > 0.3?
├─ Yes: Is NIR > 0.5?
│  ├─ Yes: Forest
│  └─ No: Grassland
└─ No: Is SWIR < 0.2?
   ├─ Yes: Water
   └─ No: Urban

Key Concepts:

• Root Node: The first decision point (top of the tree)
• Internal Nodes: Intermediate decision points
• Leaf Nodes: Final predictions (bottom of the tree)
• Splitting: How the algorithm decides which feature and threshold to use
• Depth: Number of levels in the tree (deeper = more complex)

Let’s Build a Simple Example

# Create a simple 2D classification dataset
# This simulates two spectral bands (e.g., NIR and Red)
X, y = make_moons(n_samples=200, noise=0.25, random_state=RANDOM_STATE)

# Add feature names for EO context
feature_names = ['NIR Reflectance', 'Red Reflectance']
class_names = ['Water/Urban', 'Vegetation']

print(f"Dataset shape: {X.shape}")
print(f"Number of samples: {X.shape[0]}")
print(f"Number of features: {X.shape[1]}")
print(f"Classes: {np.unique(y)}")
print(f"Class distribution: {np.bincount(y)}")

# Visualize the dataset
plt.figure(figsize=(10, 6))
scatter = plt.scatter(X[:, 0], X[:, 1], c=y, cmap='viridis', 
                     s=50, alpha=0.7, edgecolors='k', linewidth=0.5)
plt.xlabel(feature_names[0], fontsize=12)
plt.ylabel(feature_names[1], fontsize=12)
plt.title('Training Data: Two Spectral Bands', fontsize=14, fontweight='bold')
plt.colorbar(scatter, label='Class', ticks=[0, 1])
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

print("\n💡 TIP: In real EO applications, each point would represent a pixel with its spectral reflectance values.")

Train a Single Decision Tree

Let’s train a decision tree and visualize how it splits the feature space.

# Train a decision tree with limited depth
tree = DecisionTreeClassifier(max_depth=3, random_state=RANDOM_STATE)
tree.fit(X, y)

# Calculate training accuracy
train_accuracy = tree.score(X, y)
print(f"Training Accuracy: {train_accuracy:.3f}")
print(f"Tree Depth: {tree.get_depth()}")
print(f"Number of Leaves: {tree.get_n_leaves()}")

# Visualize decision boundaries
def plot_decision_boundary(model, X, y, title="Decision Boundary"):
    """
    Plot decision boundary for a 2D classification problem.
    
    Parameters:
    -----------
    model : trained classifier
    X : array-like, shape (n_samples, 2)
    y : array-like, shape (n_samples,)
    title : str
    """
    # Create mesh grid
    x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
    y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
    xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                         np.linspace(y_min, y_max, 200))
    
    # Predict on mesh grid
    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    
    # Plot
    plt.figure(figsize=(10, 6))
    plt.contourf(xx, yy, Z, alpha=0.3, cmap='viridis', levels=1)
    scatter = plt.scatter(X[:, 0], X[:, 1], c=y, cmap='viridis', 
                         s=50, alpha=0.8, edgecolors='k', linewidth=0.5)
    plt.xlabel(feature_names[0], fontsize=12)
    plt.ylabel(feature_names[1], fontsize=12)
    plt.title(title, fontsize=14, fontweight='bold')
    plt.colorbar(scatter, label='Class', ticks=[0, 1])
    plt.grid(True, alpha=0.3)
    plt.tight_layout()
    plt.show()

plot_decision_boundary(tree, X, y, 
                      title="Decision Tree: How It Splits the Feature Space")

print("\n💡 TIP: Notice the rectangular decision boundaries. Trees can only make")
print("   axis-aligned splits (e.g., 'NIR > 0.5'), not diagonal lines.")

Visualize the Tree Structure

Let’s look inside the tree to see the actual decision rules it learned.

# Plot the tree structure
plt.figure(figsize=(20, 10))
plot_tree(tree, 
         feature_names=feature_names,
         class_names=class_names,
         filled=True,
         rounded=True,
         fontsize=10)
plt.title('Decision Tree Structure', fontsize=16, fontweight='bold', pad=20)
plt.tight_layout()
plt.show()

print("\nHow to Read This Tree:")
print("━" * 60)
print("• Each box is a node with a decision rule (e.g., 'NIR <= 0.5')")
print("• 'gini' measures impurity (0 = pure, 0.5 = mixed)")
print("• 'samples' shows how many training points reach this node")
print("• 'value' shows class distribution [class 0, class 1]")
print("• Color intensity indicates class majority (darker = more confident)")
print("• Leaf nodes (bottom) make the final prediction")

🎯 Interactive Exercise: Effect of Tree Depth

Task: Experiment with different max_depth values and observe how the decision boundary changes.

Questions to consider: 1. What happens with max_depth=1 (a “decision stump”)? 2. What happens with max_depth=10 (very deep tree)? 3. Which depth seems to balance simplicity and accuracy? 4. Can you identify overfitting?

# TODO: Experiment with different max_depth values
# Try: max_depth = 1, 2, 5, 10, None (unlimited)

max_depth_to_test = 1  # TODO: Change this value

tree_experiment = DecisionTreeClassifier(max_depth=max_depth_to_test, 
                                        random_state=RANDOM_STATE)
tree_experiment.fit(X, y)

accuracy = tree_experiment.score(X, y)
print(f"Max Depth: {max_depth_to_test}")
print(f"Training Accuracy: {accuracy:.3f}")
print(f"Actual Tree Depth: {tree_experiment.get_depth()}")
print(f"Number of Leaves: {tree_experiment.get_n_leaves()}")

plot_decision_boundary(tree_experiment, X, y, 
                      title=f"Decision Tree with max_depth={max_depth_to_test}")

print("\n⚠️ COMMON MISTAKE: Setting max_depth=None can lead to overfitting!")
print("   The tree will memorize training data instead of learning patterns.")

---

C. Random Forest Voting Mechanism (15 minutes)

The Power of Ensemble Learning

A single decision tree can be unstable: - Small changes in data can lead to completely different trees - Prone to overfitting (memorizing training data) - High variance in predictions

Random Forest solves this by combining many trees:

1. Bootstrap Sampling: Each tree trains on a random subset of data (sampling with replacement)
2. Random Feature Selection: Each split only considers a random subset of features
3. Majority Voting: Final prediction is the class chosen by most trees

Analogy: Instead of asking one expert (one tree), you ask a committee of experts (forest) and take a vote. This “wisdom of the crowd” is more robust!

# Train a Random Forest with just 5 trees (for visualization)
n_trees = 5
rf_small = RandomForestClassifier(n_estimators=n_trees, 
                                 max_depth=3,
                                 random_state=RANDOM_STATE)
rf_small.fit(X, y)

rf_accuracy = rf_small.score(X, y)
print(f"Random Forest Accuracy (5 trees): {rf_accuracy:.3f}")
print(f"Single Tree Accuracy (from before): {train_accuracy:.3f}")
print(f"\nImprovement: {rf_accuracy - train_accuracy:.3f}")

Visualize Individual Trees in the Forest

Let’s see how each tree makes different decisions.

# Plot decision boundaries for each individual tree
fig, axes = plt.subplots(2, 3, figsize=(18, 12))
axes = axes.ravel()

# Plot each individual tree
for idx, tree in enumerate(rf_small.estimators_):
    ax = axes[idx]
    
    # Create mesh grid
    x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
    y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
    xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                         np.linspace(y_min, y_max, 200))
    
    # Predict
    Z = tree.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    
    # Plot
    ax.contourf(xx, yy, Z, alpha=0.3, cmap='viridis', levels=1)
    ax.scatter(X[:, 0], X[:, 1], c=y, cmap='viridis', 
              s=30, alpha=0.6, edgecolors='k', linewidth=0.3)
    ax.set_xlabel(feature_names[0])
    ax.set_ylabel(feature_names[1])
    ax.set_title(f'Tree {idx + 1}', fontweight='bold')
    ax.grid(True, alpha=0.3)

# Plot the ensemble (Random Forest)
ax = axes[5]
x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                     np.linspace(y_min, y_max, 200))
Z = rf_small.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

ax.contourf(xx, yy, Z, alpha=0.3, cmap='viridis', levels=1)
ax.scatter(X[:, 0], X[:, 1], c=y, cmap='viridis', 
          s=30, alpha=0.6, edgecolors='k', linewidth=0.3)
ax.set_xlabel(feature_names[0])
ax.set_ylabel(feature_names[1])
ax.set_title('Random Forest (Ensemble)', fontweight='bold', color='red')
ax.grid(True, alpha=0.3)

plt.suptitle('Individual Trees vs. Ensemble Decision', 
            fontsize=16, fontweight='bold', y=1.00)
plt.tight_layout()
plt.show()

print("\n💡 TIP: Notice how each tree is slightly different due to bootstrap")
print("   sampling and random feature selection. The ensemble smooths out")
print("   individual errors and creates more stable boundaries.")

Visualize Voting Confidence

Random Forest can provide prediction probabilities based on the proportion of trees voting for each class.

# Get prediction probabilities
x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                     np.linspace(y_min, y_max, 200))

# Predict probabilities for class 1 (Vegetation)
Z_proba = rf_small.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]
Z_proba = Z_proba.reshape(xx.shape)

# Plot confidence
plt.figure(figsize=(12, 7))
contour = plt.contourf(xx, yy, Z_proba, levels=20, cmap='RdYlGn', alpha=0.8)
plt.scatter(X[:, 0], X[:, 1], c=y, cmap='viridis', 
           s=50, alpha=0.7, edgecolors='k', linewidth=0.5)
plt.colorbar(contour, label='Confidence for Vegetation Class')
plt.xlabel(feature_names[0], fontsize=12)
plt.ylabel(feature_names[1], fontsize=12)
plt.title('Random Forest Prediction Confidence\n(Based on Voting Proportions)', 
         fontsize=14, fontweight='bold')
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

print("\nInterpreting Confidence:")
print("━" * 60)
print("• Green (high values): Most trees vote for 'Vegetation'")
print("• Red (low values): Most trees vote for 'Water/Urban'")
print("• Yellow (middle values): Trees are uncertain (mixed votes)")
print("\n💡 TIP: Low confidence regions often indicate:")
print("   - Class boundaries")
print("   - Mixed pixels (in EO context)")
print("   - Need for more training data")

🎯 Interactive Exercise: Effect of Number of Trees

Task: Test how the number of trees affects model stability and accuracy.

Hypothesis: More trees → more stable predictions, but diminishing returns after a certain point.

# TODO: Test different numbers of trees
tree_counts = [1, 5, 10, 50, 100, 200]
accuracies = []

for n in tree_counts:
    # TODO: Create and train a Random Forest with n trees
    rf = RandomForestClassifier(n_estimators=n, 
                               max_depth=3,
                               random_state=RANDOM_STATE)
    rf.fit(X, y)
    acc = rf.score(X, y)
    accuracies.append(acc)
    print(f"n_estimators={n:3d} → Accuracy: {acc:.4f}")

# Plot accuracy vs. number of trees
plt.figure(figsize=(10, 6))
plt.plot(tree_counts, accuracies, marker='o', linewidth=2, markersize=8)
plt.xlabel('Number of Trees', fontsize=12)
plt.ylabel('Training Accuracy', fontsize=12)
plt.title('Effect of Ensemble Size on Accuracy', fontsize=14, fontweight='bold')
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

print("\n📊 Observation: Accuracy stabilizes after ~50-100 trees.")
print("   In practice, 100-500 trees is common for EO applications.")

---

D. Feature Importance Analysis (10 minutes)

Why Feature Importance Matters in EO

Feature importance tells us: - Which spectral bands contribute most to classification - Whether derived indices (NDVI, NDWI) are valuable - If certain features are redundant - How to optimize future data collection

How Random Forest Calculates Importance: - Measures how much each feature decreases impurity (Gini or entropy) - Averaged across all trees in the forest - Higher values = more important for classification

# Create a dataset mimicking Sentinel-2 spectral bands
np.random.seed(RANDOM_STATE)

# Simulate 1000 pixels with 8 "spectral bands"
n_samples = 1000
n_features = 8

# Feature names mimicking Sentinel-2 bands and indices
eo_feature_names = [
    'Blue (B2)',
    'Green (B3)',
    'Red (B4)',
    'NIR (B8)',
    'SWIR1 (B11)',
    'SWIR2 (B12)',
    'NDVI',
    'NDWI'
]

# Create synthetic data with realistic patterns
# Class 0: Water (low NIR, high Blue, high NDWI)
# Class 1: Vegetation (high NIR, low Red, high NDVI)
# Class 2: Urban (moderate all, low NDVI, low NDWI)

X_eo = np.random.rand(n_samples, n_features)
y_eo = np.random.choice([0, 1, 2], size=n_samples)

# Add class-specific patterns
for i in range(n_samples):
    if y_eo[i] == 0:  # Water
        X_eo[i, 0] += 0.3  # Higher Blue
        X_eo[i, 3] -= 0.3  # Lower NIR
        X_eo[i, 7] += 0.4  # Higher NDWI
    elif y_eo[i] == 1:  # Vegetation
        X_eo[i, 3] += 0.5  # Higher NIR
        X_eo[i, 2] -= 0.2  # Lower Red
        X_eo[i, 6] += 0.5  # Higher NDVI
    else:  # Urban
        X_eo[i, 4] += 0.2  # Higher SWIR1
        X_eo[i, 5] += 0.2  # Higher SWIR2

# Clip to [0, 1] range
X_eo = np.clip(X_eo, 0, 1)

print(f"EO Dataset shape: {X_eo.shape}")
print(f"Features: {eo_feature_names}")
print(f"Classes: 0=Water, 1=Vegetation, 2=Urban")
print(f"Class distribution: {np.bincount(y_eo)}")

# Train Random Forest on EO-like data
rf_eo = RandomForestClassifier(n_estimators=100, 
                              max_depth=10,
                              random_state=RANDOM_STATE)
rf_eo.fit(X_eo, y_eo)

# Extract feature importances
importances = rf_eo.feature_importances_
indices = np.argsort(importances)[::-1]  # Sort descending

print("Feature Importance Ranking:")
print("━" * 60)
for i, idx in enumerate(indices):
    print(f"{i+1}. {eo_feature_names[idx]:15s}: {importances[idx]:.4f}")

# Visualize feature importances
plt.figure(figsize=(12, 7))
bars = plt.barh(range(len(importances)), importances[indices], align='center')

# Color bars by importance
colors = plt.cm.viridis(importances[indices] / importances.max())
for bar, color in zip(bars, colors):
    bar.set_color(color)

plt.yticks(range(len(importances)), [eo_feature_names[i] for i in indices])
plt.xlabel('Importance (Mean Decrease in Impurity)', fontsize=12)
plt.ylabel('Feature', fontsize=12)
plt.title('Feature Importance for Land Cover Classification', 
         fontsize=14, fontweight='bold')
plt.grid(True, axis='x', alpha=0.3)
plt.tight_layout()
plt.show()

print("\n💡 TIP: High importance doesn't always mean causation!")
print("   - NDVI is derived from NIR and Red, so they're correlated")
print("   - Consider domain knowledge alongside feature importance")
print("   - Importance can be unstable with correlated features")

🎯 Exercise: Interpret Feature Importance

Questions: 1. Which feature is most important? Why might this be? 2. Are the derived indices (NDVI, NDWI) more or less important than raw bands? 3. Which features could potentially be removed to simplify the model? 4. How does this align with your knowledge of land cover spectral signatures?

Your answers here (double-click to edit):

1. Most important feature:
   • TODO: Write your observation
2. Derived indices vs. raw bands:
   • TODO: Write your analysis
3. Features that could be removed:
   • TODO: Write your suggestions
4. Alignment with spectral signatures:
   • TODO: Write your interpretation

---

E. Confusion Matrix Interpretation (15 minutes)

Why Confusion Matrix?

Overall accuracy can be misleading! Consider: - Dataset: 95% Forest, 5% Mangrove - Model: Predicts everything as Forest - Accuracy: 95% (sounds great!) - Problem: Completely missed mangroves!

Confusion Matrix reveals: - Which classes are well-predicted - Which classes are confused with each other - Class-specific performance (precision, recall)

Key Metrics:

• Precision (User’s Accuracy): Of all pixels predicted as class X, how many are actually class X?
  • Formula: TP / (TP + FP)
  • Important when false positives are costly
• Recall (Producer’s Accuracy): Of all actual class X pixels, how many did we correctly identify?
  • Formula: TP / (TP + FN)
  • Important when false negatives are costly
• F1-Score: Harmonic mean of precision and recall
  • Formula: 2 × (Precision × Recall) / (Precision + Recall)
  • Balances both metrics

# Split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X_eo, y_eo, test_size=0.3, random_state=RANDOM_STATE, stratify=y_eo
)

print(f"Training set size: {X_train.shape[0]}")
print(f"Test set size: {X_test.shape[0]}")
print(f"Training class distribution: {np.bincount(y_train)}")
print(f"Test class distribution: {np.bincount(y_test)}")

print("\n💡 TIP: We use stratified split to maintain class proportions.")

# Train Random Forest
rf_final = RandomForestClassifier(n_estimators=100, 
                                 max_depth=10,
                                 random_state=RANDOM_STATE)
rf_final.fit(X_train, y_train)

# Make predictions
y_pred = rf_final.predict(X_test)

# Calculate overall accuracy
overall_accuracy = accuracy_score(y_test, y_pred)
print(f"Overall Test Accuracy: {overall_accuracy:.3f}")

# Generate confusion matrix
cm = confusion_matrix(y_test, y_pred)
class_labels = ['Water', 'Vegetation', 'Urban']

print("Confusion Matrix (raw counts):")
print("━" * 60)
print(cm)
print("\nRows = Actual class, Columns = Predicted class")

# Visualize confusion matrix as heatmap
plt.figure(figsize=(10, 8))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', 
           xticklabels=class_labels, 
           yticklabels=class_labels,
           cbar_kws={'label': 'Number of Samples'},
           linewidths=1, linecolor='gray')
plt.xlabel('Predicted Class', fontsize=12, fontweight='bold')
plt.ylabel('Actual Class', fontsize=12, fontweight='bold')
plt.title('Confusion Matrix: Land Cover Classification', 
         fontsize=14, fontweight='bold', pad=15)
plt.tight_layout()
plt.show()

print("\nHow to Read This Matrix:")
print("━" * 60)
print("• Diagonal (top-left to bottom-right): Correct predictions")
print("• Off-diagonal: Confusion between classes")
print("• Dark blue cells indicate high counts")
print("\n💡 TIP: Look for patterns in confusion:")
print("   - Are certain class pairs often confused?")
print("   - Do confusions make spectral sense?")

# Calculate normalized confusion matrix (percentages)
cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]

plt.figure(figsize=(10, 8))
sns.heatmap(cm_normalized, annot=True, fmt='.2%', cmap='RdYlGn', 
           xticklabels=class_labels, 
           yticklabels=class_labels,
           vmin=0, vmax=1,
           cbar_kws={'label': 'Percentage'},
           linewidths=1, linecolor='gray')
plt.xlabel('Predicted Class', fontsize=12, fontweight='bold')
plt.ylabel('Actual Class', fontsize=12, fontweight='bold')
plt.title('Normalized Confusion Matrix (Row Percentages)', 
         fontsize=14, fontweight='bold', pad=15)
plt.tight_layout()
plt.show()

print("\n💡 TIP: Normalized matrix shows recall (producer's accuracy) for each class.")
print("   Diagonal values are the percentage correctly classified for each class.")

Calculate Detailed Metrics

# Generate classification report
print("Classification Report:")
print("━" * 80)
report = classification_report(y_test, y_pred, 
                              target_names=class_labels,
                              digits=3)
print(report)

print("\nMetric Definitions:")
print("━" * 80)
print("• Precision (User's Accuracy): TP / (TP + FP)")
print("  → Of predictions for this class, how many were correct?")
print("  → Important when false alarms are costly")
print("")
print("• Recall (Producer's Accuracy): TP / (TP + FN)")
print("  → Of actual samples of this class, how many were found?")
print("  → Important when missing instances is costly")
print("")
print("• F1-Score: 2 × (Precision × Recall) / (Precision + Recall)")
print("  → Harmonic mean balancing precision and recall")
print("")
print("• Support: Number of actual samples in test set")

# Visualize per-class metrics
from sklearn.metrics import precision_score, recall_score, f1_score

precision = precision_score(y_test, y_pred, average=None)
recall = recall_score(y_test, y_pred, average=None)
f1 = f1_score(y_test, y_pred, average=None)

# Create DataFrame for easier plotting
metrics_df = pd.DataFrame({
    'Precision': precision,
    'Recall': recall,
    'F1-Score': f1
}, index=class_labels)

# Plot
ax = metrics_df.plot(kind='bar', figsize=(12, 7), width=0.8)
plt.xlabel('Land Cover Class', fontsize=12)
plt.ylabel('Score', fontsize=12)
plt.title('Per-Class Performance Metrics', fontsize=14, fontweight='bold')
plt.xticks(rotation=0)
plt.ylim([0, 1.05])
plt.legend(loc='lower right', fontsize=11)
plt.grid(True, axis='y', alpha=0.3)
plt.axhline(y=0.8, color='r', linestyle='--', alpha=0.5, label='80% threshold')
plt.tight_layout()
plt.show()

print("\n💡 TIP: In EO applications, different thresholds matter:")
print("   - Disaster mapping: High recall for affected areas (don't miss damage)")
print("   - Urban planning: High precision for built-up (avoid false alarms)")
print("   - Balanced: Use F1-score for overall assessment")

🎯 Exercise: Confusion Analysis

Task: Analyze the confusion matrix and answer these questions:

1. Which class has the highest recall (producer’s accuracy)?
2. Which class has the lowest precision (user’s accuracy)?
3. Which two classes are most often confused with each other?
4. Why might this confusion occur from a spectral perspective?
5. What could you do to improve classification of the weakest class?

Your answers here (double-click to edit):

1. Highest recall class:
   • TODO: Identify and explain
2. Lowest precision class:
   • TODO: Identify and explain
3. Most confused class pair:
   • TODO: Identify the pair
4. Spectral reason for confusion:
   • TODO: Explain using spectral signature knowledge
5. Improvement strategies:
   • TODO: List 2-3 practical approaches

---

F. Concept Check Quiz (10 minutes)

Test your understanding of Random Forest concepts!

Question 1: Decision Tree Splitting

Q: How does a decision tree decide where to split at each node?

1. Randomly selects a feature and threshold
   
2. Uses the feature and threshold that maximizes information gain (or minimizes impurity)
   
3. Always splits at the median value of each feature
   
4. Splits based on alphabetical order of feature names

Your answer: TODO (A, B, C, or D)

Click to reveal answer

Correct Answer: B

Decision trees evaluate all possible splits and choose the one that best separates classes (maximizes information gain or minimizes Gini impurity). This greedy approach finds locally optimal splits at each node.

Question 2: Bootstrap Sampling

Q: In Random Forest, what is bootstrap sampling?

1. Sampling pixels only from the edges of images
   
2. Sampling with replacement to create training subsets for each tree
   
3. Sampling only the most important features
   
4. Sampling validation data separately from training data

Your answer: TODO (A, B, C, or D)

Click to reveal answer

Correct Answer: B

Bootstrap sampling means randomly selecting samples WITH replacement. Each tree gets a different random subset of the training data (approximately 63.2% unique samples), which introduces diversity and reduces correlation between trees.

Question 3: Random Feature Selection

Q: At each split in a Random Forest tree, what does “random feature selection” mean?

1. All features are considered for splitting
   
2. Features are selected in alphabetical order
   
3. Only a random subset of features is considered (typically √n or log₂n)
   
4. The most important feature is always selected

Your answer: TODO (A, B, C, or D)

Click to reveal answer

Correct Answer: C

At each split, Random Forest only considers a random subset of features (controlled by max_features parameter). Default is √n for classification. This decorrelates trees and prevents dominant features from being used in every tree.

Question 4: Feature Importance Interpretation

Q: You’re classifying land cover and find that NDVI has the highest feature importance. What should you conclude?

1. NDVI is the only feature needed; remove all others
   
2. NDVI contributes most to reducing impurity, but other features may still be valuable
   
3. NDVI causes the land cover types (causal relationship)
   
4. All other features are completely irrelevant

Your answer: TODO (A, B, C, or D)

Click to reveal answer

Correct Answer: B

High importance means NDVI is most useful for discrimination, but: - Other features may capture complementary information - Importance doesn’t imply causation - Correlated features share importance - Context and domain knowledge matter!

Question 5: Confusion Matrix - Precision vs. Recall

Scenario: You’re mapping forest fire damage. The confusion matrix shows: - Actual Burned: 100 pixels - Predicted as Burned: 150 pixels - Correctly identified Burned: 90 pixels

Q: Calculate precision and recall for the “Burned” class. Which is more important for this application?

Your calculations: - Precision = TODO (show calculation) - Recall = TODO (show calculation) - More important: TODO (Precision or Recall, and why?)

Click to reveal answer

Answers: - Precision = 90 / 150 = 0.60 (60%) - Of pixels predicted as burned, 60% actually were - Recall = 90 / 100 = 0.90 (90%) - Of actual burned pixels, we found 90%

More Important: Recall

For fire damage assessment: - High recall is critical: We don’t want to miss burned areas (false negatives could delay aid) - Lower precision is acceptable: False alarms can be verified with field checks - It’s better to overestimate damage than underestimate

Question 6: Overfitting in Random Forest

Q: Which scenario is MOST likely to cause overfitting in Random Forest?

1. Using 100 trees instead of 10
   
2. Setting max_depth=None (unlimited depth)
   
3. Using bootstrap sampling
   
4. Using random feature selection

Your answer: TODO (A, B, C, or D)

Click to reveal answer

Correct Answer: B

Unlimited depth allows trees to grow until leaves are pure (or nearly pure), memorizing training data. Signs of overfitting: - Very high training accuracy (>99%) - Much lower test accuracy - Overly complex decision boundaries

Prevention: - Set max_depth (e.g., 10-20) - Set min_samples_split (e.g., 5-10) - Set min_samples_leaf (e.g., 2-5)

Note: More trees (A) actually reduces overfitting! Bootstrap (C) and feature selection (D) also help prevent it.

---

Summary and Key Takeaways

Decision Trees

• Learn hierarchical decision rules through recursive splitting
• Create axis-aligned decision boundaries
• Prone to overfitting if too deep
• Easy to interpret and visualize

Random Forest Ensemble

• Combines many trees to reduce variance and improve stability
• Uses bootstrap sampling (bagging) for training diversity
• Uses random feature selection to decorrelate trees
• Final prediction by majority voting (classification) or averaging (regression)
• More robust than single trees, less prone to overfitting

Feature Importance

• Measures contribution of each feature to reducing impurity
• Helps identify most informative spectral bands/indices
• Useful for feature selection and model interpretation
• Should be interpreted with domain knowledge
• Can be unstable with correlated features

Confusion Matrix & Metrics

• Overall accuracy can hide class-specific problems
• Precision (user’s accuracy): Reliability of positive predictions
• Recall (producer’s accuracy): Completeness of detection
• F1-score: Harmonic mean balancing precision and recall
• Choice of metric depends on application cost (false positives vs. false negatives)

For Earth Observation

• Random Forest works well with multi-spectral data
• No feature scaling needed (unlike neural networks)
• Feature importance reveals spectral signature insights
• Confusion patterns often reflect spectral similarity
• Fast training enables rapid iteration

---

Next Steps

In the Hands-On Session, you will: 1. Load real Sentinel-2 data for Palawan, Philippines 2. Extract training samples from land cover polygons 3. Train Random Forest for multi-class land cover classification 4. Optimize hyperparameters (n_estimators, max_depth, etc.) 5. Generate wall-to-wall land cover maps 6. Validate results and interpret errors

Prepare by reviewing: - Sentinel-2 band characteristics (B2, B3, B4, B8, B11, B12) - Philippine land cover types (forest, mangrove, agriculture, urban, water) - Google Earth Engine Python API basics

---

References

1. Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5-32.
2. Belgiu, M., & Drăguţ, L. (2016). Random forest in remote sensing: A review of applications and future directions. ISPRS Journal of Photogrammetry and Remote Sensing, 114, 24-31.
3. Scikit-learn Documentation: Random Forest Classifier
4. ESA Sentinel-2 User Handbook: https://sentinels.copernicus.eu/documents/247904/685211/Sentinel-2_User_Handbook

---

End of Theory Notebook

Developed for CoPhil 4-Day Advanced Online Training on AI/ML for Earth Observation