Definition: Percentage of correctly classified samples
\[
\text{Overall Accuracy} = \frac{\text{Correct Predictions}}{\text{Total Samples}} = \frac{85+73+95+82+78}{505} = \frac{413}{500} = 82.6\%
\]
Interpretation: - Simple, intuitive metric - Limitation: Can be misleading with imbalanced classes
Example: 95% accuracy sounds great, but if 95% of pixels are forest, a “classify everything as forest” model achieves 95%!
Definition: Percentage of actual class correctly identified (per class)
Also called: Recall, Sensitivity
\[
\text{Producer's Accuracy}_{\text{Forest}} = \frac{85}{105} = 81\%
\]
Interpretation: - “Of all actual forest pixels, how many did we correctly identify?” - Producer’s perspective: How complete is the map? - Low producer’s accuracy → omission error (missing true positives)
Example: 81% for Forest means we missed 19% of actual forest pixels.
Definition: Percentage of predicted class that is correct (per class)
Also called: Precision, Positive Predictive Value
\[
\text{User's Accuracy}_{\text{Forest}} = \frac{85}{100} = 85\%
\]
Interpretation: - “Of all pixels we labeled as forest, how many are truly forest?” - User’s perspective: How reliable is the map? - Low user’s accuracy → commission error (false positives)
Example: 85% for Forest means 15% of “forest” pixels are actually other classes.
Definition: Measure of agreement beyond chance
\[
\kappa = \frac{p_o - p_e}{1 - p_e}
\]
Where: - \(p_o\) = observed accuracy (overall accuracy) - \(p_e\) = expected accuracy by chance
Interpretation: - κ < 0.4: Poor agreement - κ = 0.4-0.6: Moderate agreement - κ = 0.6-0.8: Good agreement - κ > 0.8: Excellent agreement
Why use Kappa? Accounts for agreement by random chance, especially important for imbalanced datasets.