Home / Predictive Modeling & Machine Learning / 203.4.6 model-Bias Variance Tradeoff

203.4.6 model-Bias Variance Tradeoff

Model Bias and Variance

  • Over fitting
  • Low Bias with High Variance
  • Low training error – ‘Low Bias’
  • High testing error
  • Unstable model – ‘High Variance’
  • The coefficients of the model change with small changes in the data
  • Under fitting
  • High Bias with low Variance
  • High training error – ‘high Bias’
  • testing error almost equal to training error
  • Stable model – ‘Low Variance’
  • The coefficients of the model doesn’t change with small changes in the data

The Bias-Variance Decomposition

\[Y = f(X)+\epsilon\] \[Var(\epsilon) = \sigma^2\] \[Squared Error = E[(Y -\hat{f}(x_0))^2 | X = x_0 ]\] \[= \sigma^2 + [E\hat{f}(x_0)-f(x_0)]^2 + E[\hat{f}(x_0)-E\hat{f}(x_0)]^2\] \[= \sigma^2 + (Bias)^2(\hat{f}(x_0))+Var(\hat{f}(x_0 ))\]

Overall Model Squared Error = Irreducible Error + \(Bias^2\) + Variance

Bias-Variance Decomposition

  • Overall Model Squared Error = Irreducible Error + \(Bias^2\) + Variance
  • Overall error is made by bias and variance together
  • High bias low variance, Low bias and high variance, both are bad for the overall accuracy of the model
  • A good model need to have low bias and low variance or at least an optimal where both of them are jointly low
  • How to choose such optimal model. How to choose that optimal model complexity

Choosing optimal model-Bias Variance Tradeoff

Bias Variance Tradeoff

Test and Training Error

About admin

Check Also

204.4.12 Bootstrap Cross Validation

This will be our last post of our Model Selection and Cross Validation Series. Bootstrap …

Leave a Reply

Your email address will not be published. Required fields are marked *