🤖 AI / ML

Bias-Variance Tradeoff

bias variance tradeoff · bias-variance trade-off · bias variance trade off · underfitting overfitting balance · bias variance decomposition

Definition

The bias-variance tradeoff is the principle that a model's error is the sum of systematic error (bias) and sensitivity to the data (variance), and that the two cannot be minimised at once without more data or ensembling.

Decomposing a model’s error

The expected prediction error breaks into three components:

Error = Bias^2 + Variance + Irreducible noise

Bias — the systematic deviation of predictions from the true values. The cause is assumptions
about the data that are too simple.
Variance — the sensitivity of predictions to the specific training sample. The cause is a
model complex enough to have memorised the noise.
Irreducible noise — the random component of the data that no model can predict.

Visualising the tradeoff

Error
  |          Total error
  |        \             /
  |         \    min   /
  |  Bias^2  \       /  Variance
  |            \   /
  |             \ /
  +-------------------- Model complexity
   Simple             Complex

The optimum sits where total error is lowest — neither at maximum nor at minimum complexity.

Applying it in recommendations

Situation	Problem	Fix
A linear model misses the patterns	High bias	Move to matrix factorisation or a two-tower model
The model is excellent on history, poor on new data	High variance	Strengthen regularisation, add data
Rare items are predicted badly	High variance on few observations	A content-based fallback for the cold start

Ensembling as a balance

Ensemble methods — Random Forest, Gradient Boosting — work directly on this tradeoff:

Bagging (Random Forest): trains many trees on subsamples and averages them, which lowers
variance without raising bias
Boosting (XGBoost, LightGBM): corrects errors in sequence, which lowers bias while variance is
held in check by regularisation

Tip: in recommender systems, mixing strategies — combining popularity, collaborative filtering
and content-based models — is ensembling by another name. Each model has its own bias-variance
profile, and blending cancels out the weaknesses of each.

Frequently asked questions

How do you tell high bias from high variance?

High bias (underfitting) means error is high on both the training set and the test set — the model has not captured the patterns in the data. High variance (overfitting) means error is low on the training set but high on the test set — the model has memorised the noise. The gap between training loss and validation loss is the key diagnostic signal.

How do you reduce bias?

Increase model complexity: add layers to the network, weaken the regularisation, use a more capable architecture. Add features the model was not seeing. In recommendations, move from a simple popularity model to matrix factorisation or a neural network.

How do you reduce variance?

Add more training data — the most reliable method by far. Strengthen regularisation (L1, L2, dropout). Simplify the model. Use ensembling: averaging several models lowers variance while leaving bias where it was.

What does the tradeoff have to do with recommender systems?

Everything. Matrix factorisation with a large number of latent factors has high variance and overfits on rare items, which is part of the cold start problem. Too few factors and bias is high, so the model misses the nuances of taste. Mixing strategies — ensembling several models — is the standard way to balance this in production recommender systems.