🤖 AI / ML

Regularization

regularization · regularisation · l1 regularization · l2 regularization · lasso · ridge · dropout · elastic net

Definition

Regularization is the addition of a penalty term to the loss function for large model parameter values, constraining the complexity of the solution and reducing overfitting.

The problem regularization solves

Machine learning optimises a loss function on the training sample. With no constraints the model
pushes the error towards zero and starts memorising the training data instead of finding patterns
that generalise. On new data such a model performs badly: that is overfitting.

Regularization adds a penalty for model complexity directly into the loss function:

Loss_total = Loss_data + λ × Penalty(weights)

The parameter lambda controls the balance between accuracy on the training set and simplicity of
the model.

The main methods

L1 regularization (lasso):

Penalty = λ × Σ|wᵢ|

Produces sparse models — unimportant weights go to zero. Used for automatic feature selection.

L2 regularization (ridge):

Penalty = λ × Σwᵢ²

Shrinks every weight without zeroing any. More stable when features are multicollinear.

Elastic net: a combination of L1 and L2. Often used when both sparsity and stability are
needed.

Dropout (neural networks):

During training:  switch a neuron off with probability p (usually 0.2–0.5)
During inference: all neurons active, weights scaled by (1-p)

Regularization in recommender systems

Model	Regularization method	Why
Matrix factorization (ALS, SVD)	L2 on user and item embeddings	Stop it memorising active users
Neural CF models	Dropout + L2	Generalisation to cold-start users
Gradient boosting (XGBoost, LightGBM)	L1/L2 + min_child_weight	Control of tree depth
Logistic regression in ranking	L1 or elastic net	Selection of the relevant features

Tip: in e-commerce recommenders it matters especially to regularise the embeddings of active
shoppers. Users with thousands of purchases are heavy examples that a model tends to memorise.
Without regularization it generalises worse to ordinary shoppers with 10 to 50 transactions.

Tuning the lambda hyperparameter

The only correct way is cross-validation:

Split the data into train / validation / test.
Train the model with several lambda values (a logarithmic grid: 0.001, 0.01, 0.1, 1.0, 10.0).
Pick the lambda with the best metric on validation.
Evaluate on test at the end — once.

Never tune lambda on the test set: that is data leakage.

Frequently asked questions

How is L1 regularization different from L2?

L1 (lasso) adds a penalty proportional to the sum of the absolute weight values. That produces sparsity: unimportant weights become exactly zero, which effectively selects features. L2 (ridge) penalises squared weights, shrinking them all evenly without zeroing any. Use L1 when you want automatic feature selection, L2 when every feature is potentially useful.

How do you choose the regularization coefficient?

The coefficient lambda controls the strength of the penalty. As lambda approaches zero there is no regularization and overfitting risk returns. As lambda grows very large the model becomes too simple and underfits. It is tuned by cross-validation: several lambda values are tried on the validation set and the one with the best holdout metric wins.

What is dropout and how is it related to regularization?

Dropout is a regularization technique for neural networks: at every training iteration a share of neurons — usually 20% to 50% — is switched off at random. That prevents co-adaptation, where neurons collude and memorise the training examples together. The intuition: the model learns with unreliable components, which forces it towards more robust representations.

Why does regularization matter specifically in recommender systems?

Recommendation models — matrix factorization, neural networks — work on sparse data: most shoppers have interacted with only a small slice of the catalogue. Without regularization the model overfits to the active users with long histories and performs poorly for everyone else, and everyone else is the majority.