🤖 AI / ML

Reinforcement Learning

reinforcement learning · rl · deep rl · multi-armed bandit · thompson sampling · reward function

Definition

Reinforcement learning is a machine learning paradigm in which an agent learns an optimal action policy by interacting with an environment and maximising a cumulative reward function.

The core concepts of RL

Reinforcement learning is built around the interaction between an agent and an environment:

Agent — the system making decisions (a recommendation engine, a conversational assistant).
Environment — the context of the interaction (the shopper on the site, their session, their
history).
State — the current context: the page, the user’s history, the time of day.
Action — the system’s choice: show product A or B, ask question X or Y.
Reward — the feedback signal: a click = +0.1, a purchase = +1.0, an exit = 0.
Policy — the action-selection strategy the agent is optimising.

The RL loop:
State → Agent → Action → Environment → Reward + New State
                   ↑_______________|

Multi-armed bandit: RL in A/B testing

MAB is the most widespread application of RL in e-commerce. The name is an analogy with a slot
machine: the player faces several levers — the variations in an A/B test — each with an unknown
probability of paying out, that is, converting. The task is to maximise the total payout while still
exploring the options.

MAB algorithm	Principle	Character
Epsilon-greedy	Exploit the best with probability 1-ε, explore at random with probability ε	Simple, suboptimal
UCB	Explore the variations with high uncertainty	Deterministic, good on low traffic
Thompson Sampling	Bayesian probability estimates with sampling	Fast convergence, often best in practice

Tip: Thompson Sampling is the preferred choice in e-commerce — it concentrates traffic on the
winner quickly when the gap between variations is large, yet keeps exploring while uncertainty
remains. That reduces the conversion lost compared with a classic A/B test when variations are
unequal.

Exploration versus exploitation in practice

The RL dilemma in a recommendation context: show the shopper more of the product types they already
liked (exploitation), or try new categories (exploration)?

Exploitation only: the shopper sees the same things repeatedly, discovers new categories poorly,
and variety collapses.
Exploration only: irrelevant products get shown for the sake of learning, and conversion is lost
here and now.

In practice recommender systems use soft strategies: 80% to 90% of the slots from exploitation (the
relevant items), 10% to 20% from exploration (new categories chosen on affinity signals).

Deep RL

For complex systems — conversational assistants, marketing campaign control — deep RL is used: a
neural network approximates the value function (the Q-function) or the policy directly. That makes
it possible to handle rich states and delayed rewards, such as a purchase made three days after the
click rather than immediately. It demands considerably more data and computation than MAB.

Frequently asked questions

How is RL different from ordinary supervised learning?

In supervised learning a model trains on labelled examples of input and correct output. In RL there is no correct answer — only a reward signal after the action. The model learns by trial and error: it tries actions, observes the reward and gradually corrects its policy. That lets it optimise long-term outcomes rather than prediction accuracy alone.

What is a multi-armed bandit and how is it related to RL?

A multi-armed bandit (MAB) is a simplified, stateless form of RL: the agent picks an action — a variation in an A/B test — and receives an immediate reward such as a click or a conversion. There is no state history and no delayed reward. MAB is the right fit for A/B optimisation, where the goal is to maximise total conversions while finding the best variation.

What are exploration and exploitation in RL?

The core RL dilemma: the agent can either exploit the best known action or explore new ones to sharpen its estimates. Too much exploitation and it sticks in a local optimum. Too much exploration and it loses revenue on suboptimal variations. In MAB the balance is governed by epsilon-greedy, UCB (upper confidence bound) or Thompson Sampling.

Where is RL used in e-commerce beyond A/B tests?

Search ranking against long-term metrics — not just the click but the purchase after a return visit; ad frequency control that accounts for fatigue; dialogue control in AI assistants, deciding which clarifying question to ask next; and optimising the send time of push notifications.

Gravity Field