What is Reinforcement Learning? Types, Algorithms & Example

What is Reinforcement Learning?

Reinforcement Learning is a Machine Learning method concerned with how software agents should take actions in an environment. The agent is not shown correct answers; it learns from the reward it receives and adjusts its behaviour to maximise the cumulative reward.

Reinforcement Learning is a branch of machine learning in its own right, alongside supervised and unsupervised learning. When the agent’s policy or value function is represented by a neural network, the combination is called deep reinforcement learning — that pairing is what lets an agent attain a complex objective or maximise a specific dimension over many steps.

Important Components of the Reinforcement Learning Method

Before the algorithms make sense, it helps to name the pieces. The diagram below shows how the agent, the environment, the action and the reward signal fit together in one loop.

Here are some important terms used in Reinforcement Learning:

• Agent: The entity which performs actions in an environment to gain some reward.
• Environment (e): A scenario that an agent has to face.
• Reward (R): An immediate return given to an agent when it performs a specific action or task.
• State (s): State refers to the current situation returned by the environment.
• Policy (π): The strategy applied by the agent to decide the next action based on the current state.
• Value (V): The expected long-term return with discount, as compared to the short-term reward.
• Value Function: It specifies the value of a state, that is, the total amount of reward an agent can expect to accumulate starting from that state.
• Model of the environment: This mimics the behaviour of the environment. It lets you make inferences and also determine how the environment will behave.
• Model-based methods: Methods that solve reinforcement learning problems by first learning or using a model of the environment, then planning against it.
• Q value or action value (Q): Q value is quite similar to value. The only difference between the two is that it takes an additional parameter, the current action.

How Reinforcement Learning works?

An everyday analogy makes the mechanism clear before any notation appears.

Consider the scenario of teaching new tricks to your cat.

• As the cat does not understand English or any other human language, we cannot tell her directly what to do. Instead, we follow a different strategy.
• We emulate a situation, and the cat tries to respond in many different ways. If the cat’s response is the desired one, we give her fish.
• Now whenever the cat is exposed to the same situation, she performs a similar action even more enthusiastically, in expectation of getting more reward (food).
• That is the learning the cat gets about “what to do” from positive experiences.
• At the same time, the cat also learns what not to do when faced with negative experiences.

Example of Reinforcement Learning

The figure below maps that cat story onto the formal loop, with the household as the environment and the fish as the reward.

In this case,

• Your cat is an agent that is exposed to the environment. In this case, it is your house. An example of a state could be your cat sitting, and you using a specific word to make the cat walk.
• Our agent reacts by performing an action transition from one “state” to another “state.”
• For example, your cat goes from sitting to walking.
• The reaction of an agent is an action, and the policy is a method of selecting an action given a state in expectation of better outcomes.
• After the transition, the agent may get a reward or penalty in return.

Reinforcement Learning Algorithms

There are three approaches to implementing a Reinforcement Learning algorithm, and they differ mainly in what the agent stores and learns.

Value-Based

In a value-based Reinforcement Learning method, you try to maximise a value function V(s). In this method, the agent expects a long-term return of the current states under policy π.

Policy-based

In a policy-based RL method, you try to come up with a policy such that the action performed in every state helps you gain maximum reward in the future.

Two types of policy-based methods are:

• Deterministic: For any state, the same action is produced by the policy π.
• Stochastic: Every action has a certain probability, given by the following equation.

Stochastic policy:

π(a|s) = P[At = a | St = s]

Model-Based

In this Reinforcement Learning method, you create a virtual model for each environment. The agent learns to perform in that specific environment.

Characteristics of Reinforcement Learning

Here are important characteristics of reinforcement learning:

• There is no supervisor, only a real number or reward signal
• Sequential decision making
• Time plays a crucial role in Reinforcement problems
• Feedback is often delayed rather than instantaneous
• Agent’s actions determine the subsequent data it receives

Types of Reinforcement Learning

The word “reinforcement” is borrowed from behavioural psychology, and it comes in two forms:

Positive:

It is defined as an event that occurs because of a specific behaviour. It increases the strength and the frequency of the behaviour and impacts positively on the action taken by the agent.

This type of reinforcement helps you maximise performance and sustain change for a more extended period. However, too much reinforcement may lead to over-optimisation of state, which can affect the results.

Negative:

Negative reinforcement is defined as the strengthening of behaviour that occurs because of a negative condition which should have been stopped or avoided. It helps you define the minimum standard of performance. However, the drawback of this method is that it provides only enough to meet the minimum behaviour.

Learning Models of Reinforcement

There are two important learning models in reinforcement learning:

• Markov Decision Process
• Q learning

Markov Decision Process

The following parameters are used to get a solution:

• Set of actions – A
• Set of states – S
• Reward – R
• Policy – π
• Value – V

The mathematical approach for mapping a solution in Reinforcement Learning is formalised as a Markov Decision Process, or MDP. The schematic below shows those five parameters wired into the same agent-environment cycle.

Q-Learning

Q learning is a value-based method of supplying information that tells an agent which action to take.

Let’s understand this method with the following example:

• There are five rooms in a building which are connected by doors.
• Each room is numbered 0 to 4
• The outside of the building can be treated as one big outside area (5)
• Doors number 1 and 4 lead into the building from room 5

The floor plan below numbers those rooms and shows which doors connect them.

Next, you need to associate a reward value with each door:

• Doors which lead directly to the goal have a reward of 100
• Doors that are not directly connected to the target room give zero reward
• As doors are two-way, two arrows are assigned for each room
• Every arrow in the image above carries an instant reward value

Explanation: in this image, each room represents a state, and the agent’s movement from one room to another represents an action.

In the graph below, a state is drawn as a node while the arrows show the available actions and the reward attached to each of them.

For example, an agent traverses from room number 2 to 5:

• Initial state = state 2
• State 2-> state 3
• State 3 -> state (2,1,4)
• State 4-> state (0,5,3)
• State 1-> state (5,3)
• State 0-> state 4

Reinforcement Learning vs. Supervised Learning

The clearest way to place Reinforcement Learning is to set it beside the paradigm most readers already know.

Applications of Reinforcement Learning

Here are applications of Reinforcement Learning:

• Robotics for industrial automation.
• Business strategy planning
• Machine learning and data processing
• It helps you create training systems that provide custom instruction and materials according to the requirements of students.
• Aircraft control and robot motion control

Why use Reinforcement Learning?

Here are prime reasons for using Reinforcement Learning:

• It helps you find which situation needs an action
• Helps you discover which action yields the highest reward over the longer period
• Reinforcement Learning also provides the learning agent with a reward function
• It also allows the agent to figure out the best method for obtaining large rewards

When Not to Use Reinforcement Learning?

You cannot apply a reinforcement learning model in every situation. Here are some conditions in which you should not use one.

• When you have enough labelled data to solve the problem with a supervised learning method
• Reinforcement Learning is computing-heavy and time-consuming, in particular when the action space is large

Challenges of Reinforcement Learning

Here are the major challenges you will face while doing Reinforcement Learning:

• Feature and reward design, which can be very involved
• Parameters may affect the speed of learning.
• Realistic environments can have partial observability.
• Too much reinforcement may lead to an overload of states, which can diminish the results.
• Realistic environments can be non-stationary.

FAQs