Naive Bayes Algorithm in Machine Learning

By : Evelyn Clarke Evelyn Clarke
Hours Updated

⚡ Smart Summary

Naive Bayes is a supervised, probabilistic classification algorithm built on Bayes theorem, assuming every feature contributes independently. Its theory, a worked shopping example, the three model variants, benefits, limitations and real-world applications are covered below.

  • 🔘 Definition: A classifier that labels a record by comparing the posterior probability of every candidate class.
  • ☑️ Naive assumption: Each feature is treated as conditionally independent, which rarely holds yet still predicts well.
  • ✅ Bayes formula: P(A|B) equals P(B|A) multiplied by P(A), divided by P(B).
  • 🧪 Worked example: Day, discount and free delivery together give a 97.33 percent likelihood of purchase.
  • 🛠️ Three variants: Multinomial for word counts, Bernoulli for word presence, Gaussian for continuous values.
  • ⚠️ Limitation: Correlated features are ignored, so decision trees or SVMs suit dependent data better.
Read More

Naive Bayes Algorithm in Machine Learning

Naive Bayes Classifier Algorithm

A classifier is a machine learning algorithm that sorts data into one or more of a set of “classes.” An email classifier is one familiar example: it scans every incoming message and attaches a class label of Spam or Not Spam.

Naive Bayes Classifier in machine learning is a supervised learning algorithm used for classification tasks.

The diagram below outlines that flow.

Naive Bayes classifier assigning a class label to an input record

Naive Bayes is used for solving classification problems. It predicts on the basis of the probability of an object. Naive Bayes is based on Bayes Theorem and it is used for text classification mostly. Naive Bayes is a probabilistic classification algorithm that is easy to implement and fast to train.

Since the Naive Bayes classifier is based on Bayes theorem, it is also known as a probability classifier. It predicts based on the probability of an item.

Why Is It Called Naive Bayes?

The name Naive Bayes has two parts: Naive and Bayes. Why naive? The algorithm ignores the order in which features appear, so “You are” and “Are you” look identical to it. It also assumes that no feature influences any other. To recognise the fruit apple you use colour red, shape spherical and taste sweet, and the algorithm treats each of those clues as separate, independent evidence.

  • Naive Bayes classifier assumes that the features are independent of each other. Since this is rarely possible in real-life data, the classifier is called naive.
  • This classification algorithm is based on Bayes theorem, so it is known as the Naive Bayes Classifier.

Naive Bayes Theorem

Bayes theorem is used to find the probability of a hypothesis with conditional probabilities dependent on prior knowledge. This theorem is named after Thomas Bayes. Naive Bayes classifier works on the principle of conditional probability, as given by Bayes theorem.

To understand Bayes theorem, let us look at a simple naive bayes classifier example of tossing two coins. We can get these sample spaces by tossing two coins: {HH, HT, TH, TT}. So, the probabilities of these events will be:

  • Getting two heads = 1/4
  • At least one tail = 3/4
  • Second coin being head given the first coin is tail = 1/2
  • Getting two heads given the first coin is a head = 1/2

Bayes theorem calculates the probability of an event happening based on the probability of a different event that has already taken place. The formula for Bayes theorem is given as:

P(A|B) = (P(B|A) * P(A)) / P(B)

P(A|B) is the probability of event A when event B has already occurred. The probability P(B) must not be zero.

  • You need to find the probability of event A, which is given when event B (evidence) is true.
  • P(A) is the prior probability of A, that is, the probability of the event before any evidence is observed. Here, event B is the value of an unknown instance.
  • P(A|B) is the posterior probability of event A, that is, the probability of A after looking at the evidence B.

RELATED ARTICLES

Working Example of Naive Bayes Classifier

The quickest way to see the formula work is to run it by hand.

Let us take an example of shopping to understand the working of Bayes Naive Classifier. In this dataset, there is a small sample dataset of 30 rows for this example.

Dataset

Sample shopping dataset of 30 rows with Day, Discount, Free Delivery and Buy columns

Problem is to predict whether a person will buy a product on a specific combination of Day, Discount and Free Delivery using Naive Bayes Theorem.

Frequency table counting Buy and No Buy outcomes for each attribute value

Step 1) We will create frequency tables for each attribute using the input types mentioned in the dataset, such as days, discount, and free delivery.

Frequency tables for Day, Discount and Free Delivery attributes

Let the event ‘Buy’ denoted as ‘A’, and independent variables, namely ‘Discount’, ‘Free delivery’, and ‘Day’, denoted as ‘B’. We will use these events and variables to apply Bayes theorem.

Step 2) Now let us calculate the Likelihood tables one by one.

Likelihood table for the Day attribute against Buy and No Buy

Example 1:

Based on this likelihood table, we will calculate the conditional probabilities as below.

P(A) = P(No Buy) = 6/30 = 0.2
P(B) = P(Weekday) = 11/30 = 0.37
P(B/A) = P(Weekday / No Buy) = 2/6 = 0.33

And, find P(A/B) using Bayes theorem,

P(A/B)
= P(No Buy / Weekday)
= P(Weekday / No Buy) * P(No Buy) / P(Weekday)
= (2/6 * 6/30) / (11/30)
= 0.1818

Similarly, if A is Buy, then

= P(Buy / Weekday)
= P(Weekday / Buy) * P(Buy) / P(Weekday)
= (9/24 * 24/30) / (11/30)
= 0.8181

Note: As the P(Buy | Weekday) is more than P(No Buy | Weekday), we can conclude that a customer will most likely buy the product on a Weekday.

Step 3) Similarly, we can calculate the likelihood of occurrence of an event on the basis of all the three variables. Now we will calculate Likelihood tables for all three variables using above frequency tables.

Likelihood tables for Day, Discount and Free Delivery used in the combined calculation

Example 2:

Now, using these three Likelihood tables, we will calculate whether a customer is likely to make a purchase based on a specific combination of ‘Day’, ‘Discount’ and ‘Free delivery’.

Here, let us take a combination of these factors:

  • Day = Holiday
  • Discount = Yes
  • Free Delivery = Yes

When, A = Buy

Calculate the conditional probability of purchase on the following combination of day, discount and free delivery.

Where B is:

  • Day = Holiday
  • Discount = Yes
  • Free Delivery = Yes

And A = Buy

Therefore,

= P(A/B)
= P(Buy / Discount=Yes, Day=Holiday, Free Delivery=Yes)
= ( P(Discount=(Yes/Buy)) * P(Free Delivery=(Yes/Buy)) * P(Day=(Holiday/Buy)) * P(Buy) )
/ ( P(Discount=Yes) * P(Free Delivery=Yes) * P(Day=Holiday) )
= (19/24 * 21/24 * 8/24 * 24/30) / (20/30 * 23/30 * 11/30)
= 0.986

When, A = No Buy

Similarly, Calculate the conditional probability of purchase on the following combination of day, discount and free delivery.

Where B is:

  • Day = Holiday
  • Discount = Yes
  • Free Delivery = Yes

And A = No Buy

Therefore,

= P(A/B)
= P(No Buy / Discount=Yes, Day=Holiday, Free Delivery=Yes)
= ( P(Discount=(Yes/No Buy)) * P(Free Delivery=(Yes/No Buy)) * P(Day=(Holiday/No Buy)) * P(No Buy) )
/ ( P(Discount=Yes) * P(Free Delivery=Yes) * P(Day=Holiday) )
= (1/6 * 2/6 * 3/6 * 6/30) / (20/30 * 23/30 * 11/30)
= 0.027

Step 4) Hence,

Probability of purchase = 0.986

Probability of no purchase = 0.027

Finally, we have conditional probabilities to buy on this day. Let us now generalize these probabilities to obtain the Likelihood of the events.

  • Sum of probabilities = 0.986 + 0.027 = 1.013
  • Likelihood of purchase = 0.986 / 1.013 = 97.33 %
  • Likelihood of No purchase = 0.027 / 1.013 = 2.67 %

The two scores add up to 1.013 rather than 1 because the independence assumption makes each estimate approximate, so dividing by the total rescales them into percentages.

Note that, as 97.33% is greater than 2.67%. We can conclude that the average customer will buy on a holiday with a discount and free delivery.

Types of Naive Bayes Model

There are many types of Naive Bayes Classifiers. Here we have discussed Multinomial, Bernoulli and Gaussian Naive Bayes classifiers.

Variant Feature type Typical use
Multinomial Word counts Topic and document classification
Bernoulli Binary present or absent flags Short texts and spam filtering
Gaussian Continuous numeric values Sensor readings and measurements

1. Multinomial Naive Bayes

This type of Naive Bayes model is used for document classification problems. It works with features that represent the frequency of words in a document. The classifier considers the occurrence and count of words to determine the probability of a document belonging to a specific category, such as sports, politics, or technology.

2. Bernoulli Naive Bayes

This is similar to the multinomial Naive Bayes. Bernoulli Naive Bayes classifier is used for document classification tasks. However, it uses boolean predictors. It represents whether a word is present or not and takes only values Yes or No. The classifier calculates the probabilities based on whether a word occurs in the text or not.

3. Gaussian Naive Bayes

This classifier is used in case of continuous value but not discrete value. This classifier calculates probabilities using the parameters of the Gaussian distribution, i.e., mean and variance.

Gaussian bell curve used to model continuous features in Naive Bayes

The formula for conditional probability changes to,

Gaussian Naive Bayes conditional probability formula using mean and variance

The scikit-learn library adds two more variants: Complement Naive Bayes for imbalanced text and Categorical Naive Bayes for discrete categories.

Benefits and Limitations of Naive Bayes Classifier

There are various advantages and disadvantages of the Naive Bayes algorithm in machine learning.

Benefits of Naive Bayes Classifier

  • Simplicity and Efficiency: Naive Bayes is simple and easy to train and implement. It is efficient because of low computational cost. It can handle large datasets efficiently.
  • Fast Training and Prediction: Naive Bayes does not require as much training data because of the independence between features. It can predict fast once the model is trained.
  • Scalability: Naive Bayes can handle high-dimensional datasets with a large number of features. It performs well even when the number of features is greater than the number of training examples. It scales with the number of data points and predictors. It handles both continuous and discrete data.
  • Robustness to Irrelevant Features: It is not sensitive to irrelevant features.
  • Works well with Small Training Sets: Naive Bayes can provide reasonable results even with limited training data. It can handle situations where the number of training instances is small.

Limitation of Naive Bayes Classifier

Naive Bayes in machine learning assumes that all features are independent of each other. So, it cannot learn relationships between different features in the data. It treats each feature as if it has no relation with the others.

A second caution: the class probabilities it reports are poorly calibrated, so the confidence figure attached to a prediction is not a reliable probability.

To overcome this problem, you can use Decision Trees, Random Forests, Support Vector Machines (SVM), Neural Networks etc. These algorithms have the ability to learn complex relationships and dependencies between features in the data. So these can predict more accurate results.

Applications of Naive Bayes Classifier

Since this algorithm is fast and efficient, you can use it to make real-time predictions.

Spam Detection

Email services (such as Gmail) use this algorithm to determine whether an email is spam. This algorithm is excellent for spam filtering.

Sentiment Analysis

It can classify text as positive, negative, or neutral based on features like word choice, sentence structure, and context. It finds applications in social media monitoring, customer reviews, and market research.

Document Classification

It can classify documents into categories such as sports, politics, technology, or finance based on the frequency or presence of specific words or features within the document.

Recommender Systems

It can analyze user preferences, historical data, and item features to predict user interests or preferences for recommending products, movies, or articles.

This classifier algorithm is also used in Face Recognition, Weather Prediction, Medical Diagnosis, Shopping, News Classification etc. You can implement Naive Bayes in Python, where the sklearn.naive_bayes module provides every variant described above.

FAQs

Import the variant you need from sklearn.naive_bayes, split the data with train_test_split, then call fit() on the training rows and predict() on the test rows. GaussianNB suits continuous features, while MultinomialNB and BernoulliNB handle text counts and binary word flags.

If a category never appears with a class in training, its conditional probability becomes zero and wipes out the whole product. Laplace smoothing adds one to every count so nothing collapses to zero. Scikit-learn exposes this as the alpha parameter.

Neither wins outright. Naive Bayes trains faster, needs less data and copes with high-dimensional text. Logistic regression models correlated features and produces better-calibrated probabilities. On small text datasets Naive Bayes often leads; with more data logistic regression overtakes it.

Hold out a test set and compare predictions with the true labels using a confusion matrix, then derive precision, recall and F1. Accuracy alone misleads on imbalanced data such as spam, where one class dominates the sample.

Lowercase the text, strip punctuation, remove stop words and optionally stem the tokens, then turn each document into a count or TF-IDF vector. Bernoulli variants want binary presence flags instead of counts. Apply identical steps at training and prediction time.

Naive Bayes is the simplest Bayesian network: one class node with every feature hanging directly off it and no links between features. A general Bayesian network lets you draw those dependency edges, so it models correlations that Naive Bayes deliberately ignores.

Automated machine learning tools search smoothing values, feature representations and variant choice, then rank candidates by cross-validated score. That removes most manual trial and error — you still decide which metric matters and whether the winner behaves sensibly.

GitHub Copilot drafts the boilerplate quickly — imports, train-test split, fit and predict calls — from a short comment. Always check the variant it picks and the evaluation code, because a plausible script can still train the wrong model.

Summarize this post with: