POS Tagging with NLTK and Chunking in NLP [EXAMPLES]

POS Tagging

POS Tagging (Parts of Speech Tagging) is a process to mark up the words in text format for a particular part of a speech based on its definition and context. It is responsible for reading text in a language and assigning a specific token (Parts of Speech) to each word. It is also called grammatical tagging.

Let us learn with an NLTK Part of Speech example:

Input: Everything to permit us.
Output: [('Everything', NN),('to', TO), ('permit', VB), ('us', PRP)]

Steps Involved in the POS Tagging Example

• Tokenize text (word_tokenize)
• Apply pos_tag to the above step, that is nltk.pos_tag(tokenize_text)

NLTK POS Tags examples are listed below:

The above NLTK POS tag list contains all the NLTK POS Tags. The NLTK POS tagger is used to assign grammatical information to each word of the sentence. Installing, importing, and downloading all the packages of POS NLTK is now complete.

What is Chunking in NLP?

Chunking in NLP is a process to take small pieces of information and group them into large units. The primary use of Chunking is making groups of “noun phrases.” It is used to add structure to the sentence by following POS tagging combined with regular expressions. The resulting group of words is called “chunks.” It is also called shallow parsing.

In shallow parsing, there is a maximum of one level between roots and leaves, while deep parsing comprises more than one level. Shallow parsing is also called light parsing or chunking.

Rules for Chunking

There are no pre-defined rules, but you can combine them according to need and requirement. For example, suppose you need to tag Noun, verb (past tense), adjective, and coordinating conjunction from the sentence. You can use the rule below:

chunk:{***?}

The following table shows what the various symbols mean:

Now let us write the code to understand the rule better.

from nltk import pos_tag
from nltk import RegexpParser
text ="learn php from guru99 and make study easy".split()
print("After Split:",text)
tokens_tag = pos_tag(text)
print("After Token:",tokens_tag)
patterns= """mychunk:{***?}"""
chunker = RegexpParser(patterns)
print("After Regex:",chunker)
output = chunker.parse(tokens_tag)
print("After Chunking",output)

Output:

After Split: ['learn', 'php', 'from', 'guru99', 'and', 'make', 'study', 'easy']
After Token: [('learn', 'JJ'), ('php', 'NN'), ('from', 'IN'), ('guru99', 'NN'), ('and', 'CC'), ('make', 'VB'), ('study', 'NN'), ('easy', 'JJ')]
After Regex: chunk.RegexpParser with 1 stages:
RegexpChunkParser with 1 rules:
       '***?'>
After Chunking (S
  (mychunk learn/JJ)
  (mychunk php/NN)
  from/IN
  (mychunk guru99/NN and/CC)
  make/VB
  (mychunk study/NN easy/JJ))

The conclusion from the above Part of Speech tagging Python example is that “make” is a verb that is not included in the rule, so it is not tagged as mychunk.

Use Case of Chunking

Chunking is used for entity detection. An entity is that part of the sentence by which a machine gets the value for any intention.

Example:
Temperature of New York.
Here Temperature is the intention and New York is an entity.

In other words, chunking is used to select subsets of tokens. Please follow the code below to understand how chunking is used to select the tokens. In this example, you will see a graph that corresponds to a chunk of a noun phrase.

 import nltk
text = "learn php from guru99"
tokens = nltk.word_tokenize(text)
print(tokens)
tag = nltk.pos_tag(tokens)
print(tag)
grammar = "NP: {
?*}"
cp  =nltk.RegexpParser(grammar)
result = cp.parse(tag)
print(result)
result.draw()    # It will draw the pattern graphically which can be seen in Noun Phrase chunking

Output:

['learn', 'php', 'from', 'guru99']  -- These are the tokens
[('learn', 'JJ'), ('php', 'NN'), ('from', 'IN'), ('guru99', 'NN')]   -- These are the pos_tag
(S (NP learn/JJ php/NN) from/IN (NP guru99/NN))        -- Noun Phrase Chunking

Noun Phrase chunking Graph

From the graph, we can conclude that “learn” and “guru99” are two different tokens but are categorized as a Noun Phrase, whereas the token “from” does not belong to a Noun Phrase. Chunking is used to categorize different tokens into the same chunk. The result will depend on the grammar that has been selected. Furthermore, Chunking in NLTK is used to tag patterns and to explore text corpora.

Counting POS Tags

We have discussed various pos_tag values previously. Here you will study how to count these tags. Counting tags is crucial for text classification and for preparing features for Natural language operations. We first write the working code and then explain the steps.

from collections import Counter
import nltk
text = "Guru99 is one of the best sites to learn WEB, SAP, Ethical Hacking and much more online."
lower_case = text.lower()
tokens = nltk.word_tokenize(lower_case)
tags = nltk.pos_tag(tokens)
counts = Counter( tag for word,  tag in tags)
print(counts)

Output:

Counter({'NN': 5, ',': 2, 'TO': 1, 'CC': 1, 'VBZ': 1, 'NNS': 1, 'CD': 1, '.': 1, 'DT': 1, 'JJS': 1, 'JJ': 1, 'JJR': 1, 'IN': 1, 'VB': 1, 'RB': 1})

Elaboration of the code:

1. Use the Counter package from the collections module. Counter is a dictionary subclass that stores elements as keys and their counts as values.
2. Import nltk to tokenize the text.
3. Write the text whose pos_tag you want to count.
4. Transform all words to lower case before tokenization.
5. Pass the words through word_tokenize and calculate the pos_tag of each token.
6. Counter then counts the total occurrence of each tag in the text.

Frequency Distribution

Frequency Distribution refers to the number of times an outcome of an experiment occurs. It is used to find the frequency of each word occurring in a document. It uses the FreqDist class defined by the nltk.probability module. The count is incremented by one each time a sample occurs.

For any word, we can check how many times it occurred in a document. For example:

• Count Method: freq_dist.count(‘and’) returns the number of times ‘and’ occurred. It is called the count method.
• Frequency Method: freq_dist.freq(‘and’) returns the frequency of a given sample.

We will write a small program that calculates the frequency distribution of each word in the text.

import nltk
a = "Guru99 is the site where you can find the best tutorials for Software Testing     Tutorial, SAP Course for Beginners. Java Tutorial for Beginners and much more. Please     visit the site guru99.com and much more."
words = nltk.tokenize.word_tokenize(a)
fd = nltk.FreqDist(words)
fd.plot()

Explanation of code:

1. Import the nltk module.
2. Write the text whose word distribution you need to find.
3. Tokenize each word in the text, which is served as input to the FreqDist module of nltk.
4. Apply each word to nltk.FreqDist in the form of a list.
5. Plot the words in the graph using plot().

NOTE: matplotlib must be installed to see the graph. It counts the occurrence of each word in the text and helps in text-based sentiment analysis. The key term is “tokenize”: after tokenizing, each word is checked to determine how many times it occurred.

Collocations: Bigrams and Trigrams

Collocations are the pairs of words occurring together many times in a document. It is calculated by the ratio of the number of those pairs occurring together to the overall word count of the document.

Consider words like ultraviolet rays and infrared rays. The words ultraviolet and rays are not used individually and hence can be treated as a collocation. Another example is the CT Scan, since we do not say CT and Scan separately. Collocation can be categorized into two types:

• Bigrams: combination of two words
• Trigrams: combination of three words

Bigrams and Trigrams provide more meaningful and useful features for the feature extraction stage. These are especially useful in text-based sentiment analysis.

Bigrams Example Code:

import nltk
text = "Guru99 is a totally new kind of learning experience."
Tokens = nltk.word_tokenize(text)
output = list(nltk.bigrams(Tokens))
print(output)

Output:

[('Guru99', 'is'), ('is', 'totally'), ('totally', 'new'), ('new', 'kind'), ('kind', 'of'), ('of', 'learning'), ('learning', 'experience'), ('experience', '.')]

Trigrams Example Code:

Sometimes it becomes important to see a pair of three words in the sentence for statistical analysis and frequency count. This again plays a crucial role in forming NLP (natural language processing) features as well as text-based sentiment prediction. The same code is run for calculating the trigrams.

import nltk
text = "Guru99 is a totally new kind of learning experience."
Tokens = nltk.word_tokenize(text)
output = list(nltk.trigrams(Tokens))
print(output)

Output:

[('Guru99', 'is', 'totally'), ('is', 'totally', 'new'), ('totally', 'new', 'kind'), ('new', 'kind', 'of'), ('kind', 'of', 'learning'), ('of', 'learning', 'experience'), ('learning', 'experience', '.')]

Tagging Sentences

Tagging a sentence refers to adding labels such as verb or noun based on the context of the sentence. Identifying POS tags is complicated, so generic tagging is not possible manually, as some words have ambiguous meanings depending on sentence structure. Converting text into a list is an important step before tagging, as each word is looped and counted for a particular tag.

import nltk
text = "Hello Guru99, You have to build a very good site, and I love visiting your site."
sentence = nltk.sent_tokenize(text)
for sent in sentence:
	 print(nltk.pos_tag(nltk.word_tokenize(sent)))

Code Explanation:

1. Code to import nltk (the Natural Language Toolkit, which contains submodules such as sentence tokenize and word tokenize).
2. Text whose tags are to be printed.
3. Sentence Tokenization.
4. A for loop is implemented where words are tokenized from the sentence and the tag of each word is printed as output.

In a Corpus, there are two types of POS taggers:

1. Rule-Based POS Tagger: For words with ambiguous meaning, a rule-based approach based on contextual information is applied. It is done by analyzing the meaning of the preceding or following word. Therefore words are tagged by the grammatical rules of a particular language, such as capitalization and punctuation. For example, Brill’s tagger.

2. Stochastic POS Tagger: Approaches such as frequency or probability are applied under this method. If a word is mostly tagged with a particular tag in the training set, it is given that tag in the test sentence. The word tag depends not only on its own tag but also on the previous tag, so this method is not always accurate.

POS Tagging with Hidden Markov Model

Tagging problems can also be modeled using a Hidden Markov Model (HMM). It treats input tokens as an observable sequence, while tags are considered hidden states, and the goal is to determine the hidden state sequence. For example, x = x1,x2,…,xn, where x is a sequence of tokens, while y = y1,y2,y3,y4…yn is the hidden sequence.

How Hidden Markov Model (HMM) Works?

HMM uses a joint distribution P(x, y), where x is the input token sequence and y is the tag sequence. The tag sequence for x will be argmax over y1…yn of p(x1,x2,…xn,y1,y2,y3,…). We have categorized tags from the text, but statistics of such tags are vital, so the next part is counting these tags for statistical study.

FAQs