---
description: What is Tokenization? Tokenization is the process by which big quantity of text is divided into smaller parts called tokens. Natural language processing is used for building applications such as Text
title: "NLTK Tokenize: Words and Sentences Tokenizer with Example"
image: https://www.guru99.com/images/nltk-tokenize.png
---

[Skip to content](#main)

**⚡ Smart Summary**

NLTK Tokenize splits large text into smaller units called tokens, a foundational step in natural language processing. The toolkit provides word\_tokenize for breaking sentences into words and sent\_tokenize for dividing text into individual sentences.

- ✂️ **Tokenization:** Divides large text into smaller units called tokens for analysis.
- 🧠 **NLP Foundation:** Serves as a base step for stemming, lemmatization, and text-to-numeric conversion.
- 🔤 **word\_tokenize():** Splits a sentence into individual words, separating punctuation.
- 📝 **sent\_tokenize():** Breaks a passage into separate sentences.
- 📊 **Use Case:** Combining both lets you compute features such as average words per sentence for machine learning.

Read More

![NLTK Tokenize](https://www.guru99.com/images/nltk-tokenize.png)

## What is Tokenization?

**Tokenization** is the process by which a large quantity of text is divided into smaller parts called tokens. These tokens are very useful for finding patterns and are considered as a base step for stemming and lemmatization. Tokenization also helps to substitute sensitive data elements with non-sensitive data elements.

Natural language processing is used for building applications such as Text classification, [intelligent chatbot](https://www.guru99.com/best-ai-chatbots.html), sentimental analysis, language translation, etc. It becomes vital to understand the pattern in the text to achieve the above-stated purpose.

For the time being, don’t worry about stemming and lemmatization but treat them as steps for textual data cleaning using NLP (Natural language processing). We will discuss stemming and lemmatization later in the tutorial. Tasks such as **Text classification or spam filtering** makes use of NLP along with deep learning libraries such as Keras and [Tensorflow](https://www.guru99.com/tensorflow-tutorial.html).

Natural Language toolkit has very important module NLTK **tokenize** sentences which further comprises of sub-modules

1. word tokenize
2. sentence tokenize

## Tokenization of words

We use the method **word\_tokenize()** to split a sentence into words. The output of word tokenization can be converted to Data Frame for better text understanding in machine learning applications. It can also be provided as input for further text cleaning steps such as punctuation removal, numeric character removal or stemming. Machine learning models need numeric data to be trained and make a prediction. Word tokenization becomes a crucial part of the text (string) to numeric data conversion. Please read about [Bag of Words or CountVectorizer](https://en.wikipedia.org/wiki/Bag-of-words_model). Please refer to below word tokenize NLTK example to understand the theory better.

```
from nltk.tokenize import word_tokenize
text = "God is Great! I won a lottery."
print(word_tokenize(text))

Output: ['God', 'is', 'Great', '!', 'I', 'won', 'a', 'lottery', '.']
```

[![Tokenization of Words]()](https://www.guru99.com/images/1/100118_0611_TokenizeWor1.png)

### Code Explanation

1. word\_tokenize module is imported from the NLTK library.
2. A variable “text” is initialized with two sentences.
3. Text variable is passed in word\_tokenize module and printed the result. This module breaks each word with punctuation which you can see in the output.

## Tokenization of Sentences

Sub-module available for the above is sent\_tokenize. An obvious question in your mind would be **why sentence tokenization is needed when we have the option of word tokenization**. Imagine you need to count average words per sentence, how you will calculate? For accomplishing such a task, you need both NLTK sentence tokenizer as well as NLTK word tokenizer to calculate the ratio. Such output serves as an important feature for machine training as the answer would be numeric.

Check the below NLTK tokenizer example to learn how sentence tokenization is different from words tokenization.

```
from nltk.tokenize import sent_tokenize
text = "God is Great! I won a lottery."
print(sent_tokenize(text))

Output: ['God is Great!', 'I won a lottery ']
```

We have **12** words and **two sentences** for the same input.

[![Tokenization of Sentences]()](https://www.guru99.com/images/1/100118_0611_TokenizeWor2.png)

**Don't Miss:**

- [How to Download & Install NLTK](https://www.guru99.com/download-install-nltk.html)
- [POS Tagging with NLTK and Chunking in NLP \[EXAMPLES\]](https://www.guru99.com/pos-tagging-chunking-nltk.html "POS Tagging with NLTK and Chunking in NLP [EXAMPLES]")
- [Stemming and Lemmatization in Python NLTK with Examples](https://www.guru99.com/stemming-lemmatization-python-nltk.html)
- [Seq2seq (Sequence to Sequence) Model with PyTorch](https://www.guru99.com/seq2seq-model.html)

### Explanation of the program

1. In a line like the previous program, imported the sent\_tokenize module.
2. We have taken the same sentence. Further sentence tokenizer in NLTK module parsed that sentences and show output. It is clear that this function breaks each sentence.

Above word tokenizer [Python](https://www.guru99.com/python-tutorials.html) examples are good settings stones to understand the mechanics of the word and sentence tokenization.

## FAQs

✂️ What is the difference between word and sentence tokenization?

Word tokenization splits text into individual words and punctuation using word\_tokenize(), while sentence tokenization splits text into separate sentences using sent\_tokenize(). Both are often combined to derive features like words per sentence.

🧠 Why is tokenization important in NLP?

Tokenization is the first step that turns raw text into manageable units, enabling pattern detection, stemming, lemmatization, and conversion to numeric features that machine learning models require for tasks like classification and translation.

🌐 Can NLTK tokenize languages other than English?

Yes. NLTK supports several languages by loading the appropriate Punkt model, for example sent\_tokenize(text, language=’french’). Support varies by language, and some scripts may need specialized tokenizers.

🤖 How is tokenization used in large language models?

Large language models convert text into tokens, often subword pieces, before processing. The model predicts the next token based on previous ones, so tokenization directly affects context length, cost, and output quality.

🧩 Can AI tokenizers handle subword units?

Yes. Modern AI tokenizers use subword methods such as Byte Pair Encoding or WordPiece, splitting rare words into smaller pieces. This keeps vocabularies compact while still representing unfamiliar or compound words.

#### Summarize this post with:

ChatGPTPerplexityGrokGoogle AI

**Stay Updated on AI** **Get Weekly AI Skills, Trends, Actionable Advice.**

##### Sign up for the newsletter

Subscribe for Free

You have successfully subscribed.  
Please check your inbox.

![AI-Newsletter]() Chosen by over **350,000+** professionals

[Scroll to top](#wrapper)

×

Search for:

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://www.guru99.com/#organization","name":"Guru99","sameAs":["https://www.facebook.com/Guru99Official","https://twitter.com/guru99com"],"logo":{"@type":"ImageObject","@id":"https://www.guru99.com/#logo","url":"https://www.guru99.com/images/guru99-logo-v1-150x59.png","contentUrl":"https://www.guru99.com/images/guru99-logo-v1-150x59.png","caption":"Guru99","inLanguage":"en-US"}},{"@type":"WebSite","@id":"https://www.guru99.com/#website","url":"https://www.guru99.com","name":"Guru99","publisher":{"@id":"https://www.guru99.com/#organization"},"inLanguage":"en-US"},{"@type":"ImageObject","@id":"https://www.guru99.com/images/nltk-tokenize.png","url":"https://www.guru99.com/images/nltk-tokenize.png","width":"700","height":"250","caption":"NLTK Tokenize","inLanguage":"en-US"},{"@type":"BreadcrumbList","@id":"https://www.guru99.com/tokenize-words-sentences-nltk.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":"1","item":{"@id":"https://www.guru99.com","name":"Home"}},{"@type":"ListItem","position":"2","item":{"@id":"https://www.guru99.com/nltk","name":"NLTK"}},{"@type":"ListItem","position":"3","item":{"@id":"https://www.guru99.com/tokenize-words-sentences-nltk.html","name":"NLTK Tokenize: Words and Sentences Tokenizer with Example"}}]},{"@type":"WebPage","@id":"https://www.guru99.com/tokenize-words-sentences-nltk.html#webpage","url":"https://www.guru99.com/tokenize-words-sentences-nltk.html","name":"NLTK Tokenize: Words and Sentences Tokenizer with Example","dateModified":"2026-10-10T16:03:26+05:30","isPartOf":{"@id":"https://www.guru99.com/#website"},"primaryImageOfPage":{"@id":"https://www.guru99.com/images/nltk-tokenize.png"},"inLanguage":"en-US","breadcrumb":{"@id":"https://www.guru99.com/tokenize-words-sentences-nltk.html#breadcrumb"}},{"@type":"Person","@id":"https://www.guru99.com/author/editorial","name":"Editorial Team","url":"https://www.guru99.com/author/editorial","image":{"@type":"ImageObject","@id":"https://www.guru99.com/images/favicon-new-logo.png","url":"https://www.guru99.com/images/favicon-new-logo.png","caption":"Editorial Team","inLanguage":"en-US"},"sameAs":["https://www.guru99.com/editorial-team"],"worksFor":{"@id":"https://www.guru99.com/#organization"}},{"articleSection":"NLTK","headline":"NLTK Tokenize: Words and Sentences Tokenizer with Example","description":"What is Tokenization? Tokenization is the process by which big quantity of text is divided into smaller parts called tokens. Natural language processing is used for building applications such as Text","keywords":"nltk, python, bigdata","speakable":{"@type":"SpeakableSpecification","cssSelector":[".entry-title",".summary"]},"@type":"Article","author":{"@id":"https://www.guru99.com/author/editorial","name":"Editorial Team"},"dateModified":"2026-10-10T16:03:26+05:30","image":{"@id":"https://www.guru99.com/images/nltk-tokenize.png"},"copyrightYear":"2026","name":"NLTK Tokenize: Words and Sentences Tokenizer with Example","subjectOf":[{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"What is the difference between word and sentence tokenization?","acceptedAnswer":{"@type":"Answer","text":"Word tokenization splits text into individual words and punctuation using word_tokenize(), while sentence tokenization splits text into separate sentences using sent_tokenize(). Both are often combined to derive features like words per sentence."}},{"@type":"Question","name":"Why is tokenization important in NLP?","acceptedAnswer":{"@type":"Answer","text":"Tokenization is the first step that turns raw text into manageable units, enabling pattern detection, stemming, lemmatization, and conversion to numeric features that machine learning models require for tasks like classification and translation."}},{"@type":"Question","name":"Can NLTK tokenize languages other than English?","acceptedAnswer":{"@type":"Answer","text":"Yes. NLTK supports several languages by loading the appropriate Punkt model, for example sent_tokenize(text, language='french'). Support varies by language, and some scripts may need specialized tokenizers."}},{"@type":"Question","name":"How is tokenization used in large language models?","acceptedAnswer":{"@type":"Answer","text":"Large language models convert text into tokens, often subword pieces, before processing. The model predicts the next token based on previous ones, so tokenization directly affects context length, cost, and output quality."}},{"@type":"Question","name":"Can AI tokenizers handle subword units?","acceptedAnswer":{"@type":"Answer","text":"Yes. Modern AI tokenizers use subword methods such as Byte Pair Encoding or WordPiece, splitting rare words into smaller pieces. This keeps vocabularies compact while still representing unfamiliar or compound words."}}]}],"@id":"https://www.guru99.com/tokenize-words-sentences-nltk.html#schema-1126890","isPartOf":{"@id":"https://www.guru99.com/tokenize-words-sentences-nltk.html#webpage"},"publisher":{"@id":"https://www.guru99.com/#organization"},"inLanguage":"en-US","mainEntityOfPage":{"@id":"https://www.guru99.com/tokenize-words-sentences-nltk.html#webpage"}}]}
```