How to Remove Duplicates from a List in Python

Python remove Duplicates from a List

A list is a container that contains different Python objects, which could be integers, words, values, etc. It is the equivalent of an array in other programming languages.

So here we will go through different ways in which we can remove duplicates from a given list in Python.

Method 1) Remove duplicates from list using Set

To remove the duplicates from a list, you can make use of the built-in function set(). The specialty of the set() method is that it returns distinct elements.

We have a list : [1,1,2,3,2,2,4,5,6,2,1]. The list has many duplicates which we need to remove and get back only the distinct elements. The list is given to the set() built-in function. Later the final list is displayed using the list() built-in function.

The output that we get is distinct elements where all the duplicate elements are eliminated.

Example

my_list = [1,1,2,3,2,2,4,5,6,2,1]
my_final_list = set(my_list)
print(list(my_final_list))

Output:

[1, 2, 3, 4, 5, 6]

Method 2) Using the Temporary List

To remove duplicates from a given list, you can make use of an empty temporary list. For that first, you will have to loop through the list having duplicates and add the unique items to the temporary list. Later the temporary list is assigned to the main list.

Example

Here is a working example using a temporary list.

my_list = [1, 2, 3, 1, 2, 4, 5, 4 ,6, 2]
print("List Before ", my_list)
temp_list = []

for i in my_list:
    if i not in temp_list:
        temp_list.append(i)

my_list = temp_list

print("List After removing duplicates ", my_list)

Output:

List Before  [1, 2, 3, 1, 2, 4, 5, 4, 6, 2]
List After removing duplicates  [1, 2, 3, 4, 5, 6]

Method 3) Using Dict

We can remove duplicates from the given list by importing OrderedDict from collections. It is available from python2.7 onwards. OrderedDict takes care of returning you the distinct elements in an order in which the key is present.

Let us make use of a list and use the fromkeys() method available in OrderedDict to get the unique elements from the list.

To make use of the OrderedDict.fromkeys() method, you have to import OrderedDict from collections, as shown below:

from collections import OrderedDict

Here is an example to remove duplicates using the OrderedDict.fromkeys() method.

Example

from collections import OrderedDict

my_list = ['a','x','a','y','a','b','b','c']

my_final_list = OrderedDict.fromkeys(my_list)

print(list(my_final_list))

Output:

['a', 'x', 'y', 'b', 'c']

From Python 3.5+ onwards, we can make use of the regular dict.fromkeys() to get the distinct elements from the list. The dict.fromkeys() method returns keys that are unique and helps to get rid of the duplicate values.

An example that shows the working of dict.fromkeys() on a list to give the unique items is as follows:

Example

my_list = ['a','x','a','y','a','b','b','c']
my_final_list = dict.fromkeys(my_list)
print(list(my_final_list))

Output:

['a', 'x', 'y', 'b', 'c']

Method 4) Using for-loop

Using for-loop, we will traverse the list of items to remove duplicates.

First initialize the array to empty i.e myFinallist = []. Inside the for-loop, add a check if the items in the list exist in the array myFinallist. If the items do not exist, add the item to the array myFinallist using the append() method.

So whenever the duplicate item is encountered it will be already present in the array myFinallist and will not be inserted. Let us now check the same in the example below:

Example

my_list = [1,2,2,3,1,4,5,1,2,6]
myFinallist = []
for i in my_list:
    if i not in myFinallist:
myFinallist.append(i)
print(list(myFinallist))

Output:

[1, 2, 3, 4, 5, 6]

Method 5) Using list comprehension

List comprehensions are Python functions that are used for creating new sequences (such as lists, dictionaries, etc.) using sequences that have already been created. This helps you to reduce longer loops and make your code easier to read and maintain.

Let us make use of list comprehension to remove duplicates from the list given.

Example

my_list = [1,2,2,3,1,4,5,1,2,6]
my_finallist = []
[my_finallist.append(n) for n in my_list if n not in my_finallist]
print(my_finallist)

Output:

[1, 2, 3, 4, 5, 6]

Method 6) Using NumPy unique() method

The method unique() from the NumPy module can help us remove duplicates from the list given.

To work with NumPy, first import the numpy module:

Step 1) Import NumPy module

import numpy as np

Step 2) Use your list with duplicates inside the unique() method. The output is converted back to a list using the tolist() method.

myFinalList = np.unique(my_list).tolist()

Step 3) Finally print the list:

print(myFinalList)

The final code with output is as follows:

import numpy as np
my_list = [1,2,2,3,1,4,5,1,2,6]
myFinalList = np.unique(my_list).tolist()
print(myFinalList)

Output:

[1, 2, 3, 4, 5, 6]

Method 7) Using Pandas methods

The Pandas module has a unique() method that will give us the unique elements from the list given.

To work with the Pandas module:

Step 1) Import Pandas module

import pandas as pd

Step 2) Use your list with duplicates inside the unique() method:

myFinalList = pd.unique(my_list).tolist()

Step 3) Print the list:

print(myFinalList)

The final code with output is as follows:

import pandas as pd

my_list = [1,2,2,3,1,4,5,1,2,6]
myFinalList = pd.unique(my_list).tolist()
print(myFinalList)

Output:

[1, 2, 3, 4, 5, 6]

Method 8) Using enumerate() and list comprehension

Here we use the combination of list comprehension and enumerate() to remove the duplicate elements. Enumerate returns an object with a counter to each element in the list. For example (0,1), (1,2) etc. Here the first value is the index, and the second value is the list item.

Each element is checked to see if it exists in the list, and if it does, it is removed from the list.

Example

my_list = [1,2,2,3,1,4,5,1,2,6]
my_finallist = [i for j, i in enumerate(my_list) if i not in my_list[:j]]
print(list(my_finallist))

Output:

[1, 2, 3, 4, 5, 6]

FAQs