Python Pandas Tutorial: DataFrame, Date Range & Use

What is Pandas in Python?

Pandas is an open-source library that lets you perform data manipulation and analysis in Python, covering numerical tables and time series alike. It provides an easy way to create, manipulate and wrangle data, and it is built on top of NumPy, so NumPy must be present for Pandas to operate.

Why use Pandas?

Data scientists make use of Pandas in Python for the following advantages:

• Easily handles missing data
• It uses Series for one-dimensional data and DataFrame for multi-dimensional data
• It provides an efficient way to slice the data
• It provides a flexible way to merge, concatenate or reshape the data
• It includes a powerful time series toolkit

In short, Pandas supplies powerful, easy-to-use data structures plus the means to operate on them quickly, which is why it anchors most Python data analysis work.

How to Install Pandas

Pandas ships with Anaconda and with most managed notebook services, so it is usually installed already. If the import fails, one of the commands below adds it. The environment built in How to install TensorFlow also contains Pandas.

• pip: pip install pandas
• Anaconda: conda install -c anaconda pandas
• Inside a Jupyter Notebook cell:

import sys
!conda install --yes --prefix {sys.prefix} pandas

What is a Pandas DataFrame?

A Pandas DataFrame is a two-dimensional labelled data structure whose columns may hold different types. It is the standard way to store data in tabular format: rows hold the observations and columns name the information. For instance, price can be the name of a column and 2, 3, 4 can be the price values.

Data frames are well known to statisticians and other data practitioners. The picture below shows that shape — a labelled row index down the side and named columns across the top.

What is a Series?

A Series is a one-dimensional data structure. It can hold any data type, such as integer, float or string, and is useful when you want to perform a computation or return a one-dimensional array. A Series, by definition, cannot have multiple columns; for that case use the DataFrame structure.

The Pandas Series constructor takes the following parameters:

• Data: can be a list, dictionary or scalar value

pd.Series([1., 2., 3.])

0    1.0
1    2.0
2    3.0
dtype: float64

You can add an index with index. It names the rows, and its length must equal the size of the column.

pd.Series([1., 2., 3.], index=['a', 'b', 'c'])

Below you create a Pandas Series with a missing value in the third row. Missing values in Python are shown as NaN, and np.nan from NumPy creates one artificially.

pd.Series([1,2,np.nan])

Output

0    1.0
1    2.0
2    NaN
dtype: float64

Create Pandas DataFrame

You can convert a NumPy array into a DataFrame with pd.DataFrame(). The opposite is also possible: to convert a DataFrame back into an array, use np.array(), or df.to_numpy(), which current Pandas documentation recommends.

## Numpy to pandas
import numpy as np
h = [[1,2],[3,4]] 
df_h = pd.DataFrame(h)
print('Data Frame:', df_h)

## Pandas to numpy
df_h_n = np.array(df_h)
print('Numpy array:', df_h_n)
Data Frame:    0  1
0  1  2
1  3  4
Numpy array: [[1 2]
 [3 4]]

A dictionary works just as well as an input to pd.DataFrame().

dic = {'Name': ["John", "Smith"], 'Age': [30, 40]}
pd.DataFrame(data=dic)

Pandas Range Data

Pandas has a convenient API for creating a range of dates, which becomes the index of a time series. The call takes this form:

pd.date_range(start, periods, freq):

• The first parameter is the starting date
• The second parameter is the number of periods (optional if the end date is specified)
• The last parameter is the frequency: day D, month M and year Y

## Create date
# Days
dates_d = pd.date_range('20300101', periods=6, freq='D')
print('Day:', dates_d)

Output

Day: DatetimeIndex(['2030-01-01', '2030-01-02', '2030-01-03', '2030-01-04', '2030-01-05', '2030-01-06'], dtype='datetime64[ns]', freq='D')

# Months
dates_m = pd.date_range('20300101', periods=6, freq='M')
print('Month:', dates_m)

Output

Month: DatetimeIndex(['2030-01-31', '2030-02-28', '2030-03-31', '2030-04-30','2030-05-31', '2030-06-30'], dtype='datetime64[ns]', freq='M')

Version note: the month alias M used above is deprecated from Pandas 2.2 and removed in Pandas 3.0. On current releases pass freq='ME' for month end; the resulting index is identical.

Inspecting Data

You can check the head or tail of a dataset with head() or tail() called on the DataFrame, as the Pandas example below shows.

Step 1) Create a random sequence with NumPy. The sequence has 4 columns and 6 rows.

random = np.random.randn(6,4)

Step 2) Then you create a DataFrame using Pandas.

Use dates_m as the index, so each row is given a name corresponding to a date, and name the 4 columns with the columns argument.

# Create data with date
df = pd.DataFrame(random,
                  index=dates_m,
                  columns=list('ABCD'))

Because np.random.randn() runs without a seed, your figures will differ from the ones printed here. Call np.random.seed(0) first if you want to reproduce a fixed set.

Step 3) Using the head function

df.head(3)

Step 4) Using the tail function

df.tail(3)

Step 5) An excellent way to get a clue about the data is describe(). It reports the count, mean, std, min, max and percentiles of the dataset.

df.describe()

Slice Data

Slicing pulls a subset of rows or columns out of a DataFrame. You can use the column name to extract one column, as the Pandas example below shows.

## Slice
### Using name
df['A']

2030-01-31   -0.168655
2030-02-28    0.689585
2030-03-31    0.767534
2030-04-30    0.557299
2030-05-31   -1.547836
2030-06-30    0.511551
Freq: M, Name: A, dtype: float64

To select multiple columns you need a double bracket, [[..,..]]. The first pair means you want to select columns; the second pair states which columns to return.

df[['A', 'B']]

You can slice the rows with a colon. The code below returns the first three rows.

### using a slice for row
df[0:3]

The loc accessor selects columns by name. As usual, the value before the comma stands for the rows and the value after it for the columns, and brackets are needed to select more than one column.

## Multi col
df.loc[:,['A','B']]

There is another method for selecting multiple rows and columns. iloc[] uses integer positions instead of column names, so the code below returns the same DataFrame as above.

df.iloc[:, :2]

Drop a Column

You can drop columns with df.drop(), passing the names in the columns argument.

df.drop(columns=['A', 'C'])

Concatenation

You can concatenate two DataFrames with pd.concat(). First create the two frames.

import numpy as np
df1 = pd.DataFrame({'name': ['John', 'Smith','Paul'],
                     'Age': ['25', '30', '50']},
                    index=[0, 1, 2])
df2 = pd.DataFrame({'name': ['Adam', 'Smith' ],
                     'Age': ['26', '11']},
                    index=[3, 4])  

Then concatenate them.

df_concat = pd.concat([df1,df2]) 
df_concat

Drop Duplicates

When a dataset contains duplicate information, drop_duplicates() is an easy way to exclude the repeated rows. df_concat holds a duplicate observation: Smith appears twice in the name column.

df_concat.drop_duplicates('name')

Sort Values

You can sort a frame with sort_values(). Note that Age was created as text here, so the rows sort in string order.

df_concat.sort_values('Age')

Rename Columns

Use rename() to rename a column in Pandas. In each pair the first value is the current column name and the second is the new one.

df_concat.rename(columns={"name": "Surname", "Age": "Age_ppl"})

Pandas Methods Quick Reference

This table maps each common task to the Pandas method that performs it.

FAQs