R Select(), Filter(), Arrange(), Pipeline with Example

What is dplyr in R?

dplyr is the data manipulation package of the tidyverse. It replaces the bracket notation of base R with a small set of verbs, each one named after the action it performs and each one taking the data frame as its first argument. That consistent shape is what allows the verbs to be chained together.

This tutorial covers the first three verbs plus the pipe. The aggregate function tutorial covers group_by() and summarise() in depth.

Dataset Used in This Tutorial

The library called dplyr contains valuable verbs to navigate inside the dataset. Through this tutorial, you will use the Travel times dataset. The dataset records the trips a driver makes between home and workplace. There are fourteen variables in the dataset, including:

• DayOfWeek: Identify the day of the week the driver uses his car
• Distance: The total distance of the journey
• MaxSpeed: The maximum speed of the journey
• TotalTime: The length in minutes of the journey

The dataset holds 205 observations, and the rides took place from Monday to Friday.

First of all, you need to:

• load the dataset
• check the structure of the data.

One handy dplyr function is glimpse(). It plays the same role as base R str() but prints one row per variable with the type and the first few values, which is far easier to scan on a wide dataset.

library(dplyr) 
PATH <- "https://raw.githubusercontent.com/guru99-edu/R-Programming/master/travel_times.csv"
df <- read.csv(PATH)
glimpse(df)

Output:

## Observations: 205
## Variables: 14
## $ X              <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, ...
## $ Date           <fctr> 1/6/2012, 1/6/2012, 1/4/2012, 1/4/2012, 1/3/20...
## $ StartTime      <fctr> 16:37, 08:20, 16:17, 07:53, 18:57, 07:57, 17:3...
## $ DayOfWeek      <fctr> Friday, Friday, Wednesday, Wednesday, Tuesday,...
## $ GoingTo        <fctr> Home, GSK, Home, GSK, Home, GSK, Home, GSK, GS...
## $ Distance       <dbl> 51.29, 51.63, 51.27, 49.17, 51.15, 51.80, 51.37...
## $ MaxSpeed       <dbl> 127.4, 130.3, 127.4, 132.3, 136.2, 135.8, 123.2...
## $ AvgSpeed       <dbl> 78.3, 81.8, 82.0, 74.2, 83.4, 84.5, 82.9, 77.5,...
## $ AvgMovingSpeed <dbl> 84.8, 88.9, 85.8, 82.9, 88.1, 88.8, 87.3, 85.9,...
## $ FuelEconomy    <fctr> , , , , , , -, -, 8.89, 8.89, 8.89, 8.89, 8.89...
## $ TotalTime      <dbl> 39.3, 37.9, 37.5, 39.8, 36.8, 36.8, 37.2, 37.9,...
## $ MovingTime     <dbl> 36.3, 34.9, 35.9, 35.6, 34.8, 35.0, 35.3, 34.3,...
## $ Take407All     <fctr> No, No, No, No, No, No, No, No, No, No, No, No...
## $ Comments       <fctr> , , , , , , , , , , , , , , , Put snow tires o...	

The Comments variable clearly needs a closer look: the first observations are all empty.

sum(df$Comments =="")

Code Explanation

• sum(df$Comments == “”): Count the observations equal to an empty string in the Comments column of df

Output:

## [1] 181

With the data loaded, start with the verb that trims the columns.

select()

We will begin with the select() verb. We don’t necessarily need all the variables, and a good practice is to select only the variables you find relevant.

We have 181 missing observations, almost 90 percent of the dataset. If you decide to exclude them, you won’t be able to carry on the analysis.

The other possibility is to drop the variable Comment with the select() verb.

We can select variables in different ways with select(). Note that, the first argument is the dataset.

- `select(df, A, B ,C)`: Select the variables A, B and C from df dataset.
- `select(df, A:C)`: Select all variables from A to C from df dataset.
- `select(df, -C)`: Exclude C from the dataset from df dataset.	

You can use the third way to exclude the Comments variable.

step_1_df <- select(df, -Comments)
dim(df)

Output:

## [1] 205  14

dim(step_1_df)

Output:

## [1] 205  13

The original dataset has 14 features while the step_1_df has 13.

select() Helper Functions in R

Naming every column becomes impractical once a dataset has dozens of variables. dplyr ships with helpers that select columns by pattern or by type instead.

# Keep only the timing columns
df %>% select(ends_with("Time"))

# Keep every numeric column
df %>% select(where(is.numeric))

# Move TotalTime to the front, keep the rest in place
df %>% select(TotalTime, everything())

Two more verbs pair naturally with select(). Use rename(new_name = old_name) to relabel a column without dropping the others, and relocate() to move columns without listing them all.

filter()

The filter() verb keeps the observations that satisfy a condition. The filter() works exactly like select(), you pass the data frame first and then a condition separated by a comma:

filter(df, condition)
arguments:
- df: dataset used to filter the data
- condition:  Condition used to filter the data	

One criteria

First of all, you can count the number of observations within each level of a factor variable.

table(step_1_df$GoingTo)

Code Explanation

• table(): Count the number of observations per level. It expects a factor or character variable
• table(step_1_df$GoingTo): Count the number of trips towards each destination

Output:

## 
##  GSK Home 
##  105  100	

The function table() indicates 105 rides are going to GSK and 100 to Home.

We can filter the data to return one dataset with 105 observations and another one with 100 observations.

# Select observations where GoingTo == Home
select_home <- filter(df, GoingTo == "Home")
dim(select_home)

Output:

## [1] 100  14

# Select observations where GoingTo == GSK (work)
select_work <- filter(df, GoingTo == "GSK")
dim(select_work)

Output:

## [1] 105  14

Multiple criteria

We can filter a dataset with more than one criteria. For instance, you can extract the observations where the destination is Home and the trip occurred on a Wednesday.

select_home_wed <- filter(df, GoingTo == "Home" & DayOfWeek == "Wednesday")
dim(select_home_wed)

Output:

## [1] 23 14

23 observations matched this criterion.

Common filter() Conditions and Operators in R

filter() keeps every row for which the condition evaluates to TRUE. Rows that evaluate to NA are dropped, which is the behaviour that surprises most beginners.

Three habits prevent most filter() bugs.

• Use %in% instead of chained OR. filter(df, DayOfWeek %in% c(“Monday”, “Friday”)) is shorter and far harder to mistype than two == comparisons joined by a vertical bar.
• Never test a missing value with ==. The expression x == NA returns NA, not TRUE, so the row is silently dropped. Use is.na(x) instead.
• Commas mean AND. filter(df, a, b) is identical to filter(df, a & b), which is why the example earlier in this tutorial works either way.

The Pipe Operator in dplyr

The creation of a dataset requires a lot of operations, such as:

• importing
• merging
• selecting
• filtering
• and so on

The dplyr library comes with a practical operator, %>%, called the pipe. It makes data manipulation cleaner, faster and less prone to error.

This operator is a code which performs steps without saving intermediate steps to the hard drive. If you are back to our example from above, you can select the variables of interest and filter them. We have three steps:

• Step 1: Import data: Import the gps data
• Step 2: Select data: Select GoingTo and DayOfWeek
• Step 3: Filter data: Return only Home and Wednesday

We can use the hard way to do it:

# Step 1
step_1 <- read.csv(PATH)

# Step 2 
step_2 <- select(step_1, GoingTo, DayOfWeek)

# Step 3 
step_3 <- filter(step_2, GoingTo == "Home", DayOfWeek == "Wednesday")

head(step_3)

Output:

##   GoingTo DayOfWeek
## 1    Home Wednesday
## 2    Home Wednesday
## 3    Home Wednesday
## 4    Home Wednesday
## 5    Home Wednesday
## 6    Home Wednesday	

That is not a convenient way to chain many operations. Every intermediate result stays in the environment, and the names quickly stop meaning anything.

Use the pipe operator %>% instead. You name the data frame once at the start and every step flows from it.

Basic syntax of pipeline

New_df <- df %>%
step 1 %>%
step 2 %>%
...
arguments
- New_df: Name of the new data frame 
- df: Data frame used to compute the step
- step: Instruction for each step
- Note: the last instruction does not need a trailing `%>%`, because nothing follows it
- Note: assigning the result to a new object is optional. Without it, the output prints in the console.

You can create your first pipe following the steps enumerated above.

# Create the data frame filter_home_wed.It will be the object return at the end of the pipeline
filter_home_wed <- 

#Step 1
read.csv(PATH) %>% 

#Step 2
select(GoingTo, DayOfWeek) %>% 

#Step 3
filter(GoingTo == "Home",DayOfWeek == "Wednesday")
identical(step_3, filter_home_wed)

Output:

## [1] TRUE

The two objects are identical, so the pipe produced exactly the same result in a single readable statement.

arrange()

In the previous tutorial, you learn how to sort the values with the function sort(). The library dplyr has its sorting function. It works like a charm with the pipeline. The arrange() verb can reorder one or many rows, either ascending (default) or descending.

- `arrange(A)`: Ascending sort of variable A
- `arrange(A, B)`: Ascending sort of variable A and B
- `arrange(desc(A), B)`: Descending sort of variable A and ascending sort of B

We can sort the distance by destination.

# Sort by destination and distance
step_2_df <- step_1_df %>%
    arrange(GoingTo, Distance)
head(step_2_df)

Output:

##     X       Date StartTime DayOfWeek GoingTo Distance MaxSpeed AvgSpeed
## 1 193  7/25/2011     08:06    Monday     GSK    48.32    121.2     63.4
## 2 196  7/21/2011     07:59  Thursday     GSK    48.35    129.3     81.5
## 3 198  7/20/2011     08:24 Wednesday     GSK    48.50    125.8     75.7
## 4 189  7/27/2011     08:15 Wednesday     GSK    48.82    124.5     70.4
## 5  95 10/11/2011     08:25   Tuesday     GSK    48.94    130.8     85.7
## 6 171  8/10/2011     08:13 Wednesday     GSK    48.98    124.8     72.8
##   AvgMovingSpeed FuelEconomy TotalTime MovingTime Take407All
## 1           78.4        8.45      45.7       37.0         No
## 2           89.0        8.28      35.6       32.6        Yes
## 3           87.3        7.89      38.5       33.3        Yes
## 4           77.8        8.45      41.6       37.6         No
## 5           93.2        7.81      34.3       31.5        Yes
## 6           78.8        8.54      40.4       37.3         No

dplyr Verbs in R: Quick Reference

The table below lists every verb used in this tutorial, with the syntax and what each call returns:

glimpse(df)

select(df, A, B ,C)

select(df, A:C)

select(df, -C)

filter(df, condition1)

filter(df, condition1 & condition2)

arrange(A)

arrange(A, B)

arrange(desc(A), B)

step 1 %>% step 2 %>% step 3

FAQs