In this tutorial, you will learn
The library called dplyr contains valuable verbs to navigate inside the dataset. Through this tutorial, you will use the Travel times dataset. The dataset collects information on the trip leads by a driver between his home and his workplace. There are fourteen variables in the dataset, including:
- DayOfWeek: Identify the day of the week the driver uses his car
- Distance: The total distance of the journey
- MaxSpeed: The maximum speed of the journey
- TotalTime: The length in minutes of the journey
The dataset has around 200 observations in the dataset, and the rides occurred between Monday to Friday.
First of all, you need to:
- load the dataset
- check the structure of the data.
One handy feature with dplyr is the glimpse() function. This is an improvement over str(). We can use glimpse() to see the structure of the dataset and decide what manipulation is required.
library(dplyr) PATH <- "https://raw.githubusercontent.com/thomaspernet/data_csv_r/master/data/travel_times.csv" df <- read.csv(PATH) glimpse(df)
## Observations: 205 ## Variables: 14 ## $ X <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, ... ## $ Date <fctr> 1/6/2012, 1/6/2012, 1/4/2012, 1/4/2012, 1/3/20... ## $ StartTime <fctr> 16:37, 08:20, 16:17, 07:53, 18:57, 07:57, 17:3... ## $ DayOfWeek <fctr> Friday, Friday, Wednesday, Wednesday, Tuesday,... ## $ GoingTo <fctr> Home, GSK, Home, GSK, Home, GSK, Home, GSK, GS... ## $ Distance <dbl> 51.29, 51.63, 51.27, 49.17, 51.15, 51.80, 51.37... ## $ MaxSpeed <dbl> 127.4, 130.3, 127.4, 132.3, 136.2, 135.8, 123.2... ## $ AvgSpeed <dbl> 78.3, 81.8, 82.0, 74.2, 83.4, 84.5, 82.9, 77.5,... ## $ AvgMovingSpeed <dbl> 84.8, 88.9, 85.8, 82.9, 88.1, 88.8, 87.3, 85.9,... ## $ FuelEconomy <fctr> , , , , , , -, -, 8.89, 8.89, 8.89, 8.89, 8.89... ## $ TotalTime <dbl> 39.3, 37.9, 37.5, 39.8, 36.8, 36.8, 37.2, 37.9,... ## $ MovingTime <dbl> 36.3, 34.9, 35.9, 35.6, 34.8, 35.0, 35.3, 34.3,... ## $ Take407All <fctr> No, No, No, No, No, No, No, No, No, No, No, No... ## $ Comments <fctr> , , , , , , , , , , , , , , , Put snow tires o...
This is obvious that the variable Comments needs further diagnostic. The first observations of the Comments variable are only missing values.
- sum(df$Comments ==""): Sum the observations equalts to "" in the column comments from df
##  181
We will begin with the select() verb. We don't necessarily need all the variables, and a good practice is to select only the variables you find relevant.
We have 181 missing observations, almost 90 percent of the dataset. If you decide to exclude them, you won't be able to carry on the analysis.
The other possibility is to drop the variable Comment with the select() verb.
We can select variables in different ways with select(). Note that, the first argument is the dataset.
- `select(df, A, B ,C)`: Select the variables A, B and C from df dataset. - `select(df, A:C)`: Select all variables from A to C from df dataset. - `select(df, -C)`: Exclude C from the dataset from df dataset.
You can use the third way to exclude the Comments variable.
step_1_df <- select(df, -Comments) dim(df)
##  205 14
##  205 13
The original dataset has 14 features while the step_1_df has 13.
The filter() verb helps to keep the observations following a criteria. The filter() works exactly like select(), you pass the data frame first and then a condition separated by a comma:
filter(df, condition) arguments: - df: dataset used to filter the data - condition: Condition used to filter the data
First of all, you can count the number of observations within each level of a factor variable.
- table(): Count the number of observations by level. Note, only factor level variable are accepted
- table(step_1_df$GoingTo): Count the number of of trips toward the final destination.
## ## GSK Home ## 105 100
The function table() indicates 105 rides are going to GSK and 100 to Home.
We can filter the data to return one dataset with 105 observations and another one with 100 observations.
# Select observations if GoingTo == Home select_home <- filter(df, GoingTo == "Home") dim(select_home)
##  100 14
# Select observations if GoingTo == Work select_work <- filter(df, GoingTo == "GSK") dim(select_work)
##  105 14
We can filter a dataset with more than one criteria. For instance, you can extract the observations where the destination is Home and occured on a Wednesday.
select_home_wed <- filter(df, GoingTo == "Home" & DayOfWeek == "Wednesday") dim(select_home_wed)
##  23 14
23 observations matched this criterion.
The creation of a dataset requires a lot of operations, such as:
- and so on
The dplyr library comes with a practical operator, %>%, called the pipeline. The pipeline feature makes the manipulation clean, fast and less prompt to error.
This operator is a code which performs steps without saving intermediate steps to the hard drive. If you are back to our example from above, you can select the variables of interest and filter them. We have three steps:
- Step 1: Import data: Import the gps data
- Step 2: Select data: Select GoingTo and DayOfWeek
- Step 3: Filter data: Return only Home and Wednesday
We can use the hard way to do it:
# Step 1 step_1 <- read.csv(PATH) # Step 2 step_2 <- select(step_1, GoingTo, DayOfWeek) # Step 3 step_3 <- filter(step_2, GoingTo == "Home", DayOfWeek == "Wednesday") head(step_3)
## GoingTo DayOfWeek ## 1 Home Wednesday ## 2 Home Wednesday ## 3 Home Wednesday ## 4 Home Wednesday ## 5 Home Wednesday ## 6 Home Wednesday
That is not a convenient way to perform many operations, especially in a situation with lots of steps. The environment ends up with a lot of objects stored.
Let's use the pipeline operator %>% instead. We only need to define the data frame used at the beginning and all the process will flow from it.
Basic syntax of pipeline
New_df <- df %>% step 1 %>% step 2 %>% ... arguments - New_df: Name of the new data frame - df: Data frame used to compute the step - step: Instruction for each step - Note: The last instruction does not need the pipe operator `%`, you don't have instructions to pipe anymore Note: Create a new variable is optional. If not included, the output will be displayed in the console.
You can create your first pipe following the steps enumerated above.
# Create the data frame filter_home_wed.It will be the object return at the end of the pipeline filter_home_wed <- #Step 1 read.csv(PATH) % > % #Step 2 select(GoingTo, DayOfWeek) % > % #Step 3 filter(GoingTo == "Home",DayOfWeek == "Wednesday") identical(step_3, filter_home_wed)
##  TRUE
We are ready to create a stunning dataset with the pipeline operator.
In the previous tutorial, you learn how to sort the values with the function sort(). The library dplyr has its sorting function. It works like a charm with the pipeline. The arrange() verb can reorder one or many rows, either ascending (default) or descending.
- `arrange(A)`: Ascending sort of variable A - `arrange(A, B)`: Ascending sort of variable A and B - `arrange(desc(A), B)`: Descending sort of variable A and ascending sort of B
We can sort the distance by destination.
# Sort by destination and distance step_2_df <-step_1_df %>% arrange(GoingTo, Distance) head<step_2_df)
## X Date StartTime DayOfWeek GoingTo Distance MaxSpeed AvgSpeed ## 1 193 7/25/2011 08:06 Monday GSK 48.32 121.2 63.4 ## 2 196 7/21/2011 07:59 Thursday GSK 48.35 129.3 81.5 ## 3 198 7/20/2011 08:24 Wednesday GSK 48.50 125.8 75.7 ## 4 189 7/27/2011 08:15 Wednesday GSK 48.82 124.5 70.4 ## 5 95 10/11/2011 08:25 Tuesday GSK 48.94 130.8 85.7 ## 6 171 8/10/2011 08:13 Wednesday GSK 48.98 124.8 72.8 ## AvgMovingSpeed FuelEconomy TotalTime MovingTime Take407All ## 1 78.4 8.45 45.7 37.0 No ## 2 89.0 8.28 35.6 32.6 Yes ## 3 87.3 7.89 38.5 33.3 Yes ## 4 77.8 8.45 41.6 37.6 No ## 5 93.2 7.81 34.3 31.5 Yes ## 6 78.8 8.54 40.4 37.3 No
In the table below, you summarize all the operations you learnt during the tutorial.
|glimpse||check the structure of a df|
|Identical to str()|
|select()||Select/exclude the variables|
select(df, A, B ,C)
|Select the variables A, B and C|
|Select all variables from A to C|
|filter()||Filter the df based a one or many conditions|
|arrange()||Sort the dataset with one or many variables|
|Ascending sort of variable A|
|Ascending sort of variable A and B|
|Descending sort of variable A and ascending sort of B|
|%>%||Create a pipeline between each step|
step 1 %>% step 2 %>% step 3