intro-to-dplyr.qmd

---
title: "Introduction to dplyr"
author: "Reiko Okamoto"
date: "`r Sys.Date()`"
format: gfm
editor: visual
execute:
  echo: true
---

## 👋Welcome to the tidyverse

#### *What is the tidyverse?*

The [tidyverse](https://tidyverse.tidyverse.org/) is a collection of R packages designed for data science. Arguably, two of the most popular packages in the tidyverse are [dplyr](https://dplyr.tidyverse.org/) for data manipulation and [ggplot2](https://ggplot2.tidyverse.org/) for data visualization. These are also two of the packages we are covering today!

#### *Why learn it?*

The skills you gain are not just limited to R. The concepts, like filtering data and creating plots, are applicable to other languages like SQL and Python. This makes it easier to pick up other tools and languages later. Additionally, the tidyverse is in tune with open science practices, helping you create analyses that are more accessible, transparent, and reproducible.

#### *Keep in mind...*

There's no expectation for you to memorize everything. Even experienced programmers don't have every function memorized - they're constantly googling things! My goal today is to help you get comfortable with, and hopefully interested in, using the tidyverse for data analysis.

## 🐧Meet the Palmer penguins

![Artwork by \@allison_horst](../imgs/penguins.png){fig-align="center"}

We'll use the `penguins` data set from the [palmerpenguins](https://allisonhorst.github.io/palmerpenguins/) package. It contains measurements for adult penguins observed in the islands in the Palmer Archipelago off the Antarctic Peninsula. The data were collected and made available by Dr. Kristen Gorman and the Palmer Station Long Term Ecological Research (LTER) Program. This data set is great for learning because it's small enough to be manageable for beginners but contains a range of different types of data and features.

💻Load the necessary packages:

```{r}
library(tidyverse)
library(palmerpenguins)
```

We only need to install a package once, but we'll need to reload it every time we start a new session.

💻Open the data set documentation:

```{r}
?penguins
```

![Artwork by \@allison_horst](../imgs/culmen_depth.png){fig-align="center" width="500"}

## 1️⃣Get a glimpse of our data: glimpse()

💻Inspect the data using the [`glimpse()`](https://dplyr.tidyverse.org/reference/glimpse.html) function.

```{r}
glimpse(penguins)
```

-   *How many rows?*

-   *How many columns?*

-   *How many data types?*

-   *Are there any missing values?*

## 2️⃣Keep or drop columns: select()

Sometimes, we only need to work with a few specific columns instead of the entire data. The [`select()`](https://dplyr.tidyverse.org/reference/select.html) function makes it easy to do just that. It allows us to focus on the columns we need and ignore the rest, making our code cleaner.

💻Select one column by name:

```{r}
select(penguins, species)
```

💻Select two columns by name:

```{r}
select(penguins, species, island)
select(penguins, c(species, island))
```

💻Select all columns except certain ones:

```{r}
select(penguins, -species, -island)
```

💻Select columns that start with "bill":

```{r}
select(penguins, starts_with("bill"))
```

Similar functions like `ends_with()` and `contains()` are also available to select columns based on the ending or presence of specific characters.

💻Select numeric variables:

```{r}
select(penguins, where(is.numeric))
```

## 3️⃣Keep rows that match a condition: filter()

Filtering data is a common task in data analysis. Use the [`filter()`](https://dplyr.tidyverse.org/reference/filter.html) function to focus on specific observations.

💻Show only the penguins of the Gentoo species:

```{r}
filter(penguins, species == "Gentoo")
```

💻Find all Adelie penguins that are also on Torgersen Island:

```{r}
filter(penguins, species == "Adelie", island == "Torgersen")
```

The comma acts as an AND operator, meaning both conditions must be true for a row to be included.

💻Find all penguins that either Adelie or Gentoo:

```{r}
filter(penguins, species == "Adelie" | species == "Gentoo")
filter(penguins, species != "Chinstrap")
```

The vertical bar acts as an OR operator, meaning a row is returned if any of the conditions are true.

#### 📝Exercise 1

1.  Select the `sex` and `year` columns from the data.

2.  Filter the data to show only the rows where `flipper_length_mm` is greater than 200.

3.  Filter the data to find all penguins that are either on Biscoe Island or Torgersen Island.

    ```{r}
    select(penguins, sex, year)

    filter(penguins, flipper_length_mm > 200)

    filter(penguins, island == "Biscoe" | island == "Torgersen")
    ```

## 4️⃣Pipes

💻Filter the data to only include Adelie penguins, and then keep only the columns that start with "bill":

```{r}
select(filter(penguins, species == "Adelie"), starts_with("bill"))
```

This works, but it can get difficult to read, especially as our code grows more complex. The nested functions force us to read the code inside-out, making it less intuitive

💻Use the pipe to do the same thing in a more streamlined way:

Keyboard shortcuts for the pipe:

-   Windows: Ctrl + Shift + M

-   Mac: Cmd + Shift + M

```{r}
penguins |> 
  filter(species == "Adelie") |> 
  select(starts_with("bill"))
```

The pipe allows us to pass the output of one function directly to the next. This approach makes our code easier to read because the operations flow left to right, top to bottom.

#### 📝Exercise 2

Using the pipe, filter the data to include only female penguins with `bill_length_mm` less than or equal to 40, and then select the `species` and `bill_length_mm` columns.

```{r}
penguins |> 
  filter(sex == "female", bill_length_mm <= 40) |> 
  select(species, bill_length_mm)
```

## 5️⃣Create and modify columns: mutate()

In data analysis, it's common to derive new variables from existing ones. The [`mutate()`](https://dplyr.tidyverse.org/reference/mutate.html) function is essential for these tasks.

💻Create a new column called `body_mass_kg` (i.e., the `body_mass_g` column converted from grams to kilograms):

```{r}
penguins |> 
  mutate(body_mass_kg = body_mass_g / 1000)
```

By separating the new columns with a comma, we can create multiple new variables in one function call.

💻Use [`if_else()`](https://dplyr.tidyverse.org/reference/if_else.html) to create a column called `large_penguin`, which indicates whether the penguin's body mass is greater than 4,000 grams:

```{r}
penguins |> 
  mutate(large_penguin = if_else(body_mass_g > 4000, TRUE, FALSE))
```

💻Use [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) to categorize the penguins based on their body mass:

```{r}
penguins |> 
  mutate(size_category = case_when(
    body_mass_g < 3000 ~ "small",
    body_mass_g < 4000 ~ "medium",
    body_mass_g >= 4000 ~ "large",
    .default = "unknown"
  ))
```

`if_else()` is best for binary conditions where we need to choose between two outcomes. In contrast, `case_when()` is ideal for handling multiple conditions, allowing us to return different values based on various criteria.

💻Use [`across()`](https://dplyr.tidyverse.org/reference/across.html) to efficiently round values in multiple columns simultaneously:

```{r}
penguins |> 
  mutate(across(.cols = c(bill_length_mm, bill_depth_mm), .fns = round))
```

#### 📝Exercise 3

1.  Create a new column called `bill_depth_cm` that converts the `bill_depth_mm` columns from millimeters to centimeters.

2.  Create a new column called `flipper_size` that categorizes penguins as short, average, or long based on their `flipper_length_mm`. Hint: Define short as less than 190 mm, average as between 190 and 210 mm, and long as greater than 210 mm.

    ```{r}
    penguins |> 
      mutate(bill_depth_cm = bill_depth_mm / 0.1)

    penguins |> 
      mutate(flipper_size = case_when(
        flipper_length_mm < 190 ~ "short",
        190 <= flipper_length_mm & flipper_length_mm <= 210 ~ "average",
        flipper_length_mm > 210 ~ "long",
        .default = "unknown"
      ))
    ```

## 6️⃣Compute summary statistics: summarise()

We often need to summarize our data to understand key characteristics (e.g., measures of central tendency, measures of variability, frequency). The [`summarise()`](https://dplyr.tidyverse.org/reference/summarise.html) function lets us calculate these summary statistics efficiently.

💻Calculate the mean body mass for all the penguins in the data.

```{r}
penguins |> 
  summarise(mean_body_mass = mean(body_mass_g, na.rm = TRUE))
```

The function creates a new data frame with a single row containing the summary statistic.

## 7️⃣Group by one or more variables: group_by()

In data analysis, a common task is to split our data into groups, apply a function to each group, and then combine the results. This approach is known as the split-apply-combine paradigm. The [`group_by()`](https://dplyr.tidyverse.org/reference/group_by.html) function helps us achieve this by allowing us to specify how we want to split our data into groups.

💻Calculate mean body mass for each species:

```{r}
penguins |> 
  group_by(species) |> 
  summarise(mean_body_mass = mean(body_mass_g, na.rm = TRUE))
```

A grouped data frame has all the properties of a regular data frame but has an additional property that describes the grouping structure. R will treat each group as it were a separate data frame, so operations like `summarise()` are applied to each group separately, and then brought back together.

💻Count the number of penguins on each island:

```{r}
penguins |> 
  group_by(island) |> 
  tally()
```

Alternatively, use the [`count()`](https://dplyr.tidyverse.org/reference/count.html) function, which combines `group_by()` and `tally()` in one step.

💻Calculate the mean and standard deviation of body mass for each combination of species and sex:

```{r}
penguins |> 
  group_by(species, sex) |> 
  summarise(mean_body_mass = mean(body_mass_g, na.rm = TRUE),
            sd_body_mass = sd(body_mass_g, na.rm = TRUE))
```

By default, when we apply a grouping with multiple factors, dplyr will keep the last level of grouping after the summary. Here, the output is still grouped by `species`. To remove grouping from a data frame, use the `ungroup()` function or the `.groups = "drop"` argument in the `summarise()` function. Both methods will allow us to continue working with the data as a regular data frame.

```{r}
# option 1
penguins |> 
  group_by(species, sex) |> 
  summarise(mean_body_mass = mean(body_mass_g, na.rm = TRUE),
            sd_body_mass = sd(body_mass_g, na.rm = TRUE)) |> 
  ungroup()

# option 2
df <- penguins |> 
  group_by(species, sex) |> 
  summarise(mean_body_mass = mean(body_mass_g, na.rm = TRUE),
            sd_body_mass = sd(body_mass_g, na.rm = TRUE),
            .groups = "drop")
```

Similar to what we've seen in other functions, we can create multiple summaries in a single step by separating them with commas.

## 8️⃣Sort rows and extract specific values: arrange(), slice(), pull()

Sometimes, we're interested in extracting a particular value from a data frame, like finding the largest or smallest value in a column.

1.  💻Use the [`arrange()`](https://dplyr.tidyverse.org/reference/arrange.html) function to reorder our data by `mean_body_mass` in descending order

2.  💻Once our data is sorted, use [`slice()`](https://dplyr.tidyverse.org/reference/slice.html) to retrieve the row with the largest mean body mass

3.  💻Use [`pull()`](https://dplyr.tidyverse.org/reference/pull.html) to extract the mean body mass value from the data as a single numeric vector

```{r}
df |> 
  arrange(desc(mean_body_mass)) |> 
  slice(1) |> 
  pull(mean_body_mass)
```

#### 📝Exercise 4

Use the dplyr verbs we have learned so far to analyze penguin flipper length:

-   Select the relevant columns: `island`, `species`, and `flipper_length_mm`

-   Group the data by `island` and `species`, then calculate the **average** and **maximum** flipper length for each group

-   Arrange the results by the **average** flipper length in **ascending** order

Hint: Remember to handle missing values using `na.rm = TRUE` when calculating the summary statistics.

```{r}
penguins |> 
  select(island, species, flipper_length_mm) |> 
  group_by(island, species) |> 
  summarise(mean_flipper_len = mean(flipper_length_mm, na.rm = TRUE),
            max_flipped_len = max(flipper_length_mm, na.rm = TRUE),
            .groups = "drop") |> 
  arrange(mean_flipper_len)
```

## 🌟Bonus: lengthen and widen data: pivot_longer(), pivot_wider()

The [tidyr](https://tidyr.tidyverse.org/) package is another key component of the tidyverse. It focuses on reshaping data to ensure it's in the right format for analysis.

Sometimes we need data in a "long" format (more rows, fewer columns) for certain analyses, or in a "wide" format (more columns, fewer rows) for others. For example, a wide data set might have separate columns for different measurements (e.g., bill length, flipper length), but for some analyses or visualizations, we might need to convert this into a long format where all measurements are in a single column, with another column specifying the measurement type.

Imagine our observation of interest is the measurement itself, rather than the penguin...

💻Use the [`pivot_longer()`](https://tidyr.tidyverse.org/reference/pivot_longer.html) function to reshape the `penguins` data so that all the measurements related to the penguins are in a single column, and another column indicates what measurement type it is:

```{r}
penguins_long <- penguins |> 
  mutate(id = row_number()) |> 
  pivot_longer(
    cols = ends_with("_mm") | ends_with("_g"),
    names_to = "measurement_type",
    values_to = "value"
  )
```

What if we want to go back to the original wide format?

💻Use the [`pivot_wider()`](https://tidyr.tidyverse.org/reference/pivot_wider.html) function to reverse the process:

```{r}
penguins_wide <- penguins_long |> 
  pivot_wider(
    names_from = measurement_type,
    values_from = value
)
```

## 📚Resources

| Function                | Description                                      |
|-------------------------|--------------------------------------------------|
| `dplyr::glimpse()`      | Get a glimpse of your data                       |
| `dplyr::select()`       | Keep or drop columns using their names and types |
| `dplyr::filter()`       | Keep rows that match a condition                 |
| `dplyr::mutate()`       | Create, modify, and delete columns               |
| `dplyr::summarise()`    | Summarise each group down to one row             |
| `dplyr::group_by()`     | Group by one or more variables                   |
| `dplyr::count()`        | Count the observations in each group             |
| `dplyr::arrange()`      | Order rows using column values                   |
| `dplyr::slice()`        | Subset rows using their positions                |
| `dplyr::pull()`         | Extract a single column                          |
| `tidyr::pivot_longer()` | Pivot data from wide to long                     |
| `tidyr::pivot_wider()`  | Pivot data from long to wide                     |

-   [Data transformation with dplyr cheatsheet](https://rstudio.github.io/cheatsheets/html/data-transformation.html?_gl=1*1foigmt*_ga*Mjg1MTU3NDY3LjE3MDU2OTExOTY.*_ga_2C0WZ1JHG0*MTcyNDA3ODA5Ni4yMy4xLjE3MjQwNzgxNDkuMC4wLjA.)