Quotation.Rmd

# Quotation {#tidy-eval}

```{r setup, include = FALSE}
source("common.R")
library(rlang)
library(purrr)
```

## Introduction

Now that you understand the tree structure of R code, it's time to come back to one of the fundamental ideas that make `expr()` and `ast()` work: __quasiquotation__.  There are two pieces to quasiquotation:

* Quoting allows you to capture the AST associated with a function argument 
  without evaluating it.
  
* Unquoting allows you to selectively evaluate parts of an expression that 
  would otherwise be quoted.

This combination of these two ideas makes it easy to compose expressions that are mixtures of direct and indirect specifications and makes it easier to solve a wide variety of challenging problems. Another way of thinking about quasiquotation is that it provides a code template: you define an AST with some "holes" that get filled in using the values of other variables.

Quasiquotation is available in base R, but is implemented in the the rlang package. We'll begin the chapter by using rlang to dive into the details of quasiquotation. Next, we'll circle back to base R. You'll learn the closest equivalents to rlang's quoting functions, and the variety of techniques that base R uses for unquoting. We'll finish the chapter with a case study: using quasiquotation to construct calls "by hand". This is a useful technique for creating simple function operators with readable source code, and is a handy technique to work around functions (in base R and elsewhere) that don't support unquoting.

The ideas presented in this chapter are rather similar to Lisp __macros__, as discussed in [Programmer's Niche: Macros in R](http://www.r-project.org/doc/Rnews/Rnews_2001-3.pdf#page=10) by Thomas Lumley. However, macros are run at compile-time, which doesn't have any meaning in R, and always return expressions. They're also somewhat like Lisp [__fexprs__](http://en.wikipedia.org/wiki/Fexpr), functions where all arguments are quoted by default. These terms are useful to know when looking for related techniques in other programming languages. \index{macros} \index{fexprs}

### Motivation

We'll start with a simple and concrete example that helps motivate the need for unquoting, and hence quasiquotation. Imagine you're creating a lot of strings by joining together words:

```{r}
paste("Good", "morning", "Hadley")
paste("Good", "afternoon", "Alice")
```

You are sick and tired of writing all those quotes, and instead you just want to use bare words. To that end, you've managed to write the following function:

```{r}
cement <- function(...) {
  exprs(...) %>% 
    map(expr_name) %>% 
    paste(collapse = " ")
}

cement(Good, morning, Hadley)
cement(Good, afternoon, Alice)
```

(We'll talk about the details of this implementation later; for now just look at the results.)

Formally, this function __quotes__ the arguments in `...`; you can think of it as automatically putting quotation marks around each argument. (That's not precisely true as the intermediate objects it generates are expressions, not string, but it's a useful way to think about the problem.) 

This is nice because we no longer need to type quotes. The problem, however, comes when we want to use variables. It's easy to use variables with `paste()` as we just don't surround them with quotes:

```{r}
name <- "Hadley"
time <- "morning"

paste("Good", time, name)
```

Obviously this doesn't work with `cement()` because every input is automatically quoted:

```{r}
cement(Good, time, name)
```

We need some way to explicitly __unquote__ the input, to tell `cement()` to remove the quote marks that it adds automatically. Here we need someway to tell `cement()` that `time` and `name` should be handled differently to `Good`. Quasiquotation give us a standard tool to do so: `!!`, prounounced bang-bang.

```{r}
cement(Good, !!time, !!name)
```

It's useful to compare `cement()` and `paste()` directly. `paste()` evaluates its arguments, so we need to quote where needed; `cement()` quotes its arguments, so we need to unquote where needed.

```{r, eval = FALSE}
paste("Good", name, time)
cement(Good, !!time, !!name)
```

## Quasiquotation

Now that you've seen the basic idea, it's time to talk a little bit about the theory. The idea of quasiquotation is an old one. It was first developed by a philsopher, Willard van Orman Quine[^1], in the early 1940s. It's needed in philosophy because it helps to be precise between the use and mention of words, i.e. between the object and the words we use to refer to that object. Quasiquotation was first used in a programming language, LISP, in the mid-1970s [@bawden-1999], and has been implemented in most languages from that heritage such as racket (with `` ` `` and `@`), clojure (`` ` `` and `~`), and julia (`:` and `@`). 

[^1]: You might be familiar with the name Quine from "quines", computer programs that when run return a copy of their own source code.

Quasiquotation has only recently become available in R, through the rlang package. Despite it's newness, I teach it in this book because it is a rich and powerful theory that makes many hard problems much eaiser. Quaisquotation in R is a little different to other languages because many functions provide quasiquotation, where in LISP and descendents there is typically only one function that does quasiquotation, and you must call it explicitly when needed. This makes these languages less ambiguous (because there's a clear code signal), but is less appropriate for R because quasiquotation is such an important part of DSLs for data analysis.

The remainder of this section discusses the two sides of quasiquotation (quoting and unquoting), and finishes with a consideration of the downsides.

### Quoting

For interactive exploration, the most important quoting function is `expr()`. It returns its argument exactly as given:

```{r}
expr(x + y)
expr(1 / 2 / 3)
```

Note that that `expr()` captures the AST, not the text, which means comment and whitespace will not be preserved:

```{r}
expr({
  x +              y # comment  
})
```

You can't use `expr()` inside a function to capture an argument:

```{r}
f <- function(x) expr(x)
f(x + y + z)
```

Instead, you need to use `enexpr()`:

```{r}
f <- function(x) enexpr(x)
f(x + y + z)
```

If you need to capture all argument in `...` use `exprs()`:

```{r}
f <- function(...) exprs(...)
f(x = 1, y = 10 * z)
```

Every function that provides quasiquotation must use one of `enexpr()` or `exprs()` or the related `enquo()` and `quos()` variants. In the next chapter, you'll learn about `enquo()` and `quos()` which are important because they also capture execution environment of the expression. 

Let's compare the use of `expr()` and `enexpr()` inside a function: 

```{r}
capture_1 <- function(x) expr(x)
capture_2 <- function(x) enexpr(x)

capture_1(x + y)
capture_2(x + y)
```

`expr()` always yields whatever is passed in. When you need to construct an expression from known inputs use `expr()`. When you need to capture an expression provided by the user in an argument, use `enexpr()`.  

Depending on how you call it, `exprs()` combines some of the behaviour of `expr()` and `enexpr()`. It behaves like `enexpr()` if you pass in `...`, and behaves like `expr()` for all other arguments:

```{r}
f <- function(x, ...) {
  exprs(x = x, ...)
}
f(x = y + 1, y = y + 1)
```

Generally, you'll use `exprs()` in one of two ways:

```{r}
# Interactively creating a list of expressions
exprs(x = x ^ 2, y = y ^ 3, z = z ^ 4)
# short hand for
list(x = expr(x ^ 2), y = expr(y ^ 3), z = expr(z ^ 4))

# To capture all ... inside a function
foo <- function(...) {
  dots <- exprs(...)
}
```

There's not much you can do with a list of expressions yet, but we'll see a few techniques later on in this chapter. Using rlang with purrr to work with list of expressions turns out to be a surprisingly powerful combination.

### Evaluation

The opposite of quoting is evaluating. This is a big topic, so it is covered in depth in the next chapter. For now, we'll focus on a single function: `rlang::eval_tidy()`. This takes an expression and evaluates it.

```{r}
x <- expr(runif(5))
x

eval_tidy(x)
eval_tidy(x)
```

Notice that every time we evaluate this expression we get a different result. This makes these expression different to the lazy evaluation of functions which are only evaluated once, and then return the same results again and again.

Quoting functions side-step evaluation, allowing you to capture the code. This allows you to inspect and transform the AST, or evaluate the code in a different way ("non-standard") to usual. Functions that use these tools are often called non-standard evaluation (NSE) functions. 

### Unquoting

Evaluating works with the entire expression. We can selectively choose to evaluate, or __unquote__, inside an expression with either two tools:

* `!!` called unquote, and pronounced bang-bang
* `!!!` called unquote-splice, and pronounced bang-bang-bang. 

They both replace nodes in the AST. `!!` is a one-to-one replacement. It takes a single expression and inlines the AST at the location of the `!!`. 

```{r}
x <- expr(-1)
expr(f(!!x, y))
```

```{r, echo = FALSE, out.width = NULL}
knitr::include_graphics("diagrams/expression-bang-bang.png", dpi = 450)
```

`!!!` is a one-to-many replacement. It takes a list of expressions and replaces them at the location of the `!!!`.

```{r}
x <- exprs(-1, -2)
expr(f(!!!x, y))
```

```{r, echo = FALSE, out.width = NULL}
knitr::include_graphics("diagrams/expression-bang-bang-bang.png", dpi = 450)
```

There's one final component to quasiquotation: `:=`, called define, and pronounced colon-equals. This is needed because the LHS of `=` is always quoted in function arguments:

```{r}
name <- "x"
value <- 10

lobstr::ast(c(name = value))
```

You can't use `!!` on here because R's grammar only allows a symbol. This makes any attempt to unquote a syntax error:

```{r, eval = FALSE}
lobstr::ast(c(!!name = !!value))
#> Error: unexpected '=' in "lobstr::ast(c(!!name ="
```

To work around this issue, rlang supports `:=`. 

```{r}
lobstr::ast(c(!!name := !!value))
```

`:=` is like a vestigal organ: it's recognised by the parser, but it doesn't have any code associated with it. It looks like an `=` but allows any expression on the LHS, providing a flexible alternative to `=`. If you've used data.table, you might've also seen it used there for similar reasons.

### Calling quoting functions

A function is __referentially transparent__ if you can replace its arguments with their values and its behaviour doesn't change. For example, if a function, `f()`, is referentially transparent and both `x` and `y` are 10, then `f(x)`, `f(y)`, and `f(10)` will all return the same result. This is clearly not 

```{r}
x <- runif(5)

(exp(2 * x) - exp(-2 * x)) / (exp(2 * x) + exp(-2 * x))

y <- 2*x
(exp(y) - exp(-y)) / (exp(y) + exp(-y))

sinh_y <- (exp(y) - exp(-y)) / 2
cosh_y <- (exp(y) + exp(-y)) / 2
sinh_y / cosh_y
```

The biggest downside of NSE is that functions that use it are no longer [referentially transparent](http://en.wikipedia.org/wiki/Referential_transparency_(computer_science)). If you want to call a quoting function inside another function you need to quote and unquote.

```{r}
expr((exp(2 * x) - exp(-2 * x)) / (exp(2 * x) + exp(-2 * x)))

y <- 2 * x
expr((exp(y) - exp(-y)) / (exp(y) + exp(-y)))

y <- expr(2 * x)
expr((exp(!!y) - exp(-!!y)) / (exp(!!y) + exp(-!!y)))
```

Referentially transparent code is easier to reason about because the names of objects don't matter, and because you can always work from the innermost parentheses outwards. You can simplify code by introducing named intermediates. \index{non-standard evaluation!drawbacks} \index{referential transparency}

There are many important functions that by their very nature are not referentially transparent. Take the assignment operator. You can't take `a <- 1` and replace `a` by its value and get the same behaviour. This is one reason that people usually write assignments at the top-level of functions. It's hard to reason about code like this:

```{r}
a <- 1
b <- 2
if ((b <- a + 1) > (a <- b - 1)) {
  b <- b + 2
}
```

Not all functions can be referentially transparent, but it's worth striving for.

### Exercises

1.  It's challenging to see the AST of code like `!!x + !!y` because  
    `ast()` does unquoting. We can work around this by using the base
    equivalent of `expr()` that doesn't do unquoting: `quote()`. Why does
    this work? What does it tell you about unquoting?
    
    ```{r}
    expr <- quote(!!x + !!y)
    lobstr::ast(!!expr)
    ```

## Base R

Now that you understand the basics of quasiquotation it's time to take a look at what base R does. You'll first learn about the quoting functions that base R uses. Next, since base R doesn't have unquoting, you'll learn about the variety of techniques it uses to selectively supress quoting, which we call "non-quoting."

### Quoting functions

There are two main qutoing functions in base R `quote()` and `substitute()`. These are roughly equivalent to `expr()` and `enexpr()`.

*   `quote()` is equivalent to `expr()` without quasiquotation. 
  
    ```{r}
    x2 <- quote(x ^ 2)
    quote(!!x2 + 1)
    
    expr(!!x2 + 1)
    ```

*   `substitute()` is similar to `enexpr()`. Its primary purpose is to capture
    unevaluated arguments, and you'll often see it used in concert with 
    `deparse()` to create labels. For example, `plot.default()` uses 
    `deparse(substitute(x))` for `xlab` if its not otherwise supplied.
    
    However, `substitute()` does quite a bit more. If you give it an expression,
    rather than a symbol, it will substitute in values of symbols defined
    in the current environment. This substitution doesn't happen in the global
    environment, so we need to show it in a function:
    
    ```{r}
    foo <- function(x) {
      y <- 10
      substitute(foo(x, y))
    }
    foo(z / 2)
    ```

There is no built-in equivalent to `exprs()` but you could write your own:
  
```{r}
dots <- function(...) match.call(expand.dots = FALSE)$`...`
dots(x = 1, y = x + 2)
```

This takes advantage of another function that quotes the entire call to the function, not just individual arguments: `match.call()`. You'll see `match.call()` frequently used in modelling functions as way of capturing the complete model specification so that it can be printed in labels.

```{r}
lm(mpg ~ cyl, data = mtcars)
```

Base R also has one function that implements a form of quasiquotation: `bquote()`. It uses `.()` for unquoting, and does not support unquoting-splicing. It is not used to provide quasiquotation for any other function in R.

```{r}
y <- expr(2 * x)
bquote((exp(.(y)) - exp(-.(y))) / (exp(.(y)) + exp(-.(y))))
```

Finally, the formula, `~`, is also a quoting function. We'll come back to it in the next chapter because it is the inspiration for the `quo()` family of functions that capture both the expression and the evaluation environment.

```{r}
f <- y ~ x + 1
f
lobstr::ast(!!f)

attr(f, ".Environment")
```

### Non-quoting

Because base R doesn't use quasiquotation, most quoted arguments have a non-quoted variant. Instead of supplying an bare name, there is typically someway to provide a string instead. There are four basic forms seen in base R:

```{r, include = FALSE}
call <- names(pryr::find_uses("package:base", "match.call"))
subs <- names(pryr::find_uses("package:base", "substitute"))
eval <- names(pryr::find_uses("package:base", "eval"))

intersect(subs, eval)
```

*   A pair of quoting and non-quoting functions. For example, `$` has two 
    arguments, and the second argument is quoted. This is easier to see if you 
    write in prefix form: `mtcars$cyl` is equivalent to `` `$`(mtcars, cyl) ``. 
    If you want to refer to a variable indirectly, you use `[[`, as it 
    takes the name of a variable as a string.
      
    ```{r}
    x <- list(var = 1, y = 2)
    var <- "y"
    
    x$var
    x[[var]]
    ```
  
    `<-`/`assign()` and `::`/`getExportedValue()` work similarly.

*   A pair of quoting and non-quoting arguments. For example, `data()`, `rm()`, 
    and `save()` allow you to provide bare variable names in `...`, or a 
    character vector of variable names in `list`.

    ```{r}
    x <- 1
    rm(x)

    y <- 2
    vars <- c("y", "vars")
    rm(list = vars)
    ```

*   An argument that controls whether a different argument is quoting or 
    non-quoting. For example, in `library()`, the `character.only` argument
    controls the quoting behaviour of of the first argument, `package`:
    
    ```{r, message = FALSE}
    library(MASS)
    
    pkg <- "MASS"
    library(pkg, character.only = TRUE)
    ```
    
    `demo()`, `detach()`, `example()`, and `require()` work similarly.

*   Quoting if evaluation fails. For example, the first argument to `help()`
    is non-quoting if it's an existing variable name (that points to a string);
    otherwise it is quoting. 
    
    ```{r, eval = FALSE}
    # Shows help for var
    help(var)
    
    var <- "mean"
    # Shows help for mean
    help(var)
    ```
    
    `ls()` also autoquotes it's first argument, `name`, with a warning. 


Some quoting functions, like `subset()`, `transform()`, and `with()`, don't have a non-quoting form. Presumably this because they are primarily wrappers around wrappers around `[` and `[<-`.

### `do.call`

As well as the non-quoting variants which basically provide alternatives to `!!`, one function provides a collective alternative to `!!!` and `:=`: `do.call()`. `do.call()` allows you to call a function with a list of arguments, allowing you manipulate that list in the usual way.

```{r}
var <- "x"
value <- 1:3

do.call(data.frame, setNames(list(value), var))
```

```{r}
tibble::tibble(!!var := value)
```

Or if you have many data frames that you want to bind together, you can use `do.call()` + `rbind()`:

```{r}
dfs <- replicate(5, data.frame(x = runif(1)), simplify = FALSE)
do.call(rbind, dfs)
```

Equivalent to `bind_rows(!!!dfs)`.  (For historical reasons, `bind_rows(dfs)` also works but if I was to write it today, it would require explicit unsplicing.)

### Exercises

1.  Read the standard non-standard evaluation rules found at
    <http://developer.r-project.org/nonstandard-eval.pdf>.

1.  Why does `as.Date.default()` use `substitute()` and `deparse()`?
    Why does `pairwise.t.test()` use them? Read the source code.

1.  `pairwise.t.test()` assumes that `deparse()` always returns a length one 
    character vector. Can you construct an input that violates this expectation? 
    What happens?

1.  Base functions `match.fun()`, `page()`, and `ls()` all try to
    automatically determine whether you want standard or non-standard
    evaluation. Each uses a different approach. Figure out the essence
    of each approach then compare and contrast.

## Case study: constructing calls with quasiquotation {#construct-calls}

In base R, you can construct a call using the `call()` function. We are going to use the similar function `rlang::lang()`. The chief difference is that `lang()` supports quasiquotation. This makes it considerably easier to generate certain types of call.

The basics of `lang()` are simple. You create a call giving the name of a function, followed by the arguments:

```{r}
lang("+", 1, 2)
lang("foo", x = 1, y = 2)
```

Here we've used a convenient shortcut: we've given it the name of the fuction as a string not a call. In most cases a string is easier to type and directly equivalent to the `quote()`d equivalent:

```{r}
lang(expr(f), 1, 2)
lang("f", 1, 2)
```

However, this will not work if the function is generated by a function call. Note the subtle difference in these two calls:

```{r}
lang(quote(f()), 1, 2)
lang("f()", 1, 2)
```

The first uses the function generated by calling `f()`, the second calls a function with the confusing name `f()`:

```{r}
`f()` <- function(x) x + 1
`f()`(1)
```

To construct more complex calls, two new quasiquotation calls come in handy: 

*   `!!!`, pronounced bang-bang-bang, the unquote-splice operator. It allows you
    to splice in a list. Simply including the list in the call doesn't yield 
    quite what you want:

    ```{r}
    args <- list(x = 1, y = 2)
    lang("f", args, z = 3)
    ```
    
    Here we the unquote-splice operator:
    
    ```{r}
    lang("f", !!!args, z = 3)
    ```

*   `:=`, pronounced colon-equals, the definition operator. It works like `=` but
    allows you to splice on the left-hand side.

    ```{r}
    var <- "x"
    val <- 10
    
    lang("foo", var = val)
    lang("foo", !!var := val)
    ```

### Working around the absense of unquoting

`subset()`. `transform()`

`~` doesn't provide any way to unquote.

```{r}
make_model <- function(resp, preds) {
  pred_sum <- purrr::reduce(preds, function(x, y) expr(UQ(x) + UQ(y)))
  eval_tidy(expr(!!resp ~ !!pred_sum))
}
make_model(expr(y), exprs(a, b, c))
```

Note the use of `reduce()` to take a list of expressions and progressively add them together. This is a pleasant side effect of 

```{r}
binary_expr_reducer <- function(op) {
  op <- enexpr(op)
  
  function(x, y) {
    expr(UQ(op)(UQ(x), UQ(y)))
  }
}

x <- exprs(a, b, c, d)
purrr::reduce(x, binary_expr_reducer(`*`))
purrr::reduce_right(x, binary_expr_reducer(`*`))

purrr::reduce(x, binary_expr_reducer(f))
purrr::reduce_right(x, binary_expr_reducer(f))
```

How to use `expr()` + `eval_tidy()` to support wrap base functions.

```{r, error = TRUE, fig.keep = "none"}
library(lattice)
xyplot(mpg ~ disp, data = mtcars)

x <- quote(mpg)
y <- quote(disp)
xyplot(x ~ y, data = mtcars)
```

### Inlining and the deparser

If you construct ASTs by hand, it's possible to construct things that you could not construct by parsing code.  For example, if you forget to quote the first argument to `lang` it will literally inline the funtion call:

```{r}
lang(sum, quote(x))
```

It's also possible to inline objects that are not constants, symbols, or calls. This is useful in a handful of places (beyond the scope of the book, but typically useful in overscoping). The main thing to be aware of is that the the printed representation does not always accurately reflect the underlying tree. Trust `ast()` over what the console will print.

R will print parentheses that don't exist in the call tree:

```{r}
x1 <- lang("+", 1, lang("+", 2, 3))
x1
lobstr::ast(!!x1)
```

It will also display integer sequences as if they were generated with `:`.

```{r}
x2 <- lang("f", c(1L, 2L, 3L, 4L, 5L))
x2
lobstr::ast(!!x2)
```

If you inline more complex objects, their attributes are not printed which might lead to confusing output:

```{r}
x3 <- lang("class", data.frame(x = 10))
x3
eval(x3)

lobstr::ast(!!x3)
```

In general, if you're ever confused, remember to check the object with `ast()`!

### Exercises

1.  The following two calls look the same, but are actually different:

    ```{r}
    (a <- call("mean", 1:10))
    (b <- call("mean", quote(1:10)))
    identical(a, b)
    ```

    What's the difference? Which one should you prefer?

1.  Use `subs()` to convert the LHS to the RHS for each of the following pairs:
    * `a + b + c` -> `a * b * c`
    * `f(g(a, b), c)` -> `(a + b) * c`
    * `f(a < b, c, d)` -> `if (a < b) c else d`

2.  For each of the following pairs of expressions, describe why you can't
    use `subs()` to convert one to the other.
    * `a + b + c` -> `a + b * c`
    * `f(a, b)` -> `f(a, b, c)`
    * `f(a, b, c)` -> `f(a, b)`

1.  Concatenating a call and an expression with `c()` creates a list. Implement
    `concat()` so that the following code works to combine a call and
    an additional argument.

    ```{r, eval = FALSE}
    concat(quote(f), a = 1, b = quote(mean(a)))
    #> f(a = 1, b = mean(a))
    ```