covid <- read_covid()2 Objects, vectors and functions
Type 2 + 2 at the console and R prints 4. Nothing of that calculation is kept, so nothing can be built on it. To work with 345 days of case counts, R has to store values under names, and you have to know what kind of value each name holds.
The second requirement is the one that causes trouble. R will let you put a word into a column of numbers without raising an error: it converts the numbers into text instead, and the mean of the column then comes back as NA. This chapter covers the vocabulary needed to diagnose that quickly.
The Singapore file is used throughout, so load it now. read_covid() is a small helper that ships with this book: it reads data/sg_covid_data.csv and hands back exactly what is in the file, with nothing cleaned and nothing repaired.
2.1 Assignment
Assignment gives a value a name. R’s assignment operator is a left-pointing arrow, made of a < and a -:
first_case <- as.Date("2020-01-23")
first_case
#> [1] "2020-01-23"Two things happened. R evaluated the right-hand side, and it bound the result to the name first_case. Assignment produces no output; to see the value, type the name on its own line, as above.
That date is the first day this file records a confirmed case. Day one of the file, 22 January, has confirmed equal to zero.
In RStudio, Alt and minus (Option and minus on a Mac) types <- for you. The operator comes up constantly, so learn the shortcut early.
2.1.1 Why an arrow and not an equals sign
R inherited <- from S, and S inherited it from APL, whose keyboards had a single key that produced a real ← character. When S moved to keyboards without that key, the arrow was spelled out as two characters and stuck.
= also works for assignment at the top level. This book does not use it, for one reason: inside a function call, = already means something else. In round(x, digits = 2) the digits = 2 tells round() which of its arguments you are filling, and leaves no object called digits behind once the call has finished. Reserving <- for “make an object” and = for “fill an argument” keeps the two distinguishable when you read a line of code.
2.1.2 Naming things
Names may contain letters, numbers, dots and underscores. They cannot start with a number, and they cannot contain spaces or hyphens. daily_cases is fine, daily cases is a syntax error, and daily-cases is read as daily minus cases.
R is case sensitive. Confirmed and confirmed are different objects, and mixing them up produces one of the commonest error messages in R:
covd
#> Error:
#> ! object 'covd' not foundWhen you see object 'x' not found, R has searched every environment on its search path and found no such name. The usual causes are a typo or a chunk that was never run.
This book uses snake_case: lower case, words joined by underscores. Tidyverse function and argument names follow the same convention, so a script written this way reads consistently from your objects through to the functions you call on them. Within that convention, a useful rule is nouns for data and verbs for functions: an object is named after what it is, so covid and daily_cases describe contents, while a function is named after what it does, so read_covid() and mean() describe actions. A reader who meets either kind of name in the middle of a script can then tell what to expect of it before finding where it was defined.
Some names are best avoided entirely. TRUE and FALSE are reserved and cannot be reassigned, but their abbreviations T and F are ordinary variables. A line such as T <- 0 early in a long script changes the meaning of every later T, so write TRUE and FALSE in full. A second group of names is best left alone because each is already a function: c, data, mean and df all have existing meanings. df in particular is the density function of the F distribution:
df(1, 2, 3)
#> [1] 0.2788548A data frame named df masks stats::df() on the search path. R can usually tell a function call from a variable and will resolve the reference correctly, but the name now has two meanings on the page, and code that looks df up as a value, such as passing it to another function, gets the data frame rather than the density. Choosing a noun that describes the contents avoids the collision entirely.
2.2 R has no scalars
R has no scalar type. What looks like a single number is a vector of length one, and much of R’s behaviour follows from that.
The vector is the basic unit. c() (for combine) joins values into a longer one:
Every column of the Singapore file is a vector of length 345. Pull one out with $:
Because the vector is the basic unit, arithmetic works on whole vectors at once. R does not need you to write a loop over 345 days.
head(confirmed * 2, 10)
#> [1] 0 2 6 6 8 10 14 14 20 26This is called vectorisation, and it accounts for much of the brevity of R code for data analysis. A column of a table is a vector, so one line of arithmetic applies to all 345 days and the code never has to mention how many days there are. Without it, the same operation requires an explicit loop and an index to manage.
The same idea gives a useful quantity. The confirmed column is cumulative, a running total. The number of new cases on a given day is that day’s total minus the previous day’s. Shift the vector along by one, put a zero on the front so the lengths still match, and subtract:
# new cases are today's running total minus yesterday's, because confirmed is cumulative
previous_day <- c(0, confirmed[-length(confirmed)])
new_cases <- confirmed - previous_day
head(new_cases, 10)
#> [1] 0 1 2 0 1 1 2 0 3 3
max(new_cases)
#> [1] 1426
covid$date[which.max(new_cases)]
#> [1] "2020-04-20"The peak is 1,426 new cases on 20 April 2020, during the dormitory outbreak. The comment above the chunk records why the subtraction is set up this way rather than what each line does; the code shows the what on its own, and a comment that merely repeats it is the first thing to go out of date as the code changes. The result can also be checked: the daily figures sum back to the final cumulative total.
The zero on the front is not an arbitrary filler. The file starts on 22 January with confirmed equal to zero, so 22 January did have zero new cases. Writing that day as missing instead would drop a genuine day from every count made afterwards. dplyr::lag() does the same shift in one call, and lag(confirmed, default = 0) is where that zero goes; the version above is the same idea with the machinery visible.
2.2.1 Subsetting
Square brackets pick elements out by position, counting from 1 rather than 0.
confirmed[1]
#> [1] 0
confirmed[1:5]
#> [1] 0 1 3 3 4
confirmed[length(confirmed)]
#> [1] 58599A negative index means everything except, which is what confirmed[-length(confirmed)] did above: all of it bar the last day.
You can also subset with a logical vector, keeping the elements marked TRUE:
covid$date[new_cases > 1000]
#> [1] "2020-04-20" "2020-04-21" "2020-04-22" "2020-04-23"Four days in 2020 had more than a thousand new cases, and they run consecutively from 20 April. dplyr::filter() expresses the same operation more readably on a table, and underneath it does what this line does: build a logical vector, then retain the elements it marked TRUE.
2.3 Vector types and missing values
Every vector has a type, and every element of a vector has the same type. Four types matter for data work:
The four types are logical, integer, double and character. (Two more exist, complex and raw, which are not needed for the material in this book.)
Integer and double are both numbers, and R makes the distinction awkwardly. A number you type is a double unless you stick an L on the end. A colon range is an integer.
identical() reports that they are different objects; == reports that they hold the same value. Both are correct, and the distinction rarely matters in practice. What does matter is that doubles are stored in binary floating point, which cannot represent most decimal fractions exactly:
0.1 + 0.2 == 0.3
#> [1] FALSE
0.1 + 0.2 - 0.3
#> [1] 5.551115e-17This is not specific to R. Any language using IEEE 754 doubles behaves the same way, including spreadsheet software. Testing two calculated numbers with == is therefore unreliable; use near(), which allows a small tolerance:
near(0.1 + 0.2, 0.3)
#> [1] TRUELogical values are TRUE and FALSE, and R treats them as 1 and 0 in arithmetic. A count of how many elements satisfy a condition is therefore sum() of the logical vector, and the proportion is mean() of it:
The count is 254 days of 2020 with more than ten thousand cumulative cases, and the stringency index was above 50 on 57% of days.
2.3.1 NA
NA is R’s marker for a value that should exist but is not recorded. It is not zero, not an empty string, and not a statement that the data are complete. Almost any operation involving an NA returns NA, because the result of that operation is unknown too.
The propagation is deliberate: if one of three numbers is unknown, their mean is unknown as well, so R returns NA rather than averaging the two values it has.
The deaths column in this file makes it concrete. Singapore recorded no COVID-19 deaths for the first two months of the epidemic, and this file records those days as missing rather than as zero.
There are 59 missing days, and so the mean of the column is NA. To get an answer you have to say explicitly that you are willing to ignore the gaps:
mean(covid$deaths, na.rm = TRUE)
#> [1] 23.75874Whether that is the right thing to do is a question about the deaths data, not about R. na.rm = TRUE drops the 59 unrecorded days and averages the 286 that remain, so the number returned is the mean of the days the source reported, which is not the mean of 2020. Which of the two you want depends on the question being asked, and the default behaviour at least requires the choice to be made explicitly.
Two more properties of NA come up in daily work. Testing it with == does not work, because “an unknown value” is not equal to anything, including another unknown value:
NA == NA
#> [1] NAUse is.na() instead:
NA also comes in typed forms. Bare NA is logical; there are also NA_integer_, NA_real_ and NA_character_, so that a missing value can sit inside a vector without changing its type. You rarely type these yourself, but you will see them in error messages.
2.4 Coercion, or how a column of numbers becomes text
A vector holds one type, so combining two types forces R to choose one of them.
c(1, "a")
#> [1] "1" "a"R issued no error or warning; it converted the number 1 into the string "1", because that is the only way to hold both values in one vector. The rule runs one way only, from least to most flexible: logical values can be represented as integers, integers as doubles, and doubles as character strings, so a combination takes the type furthest along that sequence.
Nothing here is a bug, and no warning is issued at any step, which is why the conversion is easy to miss. Consider a spreadsheet of daily deaths in which one cell reads "not reported" instead of being left blank. That single cell turns the entire column to character on import, and then:
The warning is the useful part: argument is not numeric or logical: returning NA. When you see it, you have a character column that you thought was numeric. typeof() on the column confirms it in one line.
Going the other way is explicit, and R does warn:
as.numeric(c("1", "a"))
#> Warning: NAs introduced by coercion
#> [1] 1 NANAs introduced by coercion means R turned a value into NA because it could not parse it. Sometimes that is what you intended; sometimes it has removed a large part of the column. Count the missing values afterwards with sum(is.na(x)) to find out which.
2.5 Recycling
When two vectors in an arithmetic expression have different lengths, R repeats the shorter one until it is long enough. You have already relied on this: in confirmed * 2, the 2 is a vector of length one and it got recycled 345 times.
That is the useful case. Recycling also applies when neither vector has length one, which is where it causes trouble.
Predict before you run
c(1, 2, 3, 4) + c(10, 20) produces a vector of length four. Write down the four numbers you expect, and whether you expect a warning. Then run it.
No warning was issued here: four is a multiple of two, so R recycled the shorter vector twice and reported nothing. Change the length to three and a warning appears:
longer object length is not a multiple of shorter object length reports that the two vectors do not line up. The mechanism is the same in both cases; the warning appears only when the lengths are not multiples, not when the recycling is unintended. Checking length() on vectors before combining them is the reliable way to catch this.
2.6 Calling functions
A function has a name, takes arguments in brackets, and gives back a value. mean(new_cases) takes one argument and returns one number. Functions nest, and R works from the inside out:
A mean of 169.85 new cases a day against a median of 40 indicates a strongly right-skewed distribution, which matches the epidemic: most of 2020 had low daily counts, and a few weeks in April and May had very high ones.
Arguments can be given by position or by name. These three calls are identical:
Position only works if you remember the order. Names work regardless of order, and they record in the call itself which argument is being set. Pass the first argument or two positionally, since mean(new_cases) is unambiguous, and name everything after that.
Most arguments have defaults, so you only supply the ones you want to change. args() shows you what they are:
round(x, digits = 0) means that if you do not say otherwise, you get whole numbers. mean(x, ...) is less helpful, because mean is generic and the real arguments live in the method; this is one of the places where R’s help page tells you more than the code does.
round() has one behaviour to watch for:
round(0.5) is 0 and round(1.5) is 2. R rounds halves to the nearest even number rather than upward, following the IEC 60559 standard, so that rounding a long column of values does not accumulate an upward bias. ?round documents this. The behaviour is intended, and it can still surprise someone checking totals by hand.
2.6.1 Reading a help page
Every function in R ships with documentation. A question mark opens it:
?mean
help("mean")If you do not know the name, two question marks searches the text of every installed help page:
??"standard deviation"Help pages have a fixed structure, and reading them from top to bottom is rarely the quickest route. A more efficient order begins with the Usage section, which shows the function signature with every default filled in, so you can see what may be passed. From there, read only the two or three entries in Arguments that are relevant to the call you are writing. The Examples at the bottom of the page are runnable, and example(mean) runs them without opening the page; they are often the fastest way to see the function used correctly. The Details section matters most when something has already gone wrong, since that is where behaviour such as the rounding rule above is described, and the Value section settles what type the function returns, which is more often useful than it first appears.
Tidyverse functions carry the same documentation, typeset more legibly at https://tidyverse.org. Base R help pages are frequently terse, and much of that documentation was written for specialists rather than beginners. Searching for the function name plus “r” will usually find explanations in more ordinary language.
2.7 Packages and where functions live
R ships with a few thousand functions. Everything else lives in packages: bundles of functions, documentation and sometimes data, written by other people and published on CRAN.
Installing is a one-off, per machine. It downloads and compiles code onto your hard disk.
install.packages("tidyverse")Loading is per session. It takes an installed package and makes its functions available by name.
The distinction matters in practice. install.packages() belongs at the console, typed by hand, and not in a script; running it on every render costs minutes and requires an internet connection. library() belongs at the top of every script, because a script that does not load the packages it uses will fail on any machine where those packages are not already attached. The quotation marks are required by one and not the other, which is an inconsistency in R rather than a mistake on your part.
2.7.1 The masking message
library(tidyverse) prints a block of text when it loads. Read it.
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.2.1 ✔ readr 2.2.0
✔ forcats 1.0.1 ✔ stringr 1.6.0
✔ ggplot2 4.0.3 ✔ tibble 3.3.1
✔ lubridate 1.9.5 ✔ tidyr 1.3.2
✔ purrr 1.2.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errorsLoading dplyr on its own reports the same thing in more detail:
Attaching package: ‘dplyr’
The following objects are masked from ‘package:stats’:
filter, lag
The following objects are masked from ‘package:base’:
intersect, setdiff, setequal, unionNothing has broken, and the mechanism behind the message is the search path: R keeps an ordered list of the places it looks for a name.
head(search(), 15)
#> [1] ".GlobalEnv" "package:scales" "package:patchwork"
#> [4] "package:here" "package:lubridate" "package:forcats"
#> [7] "package:stringr" "package:dplyr" "package:purrr"
#> [10] "package:readr" "package:tidyr" "package:tibble"
#> [13] "package:ggplot2" "package:tidyverse" "package:stats"When you type filter(...), R walks that list from left to right and uses the first filter it finds. Every package you attach is inserted near the front, so a newly loaded package wins against anything loaded before it. package:dplyr sits ahead of package:stats, so filter means dplyr’s.
stats::filter has not been deleted, and you can still reach it by giving its full address. The double colon means “the function called filter from the package called stats, regardless of the search path”:
That is a seven-day moving average of the cumulative curve, which is the job stats::filter() does. It returns a time series object, hence the header lines about start, end and frequency, and the first three values are NA because a seven-day window cannot be centred on day one. None of this has anything to do with selecting rows from a table:
dplyr::filter(covid, stringency_index > 80)$date
#> [1] "2020-04-10" "2020-04-25" "2020-05-01"Two functions share a name and do unrelated jobs. The conflicts message exists so that the collision is visible when the package is attached, rather than later when a result looks wrong.
The same applies to lag(). dplyr::lag() shifts a vector along by one position, which is exactly what you did by hand earlier. stats::lag() shifts a time series’ index and leaves the values alone. If you ever compute daily cases and get a column of zeroes, this is why.
This is also why library() goes at the top of a script. If dplyr has not been loaded and you call filter() on a data frame, the call reaches stats::filter, and the resulting error does not point at the missing library() line.
Error: object 'confirmed' not foundstats::filter() has no idea that confirmed is a column of the thing you handed it, so it tries to find an object by that name and fails. The fix is a library(dplyr) or library(tidyverse) line you left out, and the error message does not say so.
Writing dplyr::filter() in full removes the ambiguity, at the cost of a package prefix on every call. It earns that cost in code you distribute, where the reader may have a different set of packages attached. For everyday analysis, load the package and read the conflicts message; if you would prefer collisions to raise an error instead of being resolved by precedence, the conflicted package does that.
2.8 Exercise
Exercise 2.1 (Build a week by hand) The first seven cumulative confirmed counts in the file are 0, 1, 3, 3, 4, 5 and 7.
- Store them in a vector called
week_oneand confirm that R agrees it has seven elements and holds doubles. - Work out how many new cases were recorded on each of days two to seven, using only subsetting and subtraction.
- Add the label
"week one"to the front ofweek_one, look at the type again, and then try to multiply the result by two. Explain in one sentence what R did to your numbers and why the arithmetic now fails. - Make a copy of
week_onewith the fourth element replaced byNA, then find its mean twice, once as R gives it to you and once tellingmean()to drop the missing value.
Stuck? Open for a hint (not the answer)
For part 2, two vectors of the same length subtract element by element, so you need one vector holding days two to seven and another holding days one to six. For part 3, look at typeof() before and after.