covid <- read_covid()3 Rectangles: data frames and tibbles
A data frame holds several variables of different types in one object, with the same number of rows in each. That constraint is what allows a dataset to be filtered, sorted and summarised without keeping track by hand of which value belongs with which. This chapter covers what a data frame is made of, what a tibble changes, and what tidy data means.
3.1 Three vectors and a problem
Take three days of the Singapore record, in the week the circuit breaker began, and store each variable in its own vector.
The values themselves are all present, but the correspondence between the three vectors is not recorded anywhere: that the second element of each refers to the same day is an assumption you are holding, not something R knows or enforces.
Sorting one of them shows the consequence. Sort the stringency values from highest to lowest, which is a reasonable thing to want:
sort(stringency, decreasing = TRUE)
#> [1] 76.85 62.04 45.37
day
#> [1] "2020-04-06" "2020-04-07" "2020-04-08"stringency has been reordered and day has not, so if that sorted vector is assigned back over the old one, the 7 April stringency score now sits alongside 6 April, and nothing in the session records that the two vectors have come apart. There is no error and no warning, and a plot drawn from them still looks plausible.
With 345 rows and 13 variables, maintaining that correspondence by hand is not workable, which is what data frames are for.
3.2 A data frame is a list of columns
data.frame() binds vectors together so that they can only be manipulated as a unit.
three_days <- data.frame(day, confirmed, stringency)
three_days
#> day confirmed stringency
#> 1 2020-04-06 1375 45.37
#> 2 2020-04-07 1481 62.04
#> 3 2020-04-08 1623 76.85Now sorting means sorting rows, and the rows carry their contents with them:
three_days[order(three_days$stringency, decreasing = TRUE), ]
#> day confirmed stringency
#> 3 2020-04-08 1623 76.85
#> 2 2020-04-07 1481 62.04
#> 1 2020-04-06 1375 45.37The row numbers on the left have gone 3, 2, 1. Each row moved as a piece.
Underneath, a data frame is a list of vectors, which becomes visible if you strip the class off it:
unclass(three_days)
#> $day
#> [1] "2020-04-06" "2020-04-07" "2020-04-08"
#>
#> $confirmed
#> [1] 1375 1481 1623
#>
#> $stringency
#> [1] 45.37 62.04 76.85
#>
#> attr(,"row.names")
#> [1] 1 2 3The output is a named list of three vectors plus an attribute recording the row names, and that is all a data frame contains. Everything else, the printing, nrow(), the two-dimensional subsetting, is behaviour attached to that list by its class. R will confirm the underlying structure if asked directly:
length() on a data frame counts the elements of the underlying list, which is the number of columns. lengths(), with the s, gives the length of each element, which is the number of rows repeated once per column. nrow() and ncol() are friendlier names for the same two facts.
Two properties follow directly from the list-of-vectors implementation.
Columns may be of different types. A list can hold anything, so one element can be a Date vector, the next a double vector, the next a character vector. Each column is internally uniform, which is the vector rule from Section 2.3, but columns need not agree with each other. That is what makes a data frame a suitable shape for a dataset, where dates, counts and labels sit side by side. A matrix cannot do this; every cell of a matrix has one type.
sapply(three_days, class)
#> day confirmed stringency
#> "Date" "numeric" "numeric"Rows must line up. The list is not free to hold vectors of any length, because the rectangle would stop being a rectangle. Give data.frame() mismatched lengths and it refuses:
data.frame(day = 1:3, phase = c("pre", "circuit breaker"))
#> Error in `data.frame()`:
#> ! arguments imply differing number of rows: 3, 2arguments imply differing number of rows: 3, 2 is a useful error: it arrives at the moment the mistake is made rather than several steps later.
Every operation on a rectangle, filtering rows, creating columns, grouping and summarising, relies on row i of every column referring to the same observation. filter() can drop 200 rows without any separate check that the variables stayed aligned, because misalignment is not representable.
3.2.1 Recycling in data.frame()
There is one exception to that constraint, and it is a long-standing one. If the short vector’s length divides the long one exactly, data.frame() recycles instead of raising an error:
data.frame(day = 1:6, phase = c("pre", "circuit breaker"))
#> day phase
#> 1 1 pre
#> 2 2 circuit breaker
#> 3 3 pre
#> 4 4 circuit breaker
#> 5 5 pre
#> 6 6 circuit breakerThe result is six rows with alternating labels, produced without a warning. This is the recycling rule from Section 2.5 applied to column construction. It is convenient when intended and silent when not, so a mistyped or mis-subsetted vector can produce a table that looks complete.
3.3 Tibbles
A tibble is a data frame with several of its older behaviours changed. read_csv() returns one, so this is what covid has been all along:
class(covid)
#> [1] "spec_tbl_df" "tbl_df" "tbl" "data.frame"Reading the class vector from the right, the object is a data.frame, so anything that accepts a data frame accepts it. tbl and tbl_df are the tibble classes, which change how it prints and how it is subsetted. spec_tbl_df is readr’s marker, carrying the column specification it worked out while reading the file: which column it took to be a date, which it took to be numbers, which it left as text.
The differences that matter in practice are the following.
3.3.1 Printing reports types and dimensions
covid
#> # A tibble: 345 × 13
#> administrative_area_level_1 confirmed deaths date stringency_index
#> <chr> <dbl> <dbl> <date> <dbl>
#> 1 Singapore 0 NA 2020-01-22 19.4
#> 2 Singapore 1 NA 2020-01-23 25
#> 3 Singapore 3 NA 2020-01-24 25
#> 4 Singapore 3 NA 2020-01-25 25
#> 5 Singapore 4 NA 2020-01-26 25
#> 6 Singapore 5 NA 2020-01-27 25
#> 7 Singapore 7 NA 2020-01-28 25
#> 8 Singapore 7 NA 2020-01-29 25
#> 9 Singapore 10 NA 2020-01-30 25
#> 10 Singapore 13 NA 2020-01-31 25
#> # ℹ 335 more rows
#> # ℹ 8 more variables: government_response_index <dbl>,
#> # containment_health_index <dbl>,
#> # retail_and_recreation_percent_change_from_baseline <dbl>,
#> # grocery_and_pharmacy_percent_change_from_baseline <dbl>,
#> # parks_percent_change_from_baseline <dbl>,
#> # transit_stations_percent_change_from_baseline <dbl>, …The printout shows ten rows, then a count of what was left out, then the names and types of the columns too wide to show. Read the <chr>, <dbl> and <date> labels under the column names: a column that is character when you assumed it was numeric is a frequent source of confusion, and the tibble reports the type before any analysis is written.
An old-style data frame does not do this. Print the same 345 rows as a data.frame and R sends all of them to the console without type labels, so the header scrolls off the screen. Do it once to see what it looks like, then use head():
head(as.data.frame(covid)[, 1:5])
#> administrative_area_level_1 confirmed deaths date stringency_index
#> 1 Singapore 0 NA 2020-01-22 19.44
#> 2 Singapore 1 NA 2020-01-23 25.00
#> 3 Singapore 3 NA 2020-01-24 25.00
#> 4 Singapore 3 NA 2020-01-25 25.00
#> 5 Singapore 4 NA 2020-01-26 25.00
#> 6 Singapore 5 NA 2020-01-27 25.003.3.2 No accidental factors
Before R 4.0.0, released in April 2020, data.frame() and read.csv() converted every character column into a factor by default, because the stringsAsFactors argument defaulted to TRUE. A factor stores its values as integer codes against a table of levels, which is appropriate for a categorical variable in a model and not for a column of free text or identifiers.
The behaviour is still available on request, and it has one consequence that is easy to overlook:
phase <- data.frame(label = c("pre", "circuit breaker"), stringsAsFactors = TRUE)
phase$label
#> [1] pre circuit breaker
#> Levels: circuit breaker preLevels: circuit breaker pre is alphabetical, so “circuit breaker” sorts before “pre” and any plot will show them in that order. R 4.0.0 changed the default to FALSE, one of the few backwards-incompatible changes the language has made. Tibbles never performed the conversion.
Factors are useful when requested deliberately. A severity label that should read low, medium, high rather than alphabetically is a case for factor(), because you name the levels in the order you want and everything downstream uses that order. The difference is that the conversion is then explicit in the code.
3.3.3 Subsetting stays a data frame
Single-bracket subsetting of one column behaves differently in the two classes.
Predict before you run
old below is the same data as covid, converted to an old-style data frame. Both of the next two expressions ask for one column with a single pair of square brackets. Decide, before running them, whether each returns a rectangle of 345 rows and 1 column, or a bare vector of 345 numbers.
old <- as.data.frame(covid)
class(old[, "confirmed"])
#> [1] "numeric"
class(covid[, "confirmed"])
#> [1] "tbl_df" "tbl" "data.frame"The data frame dropped a dimension and returned a vector, whereas the tibble returned a one-column table.
covid[, "confirmed"]
#> # A tibble: 345 × 1
#> confirmed
#> <dbl>
#> 1 0
#> 2 1
#> 3 3
#> 4 3
#> 5 4
#> 6 5
#> 7 7
#> 8 7
#> 9 10
#> 10 13
#> # ℹ 335 more rowsThe old behaviour comes from [.data.frame having a drop argument that defaults to TRUE: ask for a single column and R returns its contents rather than a one-column table. That is convenient interactively and a problem inside a pipeline, because a function two steps downstream that expected a data frame receives a numeric vector and fails with a message about something else entirely. Tibbles set drop = FALSE, so [ always returns a tibble.
When you do want the bare vector, say so with $ or double brackets, both of which mean “give me the contents of this element” and work identically on both kinds of rectangle:
3.3.4 No silent partial matching
$ on an old-style data frame will complete a partial column name for you:
head(old$conf, 5)
#> [1] 0 1 3 3 4There is no column called conf. R found confirmed, took it as the intended match, and returned it without comment. If the file also had a confirmed_cases_dormitory column the match would be ambiguous and $ would return NULL, so the same line would behave differently on a file that differs only in having an extra column.
Tibbles warn:
head(covid$conf, 5)
#> Warning: Unknown or uninitialised column: `conf`.
#> NULLThe result is a warning saying the column is unknown or uninitialised, followed by NULL. The same warning appears for an ordinary typo, so covid$confrimed is reported rather than silently returning NULL into a calculation.
3.3.5 Choosing between them
Use tibbles for anything you write. When an older package rejects one, and a few statistical packages still do, convert at the point of the call:
class(as.data.frame(covid))
#> [1] "data.frame"as_tibble() converts back. The two are interconvertible, so the choice is local to the call rather than a commitment for the whole project.
3.4 Tidy data
Tidy data is a rule about what goes into which part of the rectangle, and it has three clauses:
- Each variable is a column.
- Each observation is a row.
- Each value is a cell.
The three clauses are easy to state, but real spreadsheets violate them routinely: twelve columns named January to December, a column holding both a measurement and its unit, or two rows per patient because before and after were recorded side by side.
Tidyverse functions assume tidy input. Once the data are in that shape, filter(), mutate(), group_by() and ggplot() all work without further reshaping.
Applying the definition to this file means first establishing what one observation is:
The output shows 345 rows, 345 distinct dates and one country, so the unit of observation is a country-day; since the country never varies, in practice it is a day. Each column is a variable measured on that day: a cumulative case count, a cumulative death count, a policy index, six mobility series. There is one row per observation, one column per variable and one value per cell, so this file arrives tidy, which is convenient for the book; data you are handed elsewhere often will not be.
3.4.1 A column that says nothing
administrative_area_level_1 deserves a second look:
table(covid$administrative_area_level_1)
#>
#> Singapore
#> 345The column has one distinct value across all 345 rows. A column that never varies cannot explain, predict, group or distinguish anything. It is present because this extract was taken from a multi-country archive in which the column did vary, and selecting a single country left the label behind.
Counting distinct values in every column is a habit to build on any new dataset, because both ends of the result tell you something. A column with one distinct value carries no information. A column with as many distinct values as there are rows is an identifier rather than a measurement, which is what date is here. Anything in between is a variable you can actually work with.
sapply(covid, function(column) length(unique(column)))
#> administrative_area_level_1
#> 1
#> confirmed
#> 333
#> deaths
#> 27
#> date
#> 345
#> stringency_index
#> 13
#> government_response_index
#> 23
#> containment_health_index
#> 21
#> retail_and_recreation_percent_change_from_baseline
#> 50
#> grocery_and_pharmacy_percent_change_from_baseline
#> 53
#> parks_percent_change_from_baseline
#> 76
#> transit_stations_percent_change_from_baseline
#> 61
#> workplaces_percent_change_from_baseline
#> 74
#> residential_percent_change_from_baseline
#> 47The function(column) argument is an anonymous function, defined at the point of use rather than given a name.
stringency_index takes 13 distinct values over the year, which reflects how it is constructed: it is a score assembled from a handful of policy indicators, so it changes only when one of those indicators changes. The series therefore moves in steps, and each step marks a discrete policy change.
3.4.2 When the file is not tidy
Untidy files are common. The usual shapes are a header row of dates or months, one column containing two variables combined, and observations split across several rows. tidyr provides pivot_longer() and pivot_wider() for reshaping between wide and long layouts. This book does not devote a chapter to reshaping, because the Singapore file does not require it, and the operations are easier to follow once the material here is familiar.
3.5 Exercise
Exercise 3.1 (Take a tibble apart) Using covid as loaded at the top of this chapter:
- Show, with code rather than assertion, that
covidis a list, how many elements that list has, and that every element has the same length. Say in one sentence which of those two numbers is the number of rows and which is the number of columns. - Find out the type of every column without printing the whole tibble.
- Build a small tibble by hand recording three dates in Singapore’s 2020 policy timeline, 7 April, 2 June and 19 June, alongside a short label for each. Give the columns snake_case names that say what each column holds. Then explain in two sentences why holding those as a tibble is safer than holding the dates and the labels as two separate vectors.
- Convert
covidto an old-style data frame and find one behaviour, other than printing, that differs from the tibble. Show it.
Stuck? Open for a hint (not the answer)
For part 1, three functions from this chapter do the whole job: one asks whether it is a list, one counts the elements, one measures each element. For part 2, sapply() applies a function to every column and collects the answers into one named vector.