Missing values written as text

Sources write missing values in many ways ("", "NULL", "[NULL]", "<NA>", "NA", a single space). na_if_sentinel() turns them into NA in a vector; normalise_sentinels() does it in the character columns of a data frame, without modifying the input.

Usage

na_if_sentinel(x, sentinels = c("NULL", "[NULL]", "<NA>", "NA", "null"))

normalise_sentinels(
  data,
  cols = NULL,
  sentinels = c("NULL", "[NULL]", "<NA>", "NA", "null")
)

Arguments

x

Character.

sentinels

Character. Values meaning "missing". Strings made only of white space are always treated as missing.

data

A data frame.

cols

Character or NULL. Columns to normalise; NULL for all character columns.

Value

na_if_sentinel(): a character vector. normalise_sentinels(): a data.table (a copy).

Details

Some project copies also treated "0" as missing. It is not a default here: a zero is a value in many columns (a page, a count). Add it to sentinels for the columns where the source uses it for "missing".

Distinguishing what the source does not say from what was not collected matters for the analysis (Lemercier and Zalc, 2019, ch. 3, recommend to record missing information as information): normalise sentinels only when they all mean the same thing.

See also

Other normalisation: collapse_or_na(), escape_regex(), normalise_for_matching(), split_author_string()

Examples

na_if_sentinel(c("Economics", "NULL", "  ", "[NULL]", "0"))
#> [1] "Economics" NA          NA          NA          "0"        
normalise_sentinels(data.frame(a = c("x", "<NA>"), b = c("", "y"), n = 1:2))
#>         a      b     n
#>    <char> <char> <int>
#> 1:      x   <NA>     1
#> 2:   <NA>      y     2