Skip to contents

Normalise titles and names for matching

Source: R/normalise.R
normalise_for_matching.Rd

Erases the differences that sources introduce in the same title or name (capitalisation, accents, HTML markup and entities, & vs "and", punctuation, a leading "The"), so that records can then be matched by strict equality, which is fast and transparent. Real spelling differences are left to a fuzzy matching step.

Usage

normalise_for_matching(
  x,
  decode_html = TRUE,
  ampersand = TRUE,
  punctuation = c("space", "remove"),
  drop_words = c("the", "a")
)

Arguments

x

Character.

decode_html

Logical. Decode HTML entities (&amp;, &#x2019;) and remove HTML tags first: inline tags (<em>, <sub>) are deleted (CO<sub>2</sub> -> co2), line and block tags (<br/>, <p>) become a space.

ampersand

Logical. Replace & by " and " (R&D becomes "r and d").

punctuation

Character. "space" turns every character that is not a letter or a digit into a space ("don't" -> "don t"); "remove" deletes punctuation ("don't" -> "dont").

drop_words

Character or NULL. Words removed (by default the articles "the" and "a").

Value

A character vector of the same length. A string with nothing left after normalisation (empty, or made of punctuation only) becomes NA, so that empty keys never match each other.

Details

Characters are first put in Unicode compatibility form (NFKC): PDF ligatures are split (fi -> fi) and subscripts become digits (CO₂ -> co2). Letters and digits of every script are kept: accents are removed from Latin letters, but Greek, Cyrillic, Hebrew or Chinese titles keep their characters (a pattern limited to [a-z0-9] would turn them into empty keys).

The defaults follow the most careful of the versions used in the author's projects (eea_history), with two corrections found when comparing them on 50,000 JSTOR titles: HTML entities are decoded (&amp; became andamp), and line-break tags become a space instead of gluing two words (4<br/>Complexity became 4complexity). The others are reproduced, for Latin-script titles, with:

  • central_bank_database, econ_fulltext_database: normalise_for_matching(x, decode_html = FALSE, ampersand = FALSE, drop_words = NULL);

  • building_elsevier_database, building_istex_database: close to normalise_for_matching(x, punctuation = "remove", drop_words = NULL), except that they remove only a leading "the".

The same normalisation must be applied to both sides of a match, and its choice changes the matches: record it with the matching version.

See also

Other normalisation: collapse_or_na(), escape_regex(), na_if_sentinel(), split_author_string()

Examples

normalise_for_matching(c(
  "<em>The</em> Economics of Information: A Survey",
  "THE ECONOMICS OF INFORMATION &#x2013; A SURVEY",
  "Économie & Société"
))
#> [1] "economics of information survey" "economics of information survey"
#> [3] "economie and societe"