Clean texts before tokenising — clean

Remove or normalise what the tokeniser would turn into noise: links, e-mail addresses, social media handles and hashtags, HTML entities, curly apostrophes, possessives, and words cut by a hyphen at the end of a line (OCR). Each step is an argument and can be turned off. Without cleaning, a Bluesky corpus has "https", "bsky" and "social" among its most frequent tokens, and "central bank's" gives no "central_bank".

Usage

clean_texts(
  data,
  text_col = "text",
  urls = TRUE,
  emails = TRUE,
  handles = c("name", "remove", "keep"),
  hashtags = c("word", "remove", "keep"),
  html = TRUE,
  apostrophes = TRUE,
  possessives = FALSE,
  hyphenation = TRUE,
  numbers = FALSE,
  squish = TRUE,
  ...
)

Arguments

data: A character vector of texts, or a data.frame/data.table with the texts in text_col (not modified: a cleaned copy is returned).
text_col: Character scalar, the text column when data is a table.
urls: Logical: remove links (http://, https://, www.).
emails: Logical: remove e-mail addresses.
handles: How to treat handles (@name, @name.bsky.social): "name" keeps the name without @ (the people, apps and organisations cited are often part of a topic), "remove" drops them, "keep" leaves them as they are.
hashtags: How to treat hashtags: "word" keeps the word without #, "remove" drops them, "keep" leaves them.
html: Logical: decode the common HTML entities (&, <, >, ", ',  ) and remove HTML tags.
apostrophes: Logical: replace curly apostrophes and quotes (’, ‘, ʼ, “, ”) by straight ones, so that "don’t" matches the stop word "don't".
possessives: Logical: drop the possessive 's ("bank's" becomes "bank"). extract_ngrams() does it anyway; here it also reaches other tools.
hyphenation: Logical: rejoin words cut by a hyphen at a line end (OCR), see Details.
numbers: Logical: remove numbers (digits, with their decimal and thousands separators). Not needed for extract_ngrams(), which drops tokens with digits.
squish: Logical: collapse runs of spaces and tabs into one space and trim the texts. Line breaks are kept (they mark paragraphs, see chunk_texts()).
...: Not used: an argument renamed in the development version of mintR gives an error naming its replacement, any other one an error.

Value

The cleaned character vector, or a copy of data (a data.table) with the cleaned text_col.

Details

Hyphenation. Two forms are treated differently:

a hyphen at the end of a line ("eco-\nnomic"): the line break shows that the word was cut, and the parts are joined when the second one starts with a lower-case letter;
a hyphen followed by a space ("employ- ment"), which remains when the line breaks have been lost (e.g. pages of OCR joined into one line). It is ambiguous: "short- and long-run" or "pre- and post-war" are suspended hyphens, not cut words. The parts are joined only when the joined word ("employment") occurs elsewhere in data. On 1,200 JSTOR articles, about a fifth of the "word- word" sequences were joined ("employ- ment", "pay- ments"); the others were suspended hyphens.

Removing a link leaves a space, so that the words around it are not joined; squish then removes the extra spaces.

Examples

clean_texts(c(
  "The central bank’s new policy: https://t.co/x1 via @ecb.europa.eu #monetarypolicy",
  "Long-term employ- ment &amp; short- and long-run effects"
))
#> [1] "The central bank's new policy: via ecb.europa.eu monetarypolicy"
#> [2] "Long-term employ- ment & short- and long-run effects"