Extract n-grams from texts and count them by document

Tokenise texts into n-grams (unigrams, bigrams, ...), drop n-grams that contain a stop word, a digit, punctuation or a symbol, and count the remaining ones by document (or any grouping). Stop words can be tolerated inside the longer n-grams ("rate_of_growth"), see inner_stop_words_from.

Usage

extract_ngrams(
  data,
  ngrams = 2L,
  document_col = NULL,
  text_col = "text",
  min_nchar = 2L,
  stop_words = get_stop_words("en"),
  inner_stop_words_from = Inf,
  inner_stop_words = NULL,
  language_col = NULL,
  compounds = NULL,
  chunk_size = NULL,
  check_duplicates = TRUE,
  ...
)

Arguments

data: A data.frame or data.table with the texts. It is not modified.
ngrams: Integer. A scalar n computes all sizes 1:n; a vector (e.g. c(1L, 3L)) computes only those sizes.
document_col: Character vector of the column(s) identifying the documents (or any group to count by, e.g. c("institution", "year")). NULL counts over the whole corpus.
text_col: Character scalar, name of the text column.
min_nchar: Integer scalar, minimal number of characters of each word of an n-gram (inner stop words excepted). 3L removes the one- and two-letter tokens of formulas and OCR ("kt", "mt") in academic texts, but also abbreviations such as "uk" or "eu".
stop_words: Stop words: a character vector, used for all texts (default: English, get_stop_words()), or a named list of vectors by language (get_stop_words(c("en", "fr"))) used with language_col. NULL (or character(0)) uses no stop word list: only n-grams with digits, punctuation or symbols are dropped.
inner_stop_words_from: Integer scalar. From this n-gram size onwards, stop words are allowed inside an n-gram, but not as its first or last word: with 3L, "rate_of_growth" is kept but "of_growth_in" and "the_rate_of" are not. The default Inf drops every n-gram containing a stop word, and says so in a message (once per session) when n-grams of size 3 or more are asked for.
inner_stop_words: Optional character vector. With inner_stop_words_from, only these stop words are allowed inside n-grams (e.g. "of"), instead of every stop word.
language_col: Optional character scalar, a column of language codes (e.g. from detect_language()). Each text is then filtered with the stop words of its language (stop_words must be a named list); texts whose language has no list use all the lists together.
compounds: Optional n-grams to merge into single tokens before counting: the output of select_collocations(), a table with token (words joined by _) and optionally score, or a character vector (words joined by _ or by spaces). Each text is rewritten with its words in order: where a selected n-gram occurs, its words are replaced by one token ("monetary_policy"), so they are no longer counted on their own there; the same words elsewhere are still counted. When two selected n-grams overlap, the higher score wins (the longer one without score). With compounds, ngrams must be 1L: the output has the single words and the compounds, with their size in ngram.
chunk_size: Optional integer, number of rows tokenised at once. Use it on large corpora to bound memory. NULL processes all rows at once.
check_duplicates: Logical: warn when texts are duplicates of one another (see find_duplicate_texts()).
...: Not used: an argument renamed in the development version of mintR gives an error naming its replacement.

Value

A data.table with document_col, token (words joined by _), ngram (its size) and count (number of occurrences in the document).

Details

Texts are lower-cased and split into words once by tokenizers::tokenize_words(). Curly apostrophes are made straight and a final possessive 's is dropped ("bank's" and "bank’s" become "bank"); other apostrophes stay inside the word ("don't"). An n-gram is dropped when any of its words is a stop word, is shorter than min_nchar, or contains a character other than a letter or an apostrophe; n-grams do not cross texts. The test is made once per distinct word (per language), and only the n-grams kept are built, so the cost grows little with n. Clean links, handles or OCR hyphenation first with clean_texts().

Stop words in longer n-grams

Stop words are useful to drop from unigrams and bigrams ("of_the"), but they are part of many multiword terms: "rate_of_growth", "balance_of_payments". Justeson and Katz (1995) found, in samples of four technical dictionaries, that 97% of multiword terms contain only nouns and adjectives, and more than 99% only nouns, adjectives and the preposition "of"; terms with a preposition were about 3% of the total. They therefore leave prepositions out by default because candidates that contain them are mostly not terms (5 or 6 valid terms among 58 candidates with prepositions in their three test texts). Allowing inner stop words is thus a trade-off: more coverage of terms like "rate_of_growth", more noise from phrases like "growth_in_output". Restricting inner_stop_words to "of" follows their finding, and filtering the n-grams by a minimal count or by tf-idf (see filter_group_quantile()) limits the noise. Constant et al. (2017) survey the wider field of multiword expression discovery, where association measures such as pointwise mutual information are the usual tool for bigrams but are hard to extend to longer n-grams.

Languages

The stop words are English by default. When they are, texts of at least 50 words in which English stop words make up less than 10% of the words (against 40-50% in English prose) are reported with a warning: they are probably in another language, and their stop words are kept and will form topics of their own. Detect the languages with detect_language(), then filter the corpus or use language_col with get_stop_words().

References

Justeson, J. S. and Katz, S. M. (1995). Technical terminology: some linguistic properties and an algorithm for identification in text. Natural Language Engineering, 1(1), 9-27. doi:10.1017/S1351324900000048

Constant, M., Eryigit, G., Monti, J., van der Plas, L., Ramisch, C., Rosner, M. and Todirascu, A. (2017). Multiword expression processing: a survey. Computational Linguistics, 43(4), 837-892. doi:10.1162/COLI_a_00302

Examples

df <- data.frame(id = 1:2, text = c("Economic growth matters", "Growth and the bank's trade"))
extract_ngrams(df, ngrams = 2L, document_col = "id")
#>       id           token ngram count
#>    <int>          <char> <int> <int>
#> 1:     1        economic     1     1
#> 2:     1          growth     1     1
#> 3:     1         matters     1     1
#> 4:     2          growth     1     1
#> 5:     2            bank     1     1
#> 6:     2           trade     1     1
#> 7:     1 economic_growth     2     1
#> 8:     1  growth_matters     2     1
#> 9:     2      bank_trade     2     1