Skip to contents

Filter the vocabulary and report what each rule removes

Source: R/vocabulary_diagnostics.R
filter_vocabulary.Rd

Keep the tokens that pass a set of rules, the usual ones of topic modelling: in enough documents to be estimated, in few enough not to be a background word, long enough, and not in a list of words to drop. A message reports what each rule removes, in types (distinct tokens) and in occurrences, and how many documents are left empty: a rule on rare words removes many types but few occurrences, a rule on frequent words the opposite.

Usage

filter_vocabulary(
  tokens,
  token_col = "token",
  document_col = "document",
  count_col = "count",
  min_documents = 1L,
  max_document_share = 1,
  min_count = 1L,
  min_nchar = 1L,
  remove = NULL,
  by = NULL,
  verbose = TRUE,
  ...
)

Arguments

tokens

A table with one row per token and document, e.g. from extract_ngrams().

token_col, document_col, count_col

Column names of the token, the document identifier (one or several columns) and the count of the token in the document (NULL: each row counts once).

min_documents

Integer scalar: keep tokens present in at least this number of documents.

max_document_share

Numeric scalar in (0, 1]: keep tokens present in at most this share of the documents (e.g. 0.5).

min_count

Integer scalar: keep tokens with at least this total count in the corpus.

min_nchar

Integer scalar: keep tokens of at least this number of characters (_ of n-grams included).

remove

Optional character vector of tokens to drop (e.g. words of conversation, boilerplate).

by

Optional column (e.g. "ngram"): the document rules are applied within each of its values, so that bigrams, rarer than words, are not all removed by the same min_documents.

verbose

Logical scalar: report what each rule removes.

...

Not used: an argument renamed in the development version of mintR gives an error naming its replacement.

Value

The kept rows of tokens (a data.table, in their original order). The report of the rules is in its attribute "report", a table with, for each rule, the types and occurrences it removes on its own (rules overlap, so they do not add up) and, in the row "all rules", what is removed in total.

See also

vocabulary_diagnostics() to compare several filtered tables.

Examples

tokens <- extract_ngrams(
  data.frame(id = 1:4, text = c(
    "labour market reform", "labour market policy",
    "monetary policy rule", "labour policy"
  )),
  ngrams = 1L, document_col = "id"
)
kept <- filter_vocabulary(tokens, document_col = "id", min_documents = 2)
#> ℹ Vocabulary: 3 of 6 types kept (72.7% of the occurrences).
#> • `min_documents` removes 3 types (50.0%) and 3 occurrences (27.3%)
attr(kept, "report")
#> Index: <rule>
#>                  rule removed_types removed_occurrences share_types
#>                <char>         <int>               <num>       <num>
#> 1:      min_documents             3                   3         0.5
#> 2: max_document_share             0                   0         0.0
#> 3:          min_count             0                   0         0.0
#> 4:          min_nchar             0                   0         0.0
#> 5:             remove             0                   0         0.0
#> 6:          all rules             3                   3         0.5
#>    share_occurrences
#>                <num>
#> 1:         0.2727273
#> 2:         0.0000000
#> 3:         0.0000000
#> 4:         0.0000000
#> 5:         0.0000000
#> 6:         0.2727273