What each filtering step removes from the vocabulary

Filtering the vocabulary (stop words, rare words, a tf-idf threshold...) changes both the types (distinct tokens) and the occurrences the topic model sees. Give the token table after each step, in order, and this returns what remains at each step and what each step removed. Types and occurrences go in different directions: a filter on rare words removes many types but few occurrences, while a tf-idf threshold usually removes many occurrences.

Usage

vocabulary_diagnostics(
  steps,
  token_col = "token",
  count_col = "count",
  document_col = NULL,
  by = NULL,
  ...
)

Arguments

steps

A named list of token tables, in the order of the steps, the first one being the starting point (e.g. list(all = tokens, frequent = tokens[token %in% vocabulary], specific = filtered)). Each table needs token_col, count_col (unless NULL) and, if given, document_col.

token_col

Character scalar, token column.

count_col

Character scalar, number of occurrences of each row (the count of extract_ngrams() and compute_tf_idf()). It must be in every table. NULL counts each row once.

document_col

Optional character vector, column(s) identifying documents, to also count the documents left with at least one token.

by

Optional character scalar, a column to break the figures down by (e.g. "ngram"). It must be in every table.

...

Not used: an argument renamed in the development version of mintR gives an error naming its replacement.

Value

A data.table with one row per step (and per value of by): step, n_types, n_occurrences, n_documents (if document_col), the shares share_types and share_occurrences of the first step that remain, and removed_types, removed_occurrences, what the step removed relative to the previous one.

Examples

tokens <- extract_ngrams(
  data.frame(id = 1:3, text = c(
    "labour market reform", "labour market policy", "monetary policy rule"
  )),
  ngrams = 2L, document_col = "id"
)
frequent <- tokens[token %in% tokens[, .N, by = token][N >= 2, token]]
vocabulary_diagnostics(list(all = tokens, frequent = frequent), document_col = "id")
#>        step n_types n_occurrences n_documents share_types share_occurrences
#>      <fctr>   <int>         <num>       <int>       <num>             <num>
#> 1:      all      11            15           3   1.0000000         1.0000000
#> 2: frequent       4             8           3   0.3636364         0.5333333
#>    removed_types removed_occurrences
#>            <int>               <num>
#> 1:            NA                  NA
#> 2:             7                   7