Term frequency-inverse document frequency (tf-idf)

Compute, for each token and document, the term frequency, the inverse document frequency and their product: $$\mathrm{tf}_{td} = \frac{n_{td}}{\sum_u n_{ud}}, \qquad \mathrm{idf}_t = \log\frac{D}{\mathrm{df}_t}, \qquad \mathrm{tf\text{-}idf}_{td} = \mathrm{tf}_{td} \times \mathrm{idf}_t,$$ with $n_{td}$ the count of token $t$ in document $d$, $D$ the number of documents and $\mathrm{df}_t$ the number of documents containing $t$.

Usage

compute_tf_idf(
  tokens,
  token_col = "token",
  document_col = "document",
  count_col = "count",
  keep_cols = intersect("ngram", names(tokens)),
  ...
)

Arguments

tokens: A data.frame or data.table with one row per token and document (e.g. the output of extract_ngrams()) and its count in count_col, or one row per occurrence (count_col = NULL). Not modified.
token_col: Character scalar, token column.
document_col: Character vector, column(s) identifying a document (or a group of documents, such as a cluster).
count_col: Optional character scalar, the count of each row (the count of extract_ngrams()). NULL counts each row once. Missing counts are counted as 0, with a warning; negative counts are an error.
keep_cols: Character vector of other columns to keep, constant within a token and document (the first value is kept). By default ngram when present.
...: Not used: an argument renamed in the development version of mintR gives an error naming its replacement.

Value

A data.table with one row per token and document: token_col, document_col, keep_cols, count ($n_{td}$), document_total ($\sum_u n_{ud}$), corpus_count (total count of the token in the corpus), tf, df, idf and tf_idf. The count column lets the table, once filtered (e.g. by filter_group_quantile()), go directly to tokens_to_stm().

Examples

tokens <- data.frame(
  document = c("d1", "d1", "d2"),
  token = c("labour", "market", "labour"),
  count = c(2, 1, 3)
)
compute_tf_idf(tokens)
#>     token document count document_total corpus_count        tf    df       idf
#>    <char>   <char> <num>          <num>        <num>     <num> <int>     <num>
#> 1: labour       d1     2              3            5 0.6666667     2 0.0000000
#> 2: market       d1     1              3            1 0.3333333     1 0.6931472
#> 3: labour       d2     3              3            5 1.0000000     2 0.0000000
#>       tf_idf
#>        <num>
#> 1: 0.0000000
#> 2: 0.2310491
#> 3: 0.0000000