Skip to contents

Keep the rows above a quantile of a metric, within each group

Source: R/filter_group_quantile.R
filter_group_quantile.Rd

For each group, compute the probs quantile of metric and keep the rows above it. It is the filter of the pipeline "tokens, tf-idf, filter, back to tokens": the table of compute_tf_idf() (or of extract_ngrams()) is filtered, and the rows that remain are the tokens given to tokens_to_stm().

Usage

filter_group_quantile(
  data,
  metric = "tf_idf",
  probs = 0.75,
  by = NULL,
  inclusive = FALSE,
  keep_threshold = FALSE,
  verbose = TRUE,
  ...
)

Arguments

data

A data.frame or data.table, typically the output of compute_tf_idf(). Not modified.

metric

Character scalar, name of the numeric column to threshold. The columns available depend on the table:

  • from extract_ngrams(): count (count of the token in the document) and ngram (size of the token);

  • from compute_tf_idf(): tf_idf (the default: the specificity of the token to the document), tf (share of the token in the document), idf (rarity of the token across documents), df (number of documents containing the token), count, corpus_count (total count of the token) and document_total (total count of the document).

probs

Numeric scalar in [0, 1], passed to stats::quantile() (type 7, missing values ignored). 0.75 keeps the highest quarter.

by

Character vector of grouping columns: one threshold is computed per combination of their values. NULL (default) gives one threshold over all rows. See the details for useful choices.

inclusive

Logical scalar: keep rows equal to the threshold (>=) or not (>).

keep_threshold

Logical scalar: return the threshold column.

verbose

Logical scalar: report how many rows and distinct tokens are kept.

...

Not used: an argument renamed in the development version of mintR gives an error naming its replacement.

Value

A data.table with the kept rows, in their original order. Rows with a missing metric are dropped.

Details

by decides what the quantile is compared with. Tokens of different sizes do not have the same distribution of tf-idf (a bigram is rarer than a word and has a higher idf), so that one threshold for all of them would keep mostly bigrams: with by = "ngram", each size keeps its own best quarter. To keep the most specific tokens of each document (so that no document is emptied), use the document identifier, e.g. by = c("id", "ngram"). To compare documents of one period or one outlet only, use that column.

Examples

dt <- data.table::data.table(group = c("a", "a", "b"), tf = c(1, 3, 2))
filter_group_quantile(dt, metric = "tf", probs = 0.5, by = "group")
#> ℹ Kept 1 of 3 rows (33.3%) above the 0.5 quantile of tf.
#>     group    tf
#>    <char> <num>
#> 1:      a     3

# tf-idf of a few documents, thresholds by size of the n-gram
tokens <- extract_ngrams(
  data.frame(id = 1:3, text = c(
    "labour market reform", "labour market policy", "monetary policy rule"
  )),
  ngrams = 2L, document_col = "id"
)
tf_idf <- compute_tf_idf(tokens, document_col = "id")
filter_group_quantile(tf_idf, "tf_idf", probs = 0.5, by = "ngram")
#> ℹ Kept 3 of 15 rows (20%), 3 of 11 distinct tokens (27.3%) above the 0.5 quantile of tf_idf.
#>       token    id ngram count document_total corpus_count    tf    df      idf
#>      <char> <int> <int> <num>          <num>        <num> <num> <int>    <num>
#> 1:   reform     1     1     1              5            1   0.2     1 1.098612
#> 2: monetary     3     1     1              5            1   0.2     1 1.098612
#> 3:     rule     3     1     1              5            1   0.2     1 1.098612
#>       tf_idf
#>        <num>
#> 1: 0.2197225
#> 2: 0.2197225
#> 3: 0.2197225