Skip to contents

Select collocations with a rule

Source: R/collocations.R
select_collocations.Rd

Apply one of two rules to the table of score_collocations():

  • "npmi": keep the n-grams whose NPMI is at least min_npmi. NPMI does not depend on corpus size, so the threshold can be kept across corpora. It favours rare, fixed expressions, so min_count matters.

  • "top_n": keep the n n-grams with the highest rank_by (t-score by default), the rule of Lau, Baldwin and Newman (2013), who found about 1,000 bigrams to be the best trade-off for topic models.

Both rules can drop n-grams that occur mostly inside a longer candidate (max_share_nested, after the C-value of Frantzi et al. 2000). The result carries a score column (NPMI, or the ranking measure): when two selected n-grams overlap in a text, extract_ngrams() keeps the higher-scoring one.

Usage

select_collocations(
  scores,
  rule = c("npmi", "top_n"),
  min_npmi = 0.5,
  n = 1000L,
  rank_by = c("t_score", "count", "npmi", "pmi", "c_value"),
  min_count = 5L,
  max_share_nested = NULL,
  ...
)

Arguments

scores

Output of score_collocations().

rule

"npmi" (threshold) or "top_n" (ranking).

min_npmi

Numeric scalar in [-1, 1], threshold of the "npmi" rule. The default 0.5 is a starting point, not a value from the literature: read the n-grams around it and adjust.

n

Integer scalar, number of n-grams kept by the "top_n" rule (all sizes together).

rank_by

Column ranking the "top_n" rule: "t_score" (default), "count", "npmi", "pmi" or "c_value".

min_count

Integer scalar, minimal count, applied with both rules.

max_share_nested

Optional numeric scalar in [0, 1]: drop n-grams whose share of occurrences inside longer candidates exceeds it (e.g. 0.5).

...

Not used: an argument renamed in the development version of mintR gives an error naming its replacement, any other one an error.

Value

The rows of scores that pass the rule, with score, by decreasing score. Give it to extract_ngrams() as compounds.

Examples

df <- data.frame(id = 1:4, text = c(
  "monetary policy and fiscal policy", "the monetary policy rule",
  "monetary policy matters", "a policy of growth"
))
scores <- score_collocations(df, min_count = 1)
select_collocations(scores, rule = "npmi", min_npmi = 0.3, min_count = 2)
#>              token ngram count n_documents      pmi     npmi  t_score
#>             <char> <int> <int>       <int>    <num>    <num>    <num>
#> 1: monetary_policy     2     3           3 1.163151 0.694843 1.190785
#>    count_nested share_nested c_value    score
#>           <num>        <num>   <num>    <num>
#> 1:            2    0.6666667       2 0.694843
select_collocations(scores, rule = "top_n", n = 2, min_count = 1)
#>                   token ngram count n_documents      pmi      npmi  t_score
#>                  <char> <int> <int>       <int>    <num>     <num>    <num>
#> 1:      monetary_policy     2     3           3 1.163151 0.6948430 1.190785
#> 2: monetary_policy_rule     3     1           1 1.673976 0.6037594 0.812500
#>    count_nested share_nested  c_value    score
#>           <num>        <num>    <num>    <num>
#> 1:            2    0.6666667 2.000000 1.190785
#> 2:            0    0.0000000 1.584963 0.812500