Skip to contents

Find duplicated and near-duplicated texts

Source: R/duplicates.R
find_duplicate_texts.Rd

Duplicates distort a topic model: the words of a text posted 600 times (a bot's "poll results", a press release reproduced in several outlets, a speech given twice) weigh 600 times, and these copies crowd the most representative documents of their topic. Run this before building the vocabulary, and keep one text per group (keep == TRUE).

  • "exact": texts identical after normalisation (lower case, links, digits and punctuation removed, spaces squeezed), so that a bot's posts that only differ by a number or a link are grouped.

  • "near": texts sharing most of their word sequences (shingles of shingle_size words), with a Jaccard similarity of at least threshold. Candidates are found by MinHash with locality-sensitive hashing (Broder 1997; Leskovec, Rajaraman and Ullman 2014, ch. 3), then their exact Jaccard similarity is computed. Exact duplicates are found too.

Usage

find_duplicate_texts(
  data,
  document_col = "document",
  text_col = "text",
  method = c("exact", "near"),
  threshold = 0.8,
  shingle_size = 5L,
  min_words = 3L,
  ...
)

Arguments

data

A table with document_col and text_col.

document_col, text_col

Column names of the document identifier and the text.

method

"exact" or "near".

threshold

Numeric scalar in (0, 1], minimal Jaccard similarity of the shingles of two near-duplicates.

shingle_size

Integer scalar, number of words of a shingle.

min_words

Integer scalar: texts with fewer words (after normalisation) are not compared (a "thanks!" is not a duplicate worth reporting); their group is NA.

...

Not used: an argument renamed in the development version of mintR gives an error naming its replacement, any other one an error.

Value

A data.table with one row per text of data: document_col, group (identifier of the group of duplicates, NA for a unique text), group_size, similarity (to the first text of the group: 1 for exact duplicates) and keep (TRUE for the first text of each group and for unique texts). Filter data with data[document_col %in% dup[keep == TRUE, document_col]].

References

Broder, A. Z. (1997). On the resemblance and containment of documents. Proceedings of Compression and Complexity of Sequences 1997, 21-29. doi:10.1109/SEQUEN.1997.666900

Leskovec, J., Rajaraman, A. and Ullman, J. D. (2014). Mining of Massive Datasets (2nd ed.). Cambridge University Press.

Examples

posts <- data.frame(
  id = 1:4,
  text = c("Poll results: 12 votes https://t.co/a", "Poll results: 31 votes https://t.co/b",
           "A new release of the app is out", "Something else entirely")
)
find_duplicate_texts(posts, "id", min_words = 2)
#> ℹ 1 of 4 texts duplicate another one (1 group); keep the rows with `keep == TRUE`.
#>       id group similarity group_size   keep
#>    <int> <int>      <num>      <int> <lgcl>
#> 1:     1     1          1          2   TRUE
#> 2:     2     1          1          2  FALSE
#> 3:     3    NA         NA          1   TRUE
#> 4:     4    NA         NA          1   TRUE