Cut each text into smaller documents, e.g. to fit a topic model on comparable units when documents range from abstracts to books. Three ways to cut:
By paragraphs (
unit = "paragraphs", recommended when texts have paragraphs): consecutive paragraphs are grouped into chunks of aboutchunk_wordswords, without cutting a paragraph; a paragraph longer thanmax_paragraph_wordsis cut by words. A paragraph is a unit of meaning, so chunks follow the structure of the text. The paragraphs are the rows ofdatawhenparagraph_colis given (one row per paragraph, as in a database of paragraphs), or the lines of each text otherwise.By words (
unit = "words"): chunks ofchunk_wordswords, cut wherever the count falls.By number of chunks (
n_chunks, withunit = "words"): each text is cut inton_chunkspieces of (nearly) equal length, whatever its length. Chunks of a long book are then longer than those of an article.
By default a chunk can start or end in the middle of a sentence (by
words, and for the long paragraphs cut by words). With sentences = TRUE chunks are made of whole sentences: by words, sentences are
grouped to about chunk_words words; by paragraphs, only the long
paragraphs are cut, between sentences. Sentences are found with the ICU
rules and its English list of abbreviations (so "e.g." or "Fig." do not
end one); this is slower than counting words (about a second per million
words), and wrong for texts without punctuation or in other languages
with different abbreviations.
A last chunk shorter than min_last_words is merged into the previous
one. A text with more than max_chunks chunks is sampled: max_chunks
chunks are kept, spread evenly through the text, and the others are
discarded, with a warning giving the share of words lost.
Usage
chunk_texts(
data,
document_col = "document",
text_col = "text",
unit = c("paragraphs", "words"),
chunk_words = 400L,
paragraph_col = NULL,
max_paragraph_words = 3L * chunk_words,
sentences = FALSE,
n_chunks = NULL,
max_chunks = Inf,
min_last_words = chunk_words%/%4L,
keep_cols = NULL,
check_duplicates = TRUE,
verbose = TRUE,
...
)Arguments
- data
Table with
document_colandtext_col(one row per text, or one row per paragraph withparagraph_col).- document_col, text_col
Column names of the document identifier and the text.
- unit
"paragraphs"or"words", see Description.- chunk_words
Integer scalar, target number of words per chunk.
- paragraph_col
Optional column ordering the paragraphs within each document, when
datahas one row per paragraph (unit = "paragraphs").- max_paragraph_words
Integer scalar: paragraphs longer than this are cut by words (
unit = "paragraphs").- sentences
Logical: never cut a sentence (see Description). Not compatible with
n_chunks.- n_chunks
Optional integer scalar (
unit = "words"): each text is cut into this number of equal pieces (fewer for very short texts), andchunk_words,max_chunksandmin_last_wordsare not used.- max_chunks
Integer scalar, maximal number of chunks per text. The default
Infkeeps every chunk.- min_last_words
Integer scalar, minimal length of a last chunk (at most
chunk_words; default: a quarter ofchunk_words).- keep_cols
Other columns of
datato keep (their first value in each document), e.g. the year or the author.- check_duplicates
Logical: warn when texts are duplicates of one another (see
find_duplicate_texts()).- verbose
Logical: report the number and the length of the chunks.
- ...
Not used: an argument renamed in the development version of mintR gives an error naming its replacement.
Value
A data.table with document_col, chunk (position in the
text), chunk_id (<document>_<chunk>, the identifier to give to
extract_ngrams() and tokens_to_stm()), n_chunks_total (before
sampling), n_words, keep_cols and text. With unit = "words",
words are separated by single spaces (line breaks are lost); with
"paragraphs", paragraphs are separated by a line break. Empty and
missing texts give no row.
Examples
docs <- data.frame(
id = 1:2,
text = c(paste(rep("word", 25), collapse = " "), "first paragraph\nsecond paragraph")
)
chunk_texts(docs, "id", unit = "words", chunk_words = 10, min_last_words = 4)
#> ℹ Cut 2 texts into 4 chunks of 8 words (median; 90% between 4 and 10).
#> id chunk chunk_id n_chunks_total n_words
#> <int> <int> <char> <int> <int>
#> 1: 1 1 1_1 3 10
#> 2: 1 2 1_2 3 10
#> 3: 1 3 1_3 3 5
#> 4: 2 1 2_1 1 4
#> text
#> <char>
#> 1: word word word word word word word word word word
#> 2: word word word word word word word word word word
#> 3: word word word word word
#> 4: first paragraph second paragraph
chunk_texts(docs, "id", unit = "paragraphs", chunk_words = 2, min_last_words = 1)
#> ℹ Cut 2 texts into 14 chunks of 2 words (median; 90% between 2 and 2).
#> ℹ 1 paragraph longer than 6 words was cut by words.
#> id chunk chunk_id n_chunks_total n_words text
#> <int> <int> <char> <int> <int> <char>
#> 1: 1 1 1_1 12 3 word\nword word
#> 2: 1 2 1_2 12 2 word word
#> 3: 1 3 1_3 12 2 word word
#> 4: 1 4 1_4 12 2 word word
#> 5: 1 5 1_5 12 2 word word
#> 6: 1 6 1_6 12 2 word word
#> 7: 1 7 1_7 12 2 word word
#> 8: 1 8 1_8 12 2 word word
#> 9: 1 9 1_9 12 2 word word
#> 10: 1 10 1_10 12 2 word word
#> 11: 1 11 1_11 12 2 word word
#> 12: 1 12 1_12 12 2 word word
#> 13: 2 1 2_1 2 2 first paragraph
#> 14: 2 2 2_2 2 2 second paragraph