The author’s text databases range from a few gigabytes (central bank documents) to more than fifty (JSTOR, about ten million pages of text). On a machine shared with other R sessions, a language model or an OCR run, memory is the first constraint: a process that exceeds the memory available is killed, and the hours it had run are lost. This article gives the rules the package follows and the measurements behind them.
How the functions were measured
bench/benchmark_memory.R runs every case in its own Rscript process under GNU time, which reports the peak resident memory of the whole process: R’s allocations and DuckDB’s, which R-level tools such as bench::mark() do not see. Each case is repeated three times and the median kept; a process that only loads the package (about 150 MB) serves as the baseline. The synthetic data are 10 million rows of 576-character texts (4 GB on disk) or 1 million rows of 600-character texts generated in R. Results: bench/results_memory.csv (16 cores, 30 GB of RAM, btrfs disk, DuckDB 2.0).
| Operation | Variant | Time | Peak memory |
|---|---|---|---|
| Write 1M rows of text |
dbAppendTable() on an aligned copy |
5.7 s | 1.5 GB |
Registered data frame, INSERT ... BY NAME (package) |
5.6 s | 1.4 GB | |
replace_rows() with keys (registration, delete, insert, one transaction) |
6.0 s | 1.5 GB | |
| Delete 100,000 keys among 10M rows |
DELETE ... WHERE EXISTS (package) |
0.08 s | 0.36 GB |
| Rebuild the table with an anti-join | 9.2 s | 6.2 GB | |
| Fetch 2M rows of text |
dbGetQuery() at once |
9.7 s | 3.6 GB |
dbFetch() by chunks of 100,000 |
10.5 s | 3.6 GB | |
Arrow record batches (db_query_chunks()) |
5.6 s | 2.7 GB | |
| Fingerprint 10M rows | 6 GB memory limit | 3.2 s | 4.4 GB |
| 2 GB memory limit | 2.2 s | 2.3 GB | |
| Export 10M rows to Parquet | 6 GB memory limit | 3.1 s | 5.5 GB |
| 2 GB memory limit | 4.1 s | 2.1 GB, failed once out of three | |
| Stratified sample of 10M rows | Load the population, sample in R | 6.6 s | 1.5 GB |
Sample in SQL (draw_stratified_sample_db()) |
0.8 s | 0.3 GB | |
| Normalise 1M titles | Version 0.1 | 37.4 s | 1.4 GB |
| Version 0.2 (only the strings that need it) | 5.8 s | 0.8 GB | |
| Hash 1M identifiers | One digest() call per string |
9.7 s | 0.6 GB |
Vectorised digest (hash_id()) |
2.9 s | 0.5 GB |
Rules that follow
Cap DuckDB’s memory. DuckDB’s default limit is 80% of the RAM. Under a limit, it spills intermediate results to disk instead: the fingerprint of 10 million rows ran as fast under 2 GB as under 6 GB. db_connect() sets 6 GB by default (option coRpusdb.memory_limit) and warns when the limit exceeds the memory available. Some operations need room: below about 4 GB, the export of a large text table can fail.
Know what cannot spill. Aggregates that build a string or a list per group (string_agg(), list()) keep each group in memory. In the central bank database, the md5 of the whole text of each document (md5(string_agg(text ORDER BY position)) over 1.5 million text rows) failed under the 6 GB limit and needed 11.4 GB. Raise the limit in the project for such a step, or hash the rows first: md5(string_agg(md5(text), '')) holds 32 characters per row instead of the text.
Filter and aggregate in SQL, and never load what you only count. Counts by year, uniqueness checks (db_check_unique()) and samples (draw_stratified_sample_db()) run inside DuckDB without bringing the rows into R.
Stream what must come to R. db_query_chunks() passes a result to a function chunk by chunk. Fetching chunks with DBI::dbFetch() saves nothing, because the duckdb driver materialises the whole result first; Arrow record batches are streamed by DuckDB, and were also twice as fast.
Do not copy large inputs. Writes register the data frame as a virtual table and insert it by column name; samples and checks copy only the columns and rows they need.
Process only what needs processing. Most titles are plain ASCII and carry no HTML: the normalisation now skips them for the expensive steps (transliteration, entity decoding), which made it six times faster.
Check the memory before a heavy step. check_free_memory() stops a script when another heavy job already uses the machine; check_free_disk() does the same for disk space, which DuckDB fills when it spills.
Fewer threads lower the peak only a little. With 4 threads instead of 16, the export used 4.8 GB instead of 5.5 GB and took 50% longer. Prefer a memory limit (option coRpusdb.threads exists for the cases where it helps).
Rerunning the benchmark
Rscript bench/benchmark_memory.R # every case (about 20 minutes)
Rscript bench/benchmark_memory.R normalise noop # some cases
BENCH_REPS=5 BENCH_N=2000000 Rscript bench/benchmark_memory.R write_package_innerThe 10-million-row database is created once in the user cache (tools::R_user_dir("coRpusdb", "cache")/bench, about 4 GB), never in /tmp, which is a RAM disk on many Linux systems: writing gigabytes there consumes memory.