Changelog

Source: NEWS.md

coRpusdb 0.2.0

Memory and speed on large databases, measured with bench/benchmark_memory.R (each case in its own process, peak resident memory of R and DuckDB; results in bench/results_memory.csv), and the options that projects need.

• New free_memory_gb() and check_free_memory(); db_connect() warns when DuckDB’s memory limit exceeds the memory available, and reads the number of threads from the option coRpusdb.threads.
• New db_check_unique(): key uniqueness checked in SQL, without loading the table.
• New db_query_chunks(): processes a result chunk by chunk, streamed as Arrow record batches when arrow is installed (2 million texts: 5.6 s and 2.7 GB, against 10.5 s and 3.6 GB with DBI::dbFetch() chunks).
• New draw_stratified_sample_db(): stratified sample drawn by DuckDB, reproducible on any machine (MD5 ordering); on 10 million rows, 0.8 s and 140 MB against 6.6 s and 1.3 GB when loading the population into R.
• draw_stratified_sample() no longer copies the population: 5 times faster on 5 million rows, same samples for the same seed.
• normalise_for_matching() decodes entities and transliterates only the strings that need it: 1 million titles in 5.8 s instead of 37 s, half the memory, same results.
• hash_id() uses a vectorised digest (3.7 times faster).
• replace_rows(), replace_documents() and append_raw_records() insert by column name from a registered data frame instead of copying and aligning it first; they also accept data.tables (a data.table input failed before).
• run_in_background() and background_jobs() read their folders from the options coRpusdb.log_dir and coRpusdb.project_dir, and take their arguments in the order script, options, name; pipeline_log() and pipeline_status() read the log file from coRpusdb.pipeline_log and accept the status "warning".
• db_query() no longer copies the result.

coRpusdb 0.1.0

First version, gathering and unifying functions copied across the author’s database projects (building_jstor_database, building_elsevier_database, building_istex_database, central_bank_database, build_eu_database, eea_history), tested on their databases.

• Connections: db_connect() caps DuckDB’s memory; db_connect_read_only() falls back on the snapshot kept by db_refresh_snapshot() when a writer holds the lock; resolve_db_path() points a run to a copy.
• Queries: db_query() binds parameters, registers id tables, and opens and closes a read-only connection when given a path.
• Schema: schema_dictionary() turns the comments of CREATE TABLE statements into a data dictionary; check_schema() and check_documented_columns() compare it with the database and the README.
• Writing: replace_rows() and replace_documents() delete and insert in one transaction and log rows written and replaced; append_raw_records() keeps raw records as the source returned them.
• db_fingerprint(), compare_fingerprints(), export_parquet() and import_parquet() compare and share copies of a database.
• write_stamp(), check_stamp() and content_hash() record which method version produced a derived output and refuse stale ones.
• Test mode (output_path(), input_file(), cap_items()), dated files, backups compared afterwards, disk checks, a pipeline log and detached background jobs.
• normalise_for_matching() unifies the title normalisations of the projects, keeps non-Latin scripts and decodes HTML entities (compared on 50,000 JSTOR titles).
• Evaluation: draw_stratified_sample(), margin_of_error(), sample_size_for_margin(), write_judge_chunks(), read_verdicts(), cohen_kappa(), wilson_interval(), precision_recall().