coRpusdb: build, document and evaluate text databases

Shared building blocks for the research projects that collect texts into a DuckDB database (harvesting, extraction, cleaning, matching) and for the projects that read these databases. The functions encode a few principles drawn from the practice of building such databases and from the literature on corpora and quantitative sources (see the article Building a text database as a research practice):

• Raw is kept, derived is versioned. Raw records are appended, never corrected (append_raw_records()); derived outputs carry a stamp of the method and version that produced them (write_stamp(), check_stamp()).

• Writes are transactions that replace what they rewrite (replace_rows(), replace_documents()): a failed run leaves the database as it was, and a rerun does not duplicate rows.

• One writer, many readers. A writing script locks a DuckDB file; readers fall back on a snapshot (db_connect_read_only(), db_refresh_snapshot()).

• The schema documents itself (schema_dictionary(), check_schema()) and copies of a database can be compared (db_fingerprint()).

• Test runs never touch the real outputs (output_path(), cap_items()).

• Uncertain steps are measured (draw_stratified_sample(), write_judge_chunks(), cohen_kappa(), precision_recall()).

Options

• coRpusdb.memory_limit: DuckDB memory limit set by db_connect() (default "6GB").

• coRpusdb.snapshot: set to FALSE to disable db_refresh_snapshot().

• coRpusdb.min_free_gb: default threshold of check_free_disk().

• coRpusdb.threads: number of DuckDB threads set by db_connect().

• coRpusdb.log_dir, coRpusdb.project_dir: folders of run_in_background() and background_jobs().

• coRpusdb.pipeline_log: log file of pipeline_log().

• <prefix>.test_mode, <prefix>.max_docs: project switches read by is_test_mode(), output_path(), input_file() and cap_items().

See also

Useful links:

• https://agoutsmedt.codeberg.page/corpusdb/

• https://codeberg.org/agoutsmedt/coRpusdb

• Report bugs at https://codeberg.org/agoutsmedt/coRpusdb/issues

Author

Maintainer: Aurélien Goutsmedt agoutsmedt@hotmail.fr

Authors:

• Aurélien Goutsmedt agoutsmedt@hotmail.fr