Package index

Connections

Open databases safely: one writer, read-only readers that fall back on a snapshot, memory capped.

resolve_db_path(): Resolve the path of a database
db_connect(): Open a DuckDB database
db_connect_read_only(): Open a database read-only, or its snapshot when a script is writing
db_disconnect(): Close a DuckDB connection
db_snapshot_path() db_refresh_snapshot() db_remove_snapshot(): Snapshot of a database for readers

Keep memory under control on databases of millions of texts: check the memory available, check keys and draw samples in SQL, stream large results.

Define the tables once, with their documentation, and check the database and the README against them.

create_schema(): Create the tables of a schema, and add the columns added since
add_missing_columns(): Add missing columns to a table
schema_dictionary(): Data dictionary of a schema
schema_markdown(): Markdown tables of a data dictionary
check_schema(): Compare a database with its schema
check_documented_columns(): Columns of the schema missing from the documentation

Transactions that replace what they rewrite, a log of every write, and raw records kept as the source returned them.

check_unique(): Stop if a table has duplicated keys
replace_rows(): Replace the rows of a table, in one transaction
replace_documents(): Replace documents and their rows in several tables, in one transaction
append_raw_records(): Append raw records to a raw table

db_fingerprint(): Fingerprint of the tables of a database
compare_fingerprints(): Compare two fingerprints of a database
export_parquet() import_parquet(): Export a database to Parquet files, and import it back

write_stamp() read_stamp() check_stamp(): Version stamps of derived outputs
content_hash(): Fingerprint of objects or files

write_gz_text() read_gz_text(): Write and read gzipped text files
dated_file_name() file_date() latest_dated_file(): Dated file names
backup_files() compare_with_backup(): Back up files before overwriting them, and compare afterwards
free_disk_gb() check_free_disk(): Free disk space
pipeline_log() pipeline_status(): Pipeline log
is_test_mode() output_path() input_file(): Test mode of a project
cap_items(): Cap the number of items processed
run_in_background() background_jobs() stop_background_job(): Run a script in a detached background process

make_doc_id() split_doc_id(): Document identifiers
hash_id(): Stable identifier from a string
pad_id(): Padded numeric identifiers
make_slug(): Slugs for names
normalise_for_matching(): Normalise titles and names for matching
escape_regex(): Escape regular expression characters
split_author_string(): Split author strings into one name per row
na_if_sentinel() normalise_sentinels(): Missing values written as text
collapse_or_na() json_list_to_string(): Collapse values into one string, or NA

Measure uncertain steps: sample sizes, stratified samples, blind judge files, agreement between judges, precision and recall with intervals.

margin_of_error() sample_size_for_margin(): Sample size and margin of error
draw_stratified_sample(): Draw a seeded, stratified sample
draw_stratified_sample_db(): Draw a stratified sample inside the database
write_judge_chunks(): Write the files of a blind judging round
read_verdicts(): Read the verdicts of a judging round
cohen_kappa(): Cohen's kappa between two labellings
wilson_interval(): Wilson confidence interval of a proportion
precision_recall(): Precision, recall and F1 with confidence intervals

coRpusdb coRpusdb-package: coRpusdb: build, document and evaluate text databases