Skip to contents

Append raw records to a raw table

Source: R/write.R
append_raw_records.Rd

Appends records as they came from the source (scraped metadata, API responses) to a raw table, created if needed, with the time of collection. Raw tables are only appended to, never corrected: cleaning happens in derived tables, so that every decision can be revisited from the raw material. When the source adds a field, the column is added to the table rather than stopping the collection.

Usage

append_raw_records(con, data, table, collected_at = Sys.time())

Arguments

con

A DBI connection with write access.

data

A data frame of raw records. List columns are collapsed into strings (elements separated by " ; ").

table

Character. Name of the raw table (e.g. "ecb_metadata_raw").

collected_at

A date-time recorded in the column collected_at.

Value

The number of rows appended, invisibly.

Details

Columns are stored as text (VARCHAR), except dates and date-times: raw tables keep the source's wording ("input" in the vocabulary of Lemercier and Zalc), and the typing is a later decision taken in the derived tables ("categorisation").

See also

Other writing: check_unique(), replace_documents(), replace_rows()

Examples

con <- db_connect(tempfile(fileext = ".duckdb"))
#> duckdb is storing downloaded extensions and secrets under ~/.duckdb:
#> ℹ /home/agoutsmedt/.duckdb
#> This persists across sessions and is shared with the DuckDB CLI and other clients.
#> ℹ Run duckdb(shared_home = FALSE) to use a temporary directory instead.
#> ℹ See ?duckdb_storage for details and alternatives.
append_raw_records(con, data.frame(id = "1", title = "A"), "source_raw")
append_raw_records(con, data.frame(id = "2", title = "B", lang = "en"), "source_raw")
#> ℹ source_raw: added column lang.
DBI::dbGetQuery(con, "SELECT id, title, lang FROM source_raw")
#>   id title lang
#> 1  1     A <NA>
#> 2  2     B   en
db_disconnect(con)