Pipeline log

• pipeline_log() appends one event (a step starting, progressing, finished or failed) to a tab-separated log file.

• pipeline_status() reads the log and returns the last event of each step.

A plain text file rather than a database table: it can be read while a script holds the DuckDB lock, from R, the shell (tail -f) or a spreadsheet.

Usage

pipeline_log(
  step,
  status = c("start", "progress", "done", "warning", "failed"),
  message = "",
  script = NA_character_,
  log_file = getOption("coRpusdb.pipeline_log")
)

pipeline_status(log_file = getOption("coRpusdb.pipeline_log"))

Arguments

step

Character. Short name of the step (e.g. "ocr").

status

Character. One of "start", "progress", "done", "warning" (finished with problems to look at), "failed".

message

Character. Free text (counts, pages done out of total...).

script

Character. Name of the calling script.

log_file

Character. Path of the log file (created if needed). By default the option coRpusdb.pipeline_log, which a project sets once in its paths_and_packages.R.

Value

pipeline_log(): the row written, invisibly. pipeline_status(): a data.table with the last event of each script and step.

Examples

log <- tempfile(fileext = ".tsv")
pipeline_log("download", "start", script = "collecting_ecb.R", log_file = log)
pipeline_log("download", "done", "120 PDFs", script = "collecting_ecb.R", log_file = log)
pipeline_status(log)
#>                   time           script     step status  message
#>                 <char>           <char>   <char> <char>   <char>
#> 1: 2026-11-08 17:50:32 collecting_ecb.R download   done 120 PDFs