Run a script in a detached background process — run_in

Long steps (first collections, PDF extraction, OCR) take hours. A job started as a child of an R session or of a terminal dies with it. run_in_background() starts a script in a separate Rscript process, detached with setsid nohup on Linux, so that it survives the closing of the IDE or terminal. The job's parameters are passed as R options, and it writes its output to a log file and its process id to a pid file.

background_jobs() lists the jobs started from a log folder and whether they still run;
stop_background_job() stops a job by its process id.

Usage

run_in_background(
  script,
  options = list(),
  name = tools::file_path_sans_ext(basename(script)),
  log_dir = getOption("coRpusdb.log_dir"),
  wd = getOption("coRpusdb.project_dir", getwd())
)

background_jobs(log_dir = getOption("coRpusdb.log_dir"))

stop_background_job(pid)

Arguments

script: Character. Path of the script, relative to wd or absolute.
options: Named list of R options set before the script runs (e.g. list(cb.update_steps = "boe")).
name: Character. Name of the job, used in the file names.
log_dir: Character. Folder of the log and pid files (e.g. ~/data/<project>/logs). By default the option coRpusdb.log_dir, which a project sets once in its paths_and_packages.R.
wd: Character. Working directory of the job (the project root). By default the option coRpusdb.project_dir, or the current directory.
pid: Integer. Process id of the job (column pid of background_jobs()).

Value

run_in_background(): the path of the log file, invisibly. background_jobs(): a data.table with job, pid, running, script and log. stop_background_job(): TRUE if the signal was sent, invisibly.

Details

Rules that avoid losing hours of work:

never edit a script while Rscript runs it (R reads it as it goes);
stop a job by its pid, never with pkill -f <pattern>: the pattern also matches the command line that runs pkill;
one heavy job at a time (LLM, OCR, large DuckDB aggregations): memory is finite;
a writing job locks the database: readers use db_connect_read_only(), which falls back on the snapshot.

Examples

if (FALSE) { # \dontrun{
run_in_background(
  "scripts/update_database.R",
  options = list(cb.update_steps = "boe"),
  log_dir = "~/data/central_bank_database/logs"
)
# Or once in the project's configuration:
options(coRpusdb.log_dir = "~/data/central_bank_database/logs")
run_in_background("scripts/update_database.R")
background_jobs()
} # }