Skip to contents

Run a resumable queue of downloads

Source: R/download.R
queue_downloads.Rd

Downloads a list of files one at a time, and can be stopped and started again. Files already on disk are skipped, each attempt is written to a state file as it happens, and a later run tries again only what can still succeed. After too many failures in a row the queue stops, since insisting on a site that refuses or has fallen over is impolite and useless.

Usage

queue_downloads(
  files,
  state_file = NULL,
  check = is_pdf_file,
  check_failure = if (identical(check, is_pdf_file)) {
    
    "not a PDF (HTML or other content returned)"
 } else {
     "file failed the check"

    },
  pause = getOption("politecollect.delay", 2),
  max_attempts = 3L,
  max_consecutive_failures = 10L,
  overwrite = FALSE,
  ...
)

Arguments

files

Data frame with columns url and dest.

state_file

Character or NULL. CSV file of the attempts. Without it, a new run only knows what is on disk.

check

Function of a file path returning TRUE or FALSE, or NULL. By default is_pdf_file().

check_failure

Character. Reason recorded when the check fails; a reason that failure_kind() knows decides whether a later run retries.

pause

Numeric. Minimum seconds between two requests to one site.

max_attempts

Integer. Attempts per file, over all the runs.

max_consecutive_failures

Integer. Stop after this many transient or blocked failures in a row.

overwrite

Logical. Download again the files already on disk?

...

Other arguments of polite_request().

Value

A data.table, one row per file: url, dest, outcome ("downloaded", "exists", "failed", "skipped" for a failure that a retry would not solve, or "not_tried" if the queue stopped), reason and kind (failure_kind()).

Details

A file is not tried again if its last failure is permanent or blocked, or if it has already failed max_attempts times. A path disallowed by robots.txt is recorded as a failure and the queue goes on.

A queue can run for hours: start the script as a detached job with coRpusdb::run_in_background().

See also

Other downloads: download_file()

Examples

if (FALSE) { # \dontrun{
options(politecollect.user_agent = "my_project (me@example.org)")
files <- data.frame(url = pdf_urls, dest = file.path("pdf", basename(pdf_urls)))
result <- queue_downloads(files, state_file = "logs/pdf_queue.csv")
result[, .N, by = outcome]
} # }