GET a URL politely and parse it

Requests a page through polite_request() and returns its content parsed. A definitive failure gives NULL with a warning, so that one missing page does not stop a long run; last_failure() then says why.

Usage

polite_get(
  url,
  as = c("html", "json", "text", "file"),
  dest = NULL,
  pause = getOption("politecollect.delay", 2),
  reject_titles = "Request Rejected",
  ...
)

Arguments

url

Character. The URL.

as

What to return: "html" (an xml2 document), "json" (a parsed list), "text", or "file" (the body is saved in dest, whose path is returned).

dest

Character. Destination when as = "file".

pause

Numeric. Minimum seconds between two requests to this site.

reject_titles

Character. Page titles that mean the server refused the request while answering 200, such as a firewall page ("Request Rejected", seen on EU websites). Only for as = "html".

...

Other arguments of polite_request().

Value

The parsed content, or NULL for a failure.

Details

Besides HTTP errors, three answers count as failures, with these reasons:

• "response is a PDF": a link expected to be a page leads to the file (doi.org redirects). The caller can download it with download_pdf();

• "empty response";

• "rejected page": the title of the page is in reject_titles.

A disallowed path is an error, not a failure: see check_robots().

See also

Other requests: failure_kind(), normalise_url(), read_request_log()

Examples

if (FALSE) { # \dontrun{
options(politecollect.user_agent = "my_project (me@example.org)")
page <- polite_get("https://www.bis.org/speeches/index.htm")
if (is.null(page)) last_failure()
} # }