Changelog

Source: NEWS.md

politecollect 0.1.0

First version, extracted from the collection functions of the author’s projects (central_bank_database, build_eu_database, building_jstor_database, building_elsevier_database, eea_history) and compared with them on their real data (bench/validate_on_projects.R).

• polite_request(): the single door of every request (user agent, robots.txt, pace per site, back-off, log, cache), and last_failure(), which replaces the option cb.last_failure.
• polite_get(), download_file(), download_pdf(), is_pdf_file(), normalise_url(), failure_kind().
• New queue_downloads(): a resumable queue that retries only what can still succeed and stops when a site refuses the collection.
• New request log (read_request_log(), request_log_summary()), response cache and capped test mode (request_count()).
• HTML to text: html_text_or_na(), html_paragraphs(), read_html_blocks(), prefer_html_text().
• APIs: api_request(), api_paginate(), sparql_query(), and clients for Crossref, Scopus, OpenAlex and the Internet Archive.