Package index

Politeness

The door every request goes through: identification, robots.txt, pace per site, back-off, and the cap of test runs.

polite_request()

Send one polite request

check_robots() crawl_delay() clear_robots_cache()

Check that robots.txt allows a URL

last_failure()

Why the last request failed

request_count() reset_request_count()

Count and cap the requests of a session

Requests

Get a page in the format wanted, know why a request failed, keep a log.

polite_get()

GET a URL politely and parse it

failure_kind()

Classify a failed request: is a retry worth it?

normalise_url()

Normalise a URL for comparison

read_request_log() request_log_summary()

Summarise the log of requests

Downloads

Files that are checked before they count, and queues that can stop and resume.

download_file() download_pdf() is_pdf_file()

Download a file and check it

queue_downloads()

Run a resumable queue of downloads

HTML to text

html_text_or_na()

Text of the elements matched by a CSS selector

html_paragraphs()

Paragraphs of an HTML page, with the heading they fall under

read_html_blocks()

Text blocks of a saved HTML file

prefer_html_text()

Should the HTML text of a page be preferred to its PDF?

Generic API tools

Build a rate-limited request, walk through pages by cursor, query a SPARQL endpoint.

api_request()

Prepare a rate-limited API request

api_paginate()

Walk through the pages of an API

sparql_query() sparql_values()

Run a SPARQL query

Scholarly APIs

crossref_request() crossref_mailto() crossref_client_limits() crossref_doi_path() crossref_search()

Query the Crossref API

scopus_api_key() scopus_request() scopus_get_page() scopus_search() scopus_quota()

Query the Scopus API

openalex_api_key() openalex_request() openalex_search() openalex_rate_limit()

Query the OpenAlex API

ia_search() ia_metadata() ia_files() ia_is_restricted() ia_download()

Search and download from the Internet Archive

Package

politecollect politecollect-package

politecollect: polite and traceable collection of web and API sources