Skip to contents

Scholarly APIs

Source: vignettes/articles/scholarly-apis.qmd

APIs are the preferred way into a source. They are also governed by their own rules, which change: the limits below were read from the documentation of each service in November 2026, and should be checked again before a large harvest. None of the code on this page runs when the site is built.

library(politecollect)
options(politecollect.user_agent = "my_project (https://example.org; me@example.org)")

Keys belong in ~/.Renviron, never in a script. The clients send them as headers when the service allows it, so that they appear in neither URLs nor logs.

What the clients have in common

Each *_search() function walks through the pages of a query by cursor, and returns the same list:

Element Meaning
items The results of all pages
n_total The number announced by the source
n_fetched The number received
n_pages The pages fetched
status "complete", "incomplete" or "limited" (by max_pages)

Check status. A harvest that stops halfway and looks finished is the characteristic error of this kind of work. For long harvests, on_page receives each page as it arrives. Writing it to disk there makes the harvest safe from a failure and keeps large results out of memory:

res <- openalex_search(
  "works",
  filter = "primary_location.source.issn:0002-8282",
  on_page = function(results, page) {
    saveRDS(results, sprintf("raw/aer_%04d.rds", page))
  }
)

A new API needs only two functions: api_request() prepares a rate-limited request, and api_paginate() follows its cursor. robots.txt is not read for APIs, which are governed by their terms of use.

Crossref

Crossref asks for a contact address (CROSSREF_MAILTO) and announces its limits in the headers of each response, which crossref_client_limits() uses to set the pace.

res <- crossref_search(
  "journals/0002-8282/works",
  query = list(filter = "from-pub-date:2020-01,until-pub-date:2020-12"),
  mailto = crossref_mailto()
)

crossref_request() returns a prepared request, which can be sent in parallel with httr2::req_perform_parallel(). Requests sent this way do not go through polite_request(): the throttle still applies, but the log and the test cap do not.

Scopus

Elsevier limits a key by the week (20,000 requests for the Scopus Search API) and by the second (9) [as stated in its API key settings page, consulted in November 2026].

res <- scopus_search("SRCID(20628) AND PUBYEAR IS 2000")   # ELSEVIER_API_KEY
res$items

Elsevier also restricts the cursor by key and network (“Use of the cursor parameter is restricted”, observed in November 2026 outside the institutional network). By default scopus_search() tries it, and falls back on the start parameter when it is refused.

A 429 stating QUOTA_EXCEEDED stops the run at once, and says when the quota resets. So does a key refused with 401 or 403, usually because the computer is outside the institutional network. Waiting cannot cure either of these. Other 429 and 5xx responses are retried. scopus_quota() reads what remains from a response.

OpenAlex

OpenAlex works with a daily budget, measured in dollars and spent by each request. Without a key it allows 10 requests per second; a free key (OPENALEX_API_KEY) gives 30, and a budget ten times larger.

res <- openalex_search(
  "works",
  filter = "primary_location.source.issn:0002-8282,publication_year:1990",
  select = "id,doi,title,authorships"
)

Paging past the first page needs a key: without one, OpenAlex answers 401 to the second page (observed in November 2026). openalex_search() then keeps the first page, warns, and returns status = "incomplete".

A 429 can mean too many requests in a second, which is retried, or a spent budget, which stops the run. openalex_rate_limit() reads what remains.

For an entire entity, do not use the API: the monthly snapshot exists for this purpose, and its files can be given to queue_downloads().

Internet Archive

ia_search() uses the scraping API, meant for walking through long result sets. The Archive warns that such a walk may not be complete, as items can be added or deleted in the meantime.

books <- ia_search(
  "collection:toronto AND mediatype:texts AND date:[1800-01-01 TO 1850-12-31]",
  fields = c("identifier", "title", "creator", "date", "language")
)
meta <- ia_metadata(books$items$identifier[1])
ia_is_restricted(meta)
files <- ia_files(meta)
files[format == "DjVuTXT"]
ia_download(books$items$identifier[1], files[format == "DjVuTXT", name][1], "book.txt", metadata = meta)

ia_download() verifies the MD5 given in the metadata of the item. It refuses, with an error, the items reserved for lending (access-restricted-item) and private files: the package fetches only what the Archive serves to everyone. That an item can be downloaded does not settle whether its text can be reused, which depends on its copyright status.

SPARQL

sparql_query() posts a query and returns a table of character columns. sparql_values() builds a VALUES block. It is patient by default, for the sake of an endpoint that has outages of a quarter of an hour: up to 12 tries, over about 33 minutes, honouring Retry-After. Do not run several harvests of one endpoint at once.

sparql_query(paste(
  "SELECT ?work ?date WHERE {",
  sparql_values("work", work_uris),
  "?work <http://publications.europa.eu/ontology/cdm#work_date_document> ?date }"
))

Do not depend on one door

“APIs are easy to use and TOS-compliant but may vanish without warning” (Freelon 2018). Keep what a source returned, as it returned it (coRpusdb::append_raw_records()), and record the date of the harvest and the conditions of access: they are part of the description of the corpus.

References

Freelon, Deen. 2018. “Computational Research in the Post-API Age.” Political Communication 35 (4): 665–68. https://doi.org/10.1080/10584609.2018.1477506.