Skip to contents

Clean the references parsed by anystyle

Source: R/clean_ref.R
clean_ref.Rd

[Experimental]

Usage

clean_ref(data)

Arguments

data

A tibble returned by find_ref_to_df() or parse_ref_to_df() with clean = FALSE.

Value

A tibble with one row per reference: the identifiers and full_ref, then

  • type: the CSL type guessed by anystyle (article-journal, book, chapter...), corrected to report for report series and speech for speeches;

  • author: a list-column of tibbles, one row per author (author_order, family, given, particle, suffix, literal for institutions, name); first_author (family name of the first author, or the institution, or the first editor of an edited book without author), n_authors, authors ("Family, Given; Family, Given") and et_al (TRUE when the list ends with "et al.");

  • year (integer), year_suffix (the letter of "1985a"), date (the date as printed);

  • title, container_title (journal, book, or report series), volume, issue, number (of a report: working paper number...), pages, first_page, last_page, publisher, location, edition, editors, doi, url, and every other field anystyle found (isbn, note, genre...), with - replaced by _ in its name. Fields that hold one value (journal, volume, issue, pages...) keep the first value anystyle found; the others keep all values, separated by " | ";

  • other_title, other_date: further titles and dates anystyle found in the reference (often a sign that it cut a title or merged two fields);

  • flag: the rules that changed something, and the problems met, separated by "; " (NA if none), to filter the references to check.

Details

Turns the raw output of anystyle (from find_ref_to_df() or parse_ref_to_df() with clean = FALSE) into one row per reference with flat columns, and corrects frequent parsing errors with rules checked against references whose fields are known (see Details).

The rules, and their flag:

  • Text fields are trimmed of the punctuation anystyle leaves around them (commas, colons, final full stops, unbalanced brackets) and of the quotation marks around titles; question and exclamation marks are kept.

  • Authors.

    • "et al." (others in anystyle) becomes et_al = TRUE.

    • Names written "Family I." without comma ("Brainard W.C., Tobin J.") are read by anystyle with the initials as family name: they are swapped (family_given_swapped).

    • Given names in capitals ("MARTIN, DONALD") are split into initials ("D.O.N.A.L.D.") by anystyle: they are read back from full_ref (given_restored).

    • Institutions taken for persons ("Bank of England" read as given name "Bank", family name "England") or for titles ("Committee on the Global Financial System (2012), ...") become a single literal name (institutional_author).

    • Ditto marks ("———", ".", "–") stand for the authors of the previous reference. anystyle resolves long dashes but not single characters: these are filled from the previous reference of the same document (ditto_filled; ditto_without_author when no previous reference has authors). " and K. Shimomura" gets the first previous author.

    • An edited book without author: the first editor stands for the author in first_author (editor_as_first_author).

  • Dates. year is the first plausible year (1500 to next year) of the dates found by anystyle. If there is none, the year in brackets in full_ref, or else the only year full_ref holds, is used (year_from_full_ref). anystyle drops the letter of "1985a": it is read back from full_ref. A title made only of a year is dropped.

  • Pages. Dashes are normalised, "pp." removed and the range is split into first_page and last_page; abbreviated last pages are expanded ("1145-64" gives 1145 and 1164, last_page_expanded).

  • Volume and issue. "Vol." and "No." are removed; anystyle's "ue 2" (from "Issue 2") is repaired; a volume glued to its issue ("47 2", "11, 2") is split (volume_issue_split); roman volumes ("LIV") are converted (volume_from_roman).

  • Grey literature. References in a report series (working, discussion, staff, occasional, policy... papers, in several languages) are typed report, with the series in container_title and its number in number (report_series). Speeches, remarks and testimonies are typed speech (speech).

  • DOI. Taken from the doi field, else from the URL, else from full_ref (doi_from_full_ref), without the resolver and the punctuation glued to it.

  • Not references. Lines without year, author, container, publisher, DOI or URL, or with the words of a disclaimer ("views expressed", "corresponding author"), are flagged not_a_reference (not removed). Several values for a one-value field are flagged several_<field>.

Each rule was written after reading the disagreements between anystyle and the fields of thousands of references indexed by Scopus, and of central bank documents, and kept only if it improved agreement without creating new errors. See the article How good is the parsing? for the measures.

Examples

if (FALSE) { # \dontrun{
raw <- parse_ref_to_df(
  paste(
    "MARTIN, DONALD. \"The Economics of Jury Conscription,\"",
    "J. Polit. Econ., July/Aug. 1972, 80(4), pp. 680-702."
  ),
  clean = FALSE
)
clean_ref(raw)
} # }