Support CSL test cases that use the CITATIONS section

[YDX-2147483647]

The CSL test suite defines two sections for testing citations: CITATION-ITEMS and CITATIONS. Only one of them may be used in a single test case¹.

Previously, only CITATION-ITEMS was supported, and all test cases that use CITATIONS were skipped.
I think it's necessary to support CITATIONS.
For example, at present, Typst does not work well with the CSL variable first-reference-note-number (#500). However, all test cases relevant to first-reference-note-number use CITATIONS, so they are all skipped.

This PR adds support for the CITATIONS sections by normalizing them into CITATION-ITEMS.
(In most cases, CITATIONS is just a verbose alternative representation of CITATION-ITEMS.)
42 test cases are affected: 16 skipped → passed, 13 skipped → failed, and 13 are still skipped. Details are shown at the end.
Only tests in the CSL test-suite are normalized; local tests in this repo do not require normalization.

This PR is ready for review, but I guess this won't be the final form.
Drodt is working on restructuring the repo in #489, so it's likely that merge conflicts will occur.
Therefore, I keep most changes in mod citeproc_js {…}. I hope this will make git merge easier…

This PR depends on typst/citationberg#45. (Now merged)

Further details

CSL test suite and citeproc.js

CITATION-ITEMS vs. CITATIONS

Spec: http://github.com/citation-style-language/test-suite/blob/master/README.rst

An example with explanation

One @a, two @b @c, three @d.

Consider the citations in the above Typst document.

CITATION-ITEMS represents it as:

[
  [{ "id": "a" }],
  [{ "id": "b" }, { "id": "c" }],
  [{ "id": "d" }]
]

And CITATIONS represents it as three invocations to citeproc.js processCitationCluster():

• citation: CSL-JSON citation for the target citation cluster.
• citationsPre: A list of citationID/note-number pairs preceding the target.
• citationsPost: A list of citationID/note-number pairs following the target. Note numbers to reflect the state of the document after the insertion.

[
  [
    {
      "citation_id": "CITATION-1",
      "citation_items": [{ "id": "a" }]
    },
    [],
    []
  ],
  [
    {
      "citation_id": "CITATION-2",
      "citation_items": [{ "id": "b" }, { "id": "c" }]
    },
    [["CITATION-1", 0]],
    []
  ],
  [
    {
      "citation_id": "CITATION-3",
      "citation_items": [{ "id": "d" }]
    },
    [["CITATION-1", 0], ["CITATION-1", 1]],
    []
  ]
]

In some test cases, a citation is first added and then modified in a subsequent invocation.
For example, the above example can also be represented as:

1. Add CITATION-1 @a.
2. Add an empty CITATION-2.
3. Add CITATION-3 @d.
4. Replace the empty CITATION-2 with @b @c.

[
  [
    {
      "citation_id": "CITATION-1",
      "citation_items": [{ "id": "a" }]
    },
    [],
    []
  ],
  [
    {
      "citation_id": "CITATION-2",
      "citation_items": []
    },
    [["CITATION-1", 0]],
    []
  ],
  [
    {
      "citation_id": "CITATION-3",
      "citation_items": [{ "id": "d" }]
    },
    [["CITATION-1", 0], ["CITATION-1", 1]],
    []
  ],
  [
    {
      "citation_id": "CITATION-2",
      "citation_items": [{ "id": "b" }, { "id": "c" }]
    },
    [["CITATION-1", 0]],
    [["CITATION-3", 2]],
  ]
]

For Hayagriva, it is meaningless to test modifications, so it's okay to collect those citation_items and normalize CITATIONS as CITATION-ITEMS.

Citation RESULT

Ref:

• https://github.com/Juris-M/citeproc-js/blob/cc9153c45293af878de08cafddbefe6ea150c380/src/test_runner.js#L179-L212
• https://github.com/Juris-M/citeproc-js/blob/cc9153c45293af878de08cafddbefe6ea150c380/src/test_runner.js#L352-L354

For test cases that use CITATIONS and in the citation mode, the citeproc.js test runner prepends >> to entries updated in the last citation request, and prepends .. to others. It also prepends indices
as [n] (with a trailing space).

As a result, we'll see something like this in the RESULT section:

>>===== RESULT =====>>
..[0] Doe, “Life of Doe” (1901).
..[1] Roe, “Life of Roe” (1900).
..[2] Doe, supra note 1.
>>[3] Roe, supra note 2.
<<===== RESULT =====<<

For Hayagriva, it is also meaningless to test which entries were updated. Therefore, I just trim these prefixes as follows before comparison.

>>===== RESULT =====>>
Doe, “Life of Doe” (1901).
Roe, “Life of Roe” (1900).
Doe, supra note 1.
Roe, supra note 2.
<<===== RESULT =====<<

Affected test cases

42 tests are affected. Previously, all of them were skipped.

Skipped → Passed (16 tests)

• bugreports_AutomaticallyDeleteItemsFails
• bugreports_CreepingAddNames
• bugreports_DelimiterOnLayout
• bugreports_EtAlSubsequent
• bugreports_OverwriteCitationItems
• disambiguate_DisambiguationHang
• disambiguate_YearSuffixFiftyTwoEntries
• disambiguate_YearSuffixFiftyTwoEntriesByCite
• integration_DeleteName
• integration_DisambiguateAddGivenname1
• integration_DisambiguateAddGivenname2
• integration_SimpleFirstReferenceNoteNumber
• integration_SubsequentWhenInterveningFootnote
• integration_YearSuffixOnOffOn
• sort_CitationNumberPrimaryAscendingViaMacroCitation
• sort_CitationNumberPrimaryAscendingViaVariableCitation

Skipped → Failed (13 tests)

Manual check shows that these are real failures.

• Name (5 tests)

  • Works by the same author in the same citation group is not collapsed as expected.

    For example, expected Fitzpatrick, 2005a, 2005b, but got Fitzpatrick, 2005a; Fitzpatrick, 2005b.

    • sort_AuthorDateWithYearSuffix
    • sort_GroupedByAuthorstring

  • la Berge becomes Berge.

    • integration_DuplicateItem
    • integration_DuplicateItem2

  • <name form="long"/> is not respected.

    • name_BibliographyNameFormNeverShrinks

• Position (3 tests)

  • <if position="…"> does not match ibid or near-note as expected.

    • position_IbidInText
    • position_NearNoteSameNote

  • The term ibid is not capitalized as expected.

    • position_FirstTrueOnlyOnce

• Delimiter (5 tests)

  • The delimiter on <layout> is applied to the wrong place.

    For example, John Doe, His Book becomes John Doe; , His Book, where ; is the delimiter on <layout>.

    • disambiguate_BasedOnSubsequentFormWithBackref2
    • group_LegalWithAuthorDate
    • integration_IbidWithDifferentLocators
    • integration_SimpleIbid

  • Some delimiters are added incorrectly. This case is a bit complicated and I'm not sure about the exact reason.

    • punctuation_SuppressPrefixPeriodForDelimiterSemicolon

Cannot normalize (8 tests)

These tests are still skipped, because their citations fields are too complicated to be normalized as citation_items.

• NonSequentialCitations (6 tests)

  These test cases are about inserting new citations before existing ones. I don't quite understand the convention of citationsPre and citationsPost in this case, so I keep them skipped.

  • bugreports_ikeyOne (cannot normalize: only sequential citations can be tested at present, but got seen = ["CITATION-1", "CITATION-2"], current = "CITATION-2", pre = [], post = [])
  • collapse_CitationNumberRangesInsert (cannot normalize: only sequential citations can be tested at present, but got seen = ["CITATION-1"], current = "CITATION-2", pre = [], post = ["CITATION-1"])
  • integration_FirstReferenceNoteNumberPositionChange (cannot normalize: only sequential citations can be tested at present, but got seen = ["CITATION-1", "CITATION-2", "CITATION-3", "CITATION-4"], current = "CITATION-5", pre = [], post = ["CITATION-1", "CITATION-2", "CITATION-3", "CITATION-4"])
  • integration_IbidOnInsert (cannot normalize: only sequential citations can be tested at present, but got seen = ["CITATION-1", "CITATION-2", "CITATION-3"], current = "CITATION-4", pre = ["CITATION-1"], post = ["CITATION-2", "CITATION-3"])
  • position_ResetNoteNumbers (cannot normalize: only sequential citations can be tested at present, but got seen = ["CITATION-1", "CITATION-2", "CITATION-3"], current = "CITATION-4", pre = [], post = ["CITATION-1", "CITATION-2", "CITATION-3"])
  • sort_RangeUnaffected (cannot normalize: only sequential citations can be tested at present, but got seen = ["CITATION-1"], current = "CITATION-2", pre = [], post = ["CITATION-1"])

• NonSequentialNoteNumbers (2 tests)

  I can't understand what's going on in these test cases… Keep them skipped for now.

  • position_IbidSeparateCiteSameNote (cannot normalize: only sequential note numbers can be tested at present, but got pre = [1, 2, 2], post = [])
  • position_IbidWithMultipleSoloCitesInBackref (cannot normalize: only sequential note numbers can be tested at present, but got pre = [1, 1], post = [])

• MissingResultPrefix (0 test)

  This error variant is left for trimming the citation result as described above. All test cases match the expectation, so the error is never triggered.

Normalized but still skipped (5 tests)

These tests are still skipped, for reasons other than the citations → citation_items normalization.

• (cause: unsupported test feature)

  These test cases use affixes in citations.

  • affix_WithCommas
  • magic_TermCapitalizationWithPrefix
  • position_IbidWithPrefixFullStop

• (cause: HTML suspected in citation result)

  These test cases use <i>, <b>, <span style="font-variant:small-caps;">, or other HTML markups.

  • bugreports_OldMhraDisambiguationFailure
  • testers_FirstAutoGeneratedZoteroPluginTest

Footnotes

1. bugreports_EnvAndUrb is an exception. I believe it's an error and I've reported in bugreports_EnvAndUrb is malformed: It contains both CITATION-ITEMS and CITATIONS citation-style-language/test-suite#88. ↩

[Drodt]

Thanks! That looks great. Two questions:

1. You say this is not the final form. Do you want to hold off on merging this PR until Restructure Hayagriva as a Workspace #489 is merged (I hope we get significant progress on that PR this month)?
2. Do you plan to support citationsPre and citationsPost? I can also take a look if I can make sense of that if you like.

[YDX-2147483647]

1. Do you want to hold off on merging this PR until Restructure Hayagriva as a Workspace #489 is merged?

Both options are okay to me. Either you or I will have to resolve merge conflicts.

(What I meant was that the relevant code will need further modifications before the next hayagriva release, so it's not the final form. Sorry if there was any ambiguity caused.)

Do you plan to support citationsPre and citationsPost?

No. This only involves six test cases, and the bugs they may catch look irrelevant to hayagriva.
I think we should prioritize other things, such as supporting CITATION-ITEMS with prefix/suffix.

[YDX-2147483647]

I've updated Cargo.toml to replace my citationberg fork with the latest official commit.