Restructure Hayagriva as a Workspace

[Drodt]

This PR reworks Hayagriva to be a workspace with the different parts of Hayagriva split into separate crates.

The crates and their purposes are:

• hayagriva-archive: Contains the data structures of Hayagriva's archived styles and locales
• hayagriva-biblatex: Empty right now but will contain the taxonomy (i.e., EntryLike implementation for biblatex's Entry)
• hayagriva-cli: The CLI, Hayagriva's binary
• hayagriva-core: Important trait and type definitions
• hayagriva-csl: Rendering for CSL
• hayagriva-format: Entry and other definitions for the Hayagriva format (and, right now, the translation from biblatex to Hayagriva)
• hayagriva-json: Definition for CSL-JSON and an EntryLike implementation for it

Building on this, we can achieve directly going from biblatex or CSL-JSON to the rendering stage without going through the (lossy) Hayagriva translation steps. Users can also include just the parts of Hayagriva they actually need.

This is still a draft. The TODOs are:

• Document the new structure
• Maybe get rid of hayagriva-util?

TODOs by @laurmaedje:

• Clarify crate structure and API
  • private LOCALES
  • No test_util in core
  • format modules
  • Hide utils in core
  • Move the taxonomy stuff of format into csl and make it depend on format?
  • Clarify split between hayagriva-core and hayagriva-format. E.g. PersonsWithRoles.
  • Rename hayagriva_json to hayagriva_csl_json.
  • Remove hayagriva-biblatex for now.
  • What will happen to the existing main hayagriva crate? Just deprecate it or would it make sense to have this in some way as an umbrella crate?
• Fix crates.io issues
• Fix test structure
• Nits
  • Doc on all types
  • crate descriptions
  • crate keywords
  • investigate why Swedish locale changed
  • use "crates/*" glob
  • remove extern crate
  • format imports correctly

[YDX-2147483647]

hayagriva-biblatex: Empty right now but will contain the taxonomy (i.e., EntryLike implementation for biblatex's Entry)

Has it been decided that impl hayagriva_core::EntryLike for biblatex::Entry will happen?

If I understand correctly, it will replace the arrow Bib(La)TeX *.bib → hayagriva::Entry with Bib(La)TeX *.bib → CSL variables in the diagram I posted in #485.
Therefore, the interoperability between Bib(La)TeX and CSL will be better, but it'll also be more likely that Bib(La)TeX *.bib and Hayagriva *.yaml produce inconsistent results¹.

So adding impl EntryLike for biblatex::Entry is a trade-off and not 100% perfect. It might require additional consideration.

Footnotes

1. Take is-numeric="volume" for 第1卷 should be false #439 as an example. At present, <if is-numeric="volume"> in CSL depends on matches!(resolve_number_variable(Volume), MaybeTyped::Typed(_)). If biblatex::Entry and hayagriva_core::Entry use different code paths to resolve_number_variable(), then it's possible that volume = {第1卷} in *.bib is regarded as numeric but volume: 第1卷 in *.yaml is not. In that case, *.bib and *.yaml will produce different results using the same CSL style. ↩

[Drodt]

I see the potential for discrepancies between biblatex and Hayagriva but I still think this is the right course for two reasons:

1. The current way is not good, as you know. The conversion from biblatex to Hayagriva is imperfect and information is lost. No conversion here is perfect (whether to Hayagriva or CSL-JSON) and if we have one step rather than two, less information may be lost.
2. The conversion from biblatex to CSL-JSON is a known and somewhat fixed problem. We can follow what Zotero's Better BibTeX does to a large degree. Many authors use Zotero and biblatex together (myself, for instance) and it works reasonably well.

My hope is -- and I will test this -- that Hayagriva's and biblatex's data structures for numbers, persons, etc. are not so different and that we can share code to a large extent. In my experience, the hard part here are not the data types but the different entry and field types.

If you don't agree or have any other thoughts, I'd be happy for your input.

[Drodt]

@laurmaedje This PR is ready for input. It's still a draft but if you have any feedback on the structure/approach, let me know.

[retorquere]

We can follow what Zotero's Better BibTeX does to a large degree

if this means you are aiming for a good bib(la)tex parser in rust, I am very interested. I have tried to get a few rust-based parser projects to the point they can cleanly parse the BBT test suite but that got kludgy very fast and I stranded on each. BBT builds an CST-ish of the input before it applies the myriad of arcane rules of (bib)latex that express how the input should be handled and I couldn't get that into any of them.

I would love to say goodbye to unified-latex. BBTs parser is a separate project BTW, at https://github.com/retorquere/bibtex-parser/

[Drodt]

For parsing of bib(la)tex we will still rely on typst/biblatex. But I really should test that against the BBT test suite. I expect I'll have to take a closer look at BBT's parser.

[retorquere]

That makes sense of course. But I think that was one of the projects I tried to adapt.

[Drodt]

Fair enough. I'll give the test suite a try and then pick your brain for ideas (also for #497).

[retorquere]

The BBT parser has a test suite that just does field-by-field parsing/transform and doesn't do further transformation to another format, so it should be fairly easy to test against. I am open to the possibility that there are testcases that turn out to be wrong, but by and large, I think it is good and representative. Whether you think all cases should be catered for is another matter -- the parser handles some cases that are uncommon occurrences. I find the added complexity worthwhile, but that is definitely a value judgement.

That parser has a few modes for recalcitrant users that do not want sentence-casing or case protection; it also transforms markup to the limited subset of HTML that Zotero supports. Open to changing that as long as I can keep targeting Zotero.

[laurmaedje]

Hey, I managed to take a look now. I like the overall direction of the multi-crate structure, but have some remarks. :)

Crate structure and public APIs

• I would like to see the public APIs tightended to the minimum that's necessary. This makes them clearer to use and easier to evolve without having public-facing breaking changes.

  Here are some examples:

  • LOCALES could be private since the public entry point is locales
  • For hayagriva-core, it's unfortunate to have a public test_util module
  • hayagriva-format has empty public interop and taxonomy modules and the citation_label module has just one Item. Maybe that isn't worth a module and could be reexported at the root or moved.

  In essence, I would try to have a more deliberatly crafted public API that does not necessarily just reflect the source code structure, but aims to be most easy to use and understand for a consumer of the crate.

  Sometimes, definitions are needed in other crates but not really supposed to be used by consumer. I'm primarily thinking of hayagriva_core::utils. We could consider making such things doc(hidden) and semver excempt.

  (The trade-off here is also a bit different than for Typst, I would say, as the compiler APIs are supposed to be permanently unstable, while here we could potentially want to release a stable version at some point.)

• It feels a bit strange that hayagriva_format (which I would think should be CSL-independent) depends on and re-exports citationberg. Would it make sense to move the taxonomy stuff into hayagriva-csl and make it depend on hayagriva-format?

• On that topic: The split between hayagriva-core and hayagriva-format is not 100% clear to me. E.g. PersonsWithRoles seems to only be used by hayagriva-format and could reasonably live there?

• The crate name hayagriva_json is a bit generic as the crate is CSL-specific. From the purpose, I would rather expect hayagriva_csl_json. But perhaps this isn't worth its own crate and can just be folded into hayagriva_csl?

• Is hayagriva-biblatex supposed to stay empty in this PR or is it supposed to be populated still? If it is supposed to stay empty, I would maybe just introduce it in a later PR.

• I'm wondering what will happen to the existing main hayagriva crate? Just deprecate it or would it make sense to have this in some way as an umbrella crate?

Making it work with crates.io

There are two issues which will cause issues once we release to crates.io.

• Including the archives from outside the crate root of hayagriva-archive will not work when the crate is published to crates.io. We would need to move the archive files into that crate.

• The current path dependencies will not work with crates.io. Instead of hayagriva-core = { path = "../hayagriva-core" }, I would use hayagriva-core = { workspace = true } and at the workspace root add hayagriva-core = { path = "crates/hayagriva-core", version = ".." }. You can see this pattern in action in typst/typst.

Tests

To me, the current nested tests/tests structure is a bit confusing. Also, there is no need for empty tests/src/lib.rs. I would eliminate one layer of nesting and I would also lift that folder out of crates to the top-level. You can see an example of just this pattern in typst/typst.

Nits

Finally, just some nits. I expect that some of these are just not addressed yet because we want to settle the broad picture first, but I thought I'd just mention everything I noticed for completeness.

• It would be nice to have docs on all types that are visible on any of the crates via cargo doc --workspace
• It would be also be nice if all crates had a description (Side note: The description for hayagriva-biblatex is currently wrong)
• Why did the sv-SE locale change? If there was an intentional locale update, maybe it would be better to keep it separate from this PR
• You have set keywords.workspace = true in the crates, but deleted the top-level keywords array
• The workspace members array can use globs such as "crates/*"
• The extern crate declarations are not necessary
• There are some cases where the standard import order (std / extern / local blocks separate by blanks) is not respected. Also, we generally don't use nested imports for modules in our style (unfortunately not enforcable via rustfmt on stable)