Expand the syntax definition page

[michaelblyons]

• Groups syntax sections in guide and reference
• Adds a parallel tutorial for sublime-syntax to the existing YAML-tmLanguage one.
• Splits syntax engine discourse from the tutorial into a new page.

I did not use the "all 1.s" guidance in CONTRIBUTING so it's easier to notice if the "step X" references fall out of line.

[FichteFoll]

Thanks for taking a jab at the syntax definition tutorial. I understand that you mostly used the legacy one for YAML-tmLanguage as the base and iterated on it from there (it shows), which I believe is a good approach. However, the old guide did not only use an old format but it was also not so great in a couple of ways.

I have annotated a couple points already that I would like to see a syntax guide to do better. I did not finish the review, but you can probably interpolate from the comments so far in what direction I personally would like the guide to go. Feel free to address the comments so far, but even if you do not I will try to set aside some time to work on this myself. As usual, no promises for an ETA, though.

Either way, before this can be merged it must provide value over the official documentation and should not provide obviously incomplete or wrong information. It does not have to be perfect, but it should not cause confusion either.

[FichteFoll]

I'm missing a general introduction of what "patterns" are used for, how syntax parsing works/is structured and what a context is. A comparison to a push-down automaton will also be helpful, perhaps even some graphical representation (optional). I believe this to be important to clarify before looking at what a "pattern" looks like.

[michaelblyons]

This is in the new index.md, but I can link it from here. I may come back to this section after I work on the other comments.

[michaelblyons]

Let me know what you think about the changes.

[michaelblyons]

Is there a diagram engine in the builder already?

sequenceDiagram
    Context->>+Character:Next character

    loop
        Character->>+Rule:First rule of context

        loop Try each rule in context
            alt Rule did not match
                Rule->>Rule:Next rule of context
                deactivate Character
            else Simple rule matched
                Rule->>Character:Consume characters with rule
            else Push / Pop / Set rule matched
                Rule->>-Context:Consume characters and change context stack
            end

            Rule->>Character:Next character
        end
    end

Loading

[michaelblyons]

Alternative:

graph TB
    Start([Start])
    End([End])

    Contexts[(Contexts)]
    Context[/Current context/]
    Characters[(Characters)]
    Character[/Current char position/]
    Rules[(Rules)]
    Rule[/Current rule/]
    Match[/Match/]

    context-main[Set `main` context]

    char-eof{Is EOF?}

    rule-exists{Exists?}
    rule-matches{Matches?}
    rule-exists-no[Advance\n one char]

    Start-->context-main
    context-main-->Contexts
    Contexts-->Context
    Context-->Characters
    Characters-->Character
    Character-->char-eof
    char-eof--Yes.-->End
    char-eof--No.-->Rules
    Rules-->Rule
    Rule-->rule-exists
    rule-exists--No.-->rule-exists-no
    rule-exists-no-->Characters
    rule-exists--Yes.-->rule-matches
    rule-matches--No.-->Rules
    rule-matches--Yes.-->Match
    Match-->scope[Apply Scopes]
    scope-->advance-chars[Advance chars\n to end of match]
    advance-chars-->match-type{Context change?\n Push / Pop / Set}
    match-type--No.-->Characters
    match-type--Yes.-->change-context[Modify context stack]
    change-context-->Contexts

Loading

[FichteFoll]

It's really not that many nowadays.

[michaelblyons]

Not in the shipped packages, but there's a whole ecosystem of third-party languages made God-knows-when.

[FichteFoll]

I ran out of time here. (this is just a marker)