Skip to content

multiple slashes within a string not properly processed #40

Description

@maia

A string containing multiple slashes will only be split by its last ocurrence of a slash:

string = "Washington/London/Paris/Tokyo"
PragmaticTokenizer::Tokenizer.new().tokenize(string)
=> ["washington/london/paris", "tokyo"]

This is caused by the NOT_URL regex, which is used in the SLASH_NOT_URL regex. When combining the sources, the resulting regex is the one used here. I hope someone can come up with a fix, as I forgot a lot about regex in the past months and cannot provide a quick fix myself.

See here:
https://rubular.com/r/scNIh27gBLIX03

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions