A string containing multiple slashes will only be split by its last ocurrence of a slash:
string = "Washington/London/Paris/Tokyo"
PragmaticTokenizer::Tokenizer.new().tokenize(string)
=> ["washington/london/paris", "tokyo"]
This is caused by the NOT_URL regex, which is used in the SLASH_NOT_URL regex. When combining the sources, the resulting regex is the one used here. I hope someone can come up with a fix, as I forgot a lot about regex in the past months and cannot provide a quick fix myself.
See here:
https://rubular.com/r/scNIh27gBLIX03
A string containing multiple slashes will only be split by its last ocurrence of a slash:
This is caused by the NOT_URL regex, which is used in the SLASH_NOT_URL regex. When combining the sources, the resulting regex is the one used here. I hope someone can come up with a fix, as I forgot a lot about regex in the past months and cannot provide a quick fix myself.
See here:
https://rubular.com/r/scNIh27gBLIX03