lex doesn't seem to support non-ascii characters

[moechakucha]

discovered the issue when I was attempting to add starship-based prompt. fancy prompt isn't much of an issue, but as I'm a chinese I'd work a LOT with chinese characters which is indeed a massive issue.

I don't know if this is one of the non-features (I'd assume not since utf-8 was born on plan 9) as it's not detailed in the readme, however I do think this is pretty much a big deal, hence the issue.

[enzv]

Thanks for reporting this @moechakucha. I think you're right, this probably shouldn't be an ASCII-only thing.

Looking at the lexer, I suspect these tables are part of the problem:

var wordCharTable [128]bool
var idCharTable [128]bool

and then things like:

func wordchr(c int) bool {
    return c >= 0 && c < 128 && wordCharTable[c]
}

So anything outside ASCII immediately stops being a word character.

I'm not quite sure what the nicest fix is yet.

Maybe the lexer can stay byte-based and just let bytes >= 128 pass as part of a word:

if c >= 128 {
    return true
}

Since rc syntax itself is ASCII, I wonder if that's actually enough. UTF-8 could just pass through untouched.

But I'm not sure if that starts getting weird around $ identifiers, positions, concatenation, or some other lexer state I'm missing.

The other direction would be making peek / advance rune-aware:

r, n := utf8.DecodeRuneInString(src[pos:])

That feels more correct, but also potentially like quite a bit more machinery for something the lexer may not need.

I haven't chased all the consequences, so I wouldn't prescribe either approach yet.

If you feel like poking at it, I'd be happy to take a PR. A few UTF-8 lexer tests would probably make the right direction clearer. Maybe CJK text since that's what exposed the bug, plus another non-ASCII case or two, across plain words, quotes, concatenation, and variables where applicable.