Lexing of string interpolation

[kubouch]

Nushell supports string interpolation:

• $"foo(<expression>)bar" (double-quoted string supports escapes)
• $'foo(<expression>)bar' (single-quoted string doesn't have escapes)
  where <expression> can be any valid Nushell syntax.

This is problematic to lex because we need to lex the parts outside the () as string chunks and tha parts inside () as full Nushell syntax. We need to somehow be able to switch to a different lexing mode when encountering $" or $', then switch to the default mode when encountering (, go back to the string interpolation mode on ) and finally return to the default mode on " or '.

Two possible solutions:

• Use morph() inside lex() to switch between the string interpolation and default Nushell lexers. Related Logos issue: Context-dependent lexing maciejhirsz/logos#116, example code is mentioned in the issue.
• If the above is too unwieldy, switch back to on-demand lexing, i.e., make Tokens.advance() lex and fetch the next token from the Logos lexer iterator instead of lexing everything in advance. This would make is possible to switch the lexer during parsing, making it possibly easier to implement, but also possibly slower.