Split some tokens - #4199
Conversation
This more closely matches parse.y, and simplifies a bit of the lex state as seen from the outside.
This also more closely matches parse.y, and makes it easier to delineate which brace belongs to which type.
Now we fully match parse.y, and it is clearer to distinguish betewen backticks that are used for method names and backticks that begin xstring literals.
|
I would welcome to remove the state from the lex result. But doing so in a compatible way might be a bit bothersome. When I rewrote BTW, this broke some irb colorizer tests. That's fine and easy to fix though |
Changed in ruby/prism#4199. Other token changes are irrelevant since they are already not used.
Changed in ruby/prism#4199. Other token changes are irrelevant since they are already not used.
|
I see you crossed out |
|
That's the new trailing comma syntax. I tested against older irb locally and forgot that this changed. Oops |
Changed in ruby/prism#4199. Other token changes are irrelevant since they are already not used.
2 participants
Some tokens can represent multiple things depending on context, like
(,{, and ```. These tokens are represented in parse.y as different tokens, and therefore have different grammar rules applied to them. This PR mirrors that logic in order to make comparing against the upstream grammar simpler.It has the added benefit of making it so that we no longer have to rely on any internal state in our translation engines (except for in lex compat).
@Earlopain do you think we could drop the lex state from the public API after this?