A list of start-end pairs.
A list of start-end pairs. Anything between the start string and the end string will be ignored. Comments are not recursive. Comments are automatically closed at the end of file.
As comments are checked before anything else, the starting string of a comment must not be the prefix of any other valid token (or else that token will never be matched).
Example:
val comments = List("//" -> "\n", "/*" -> "*/")
A list of tokens that fit the rules for being identifiers, but should be tokenized as - * something else instead
A list of tokens that fit the rules for being identifiers, but should be tokenized as - * something else instead
A list of tokens that aren't integers, strings, comments, spaces or identifiers and don't overlap with either.
A list of tokens that aren't integers, strings, comments, spaces or identifiers and don't overlap with either.