SimpleTokenizer
Related Doc:
package compacom
A list of start-end pairs.
A list of start-end pairs. Anything between the start string and the end string will be ignored. Comments are not recursive. Comments are automatically closed at the end of file.
As comments are checked before anything else, the starting string of a comment must not be the prefix of any other valid token (or else that token will never be matched).
Example:
val comments = List("//" -> "\n", "/*" -> "*/")
A list of tokens that fit the rules for being identifiers, but should be tokenized as - * something else instead
A list of tokens that aren't integers, strings, comments, spaces or identifiers and don't overlap with either.
Implement a tokenizer by simply providing a list of keywords, symbols and comment delimiters.
Anything that matches an item in one of the lists is tokenized accordingly. Any sequence of alphanumeric characters starting with a letter or underscore is tokenized as an identifier if it is not a keyword. A sequence of digits is tokenized as an integer. A sequence of characters starting with a quote and ending with a quote is tokenized as a string. Anything else is an invalid token.
This is the easiest, but least flexible way to define a tokenizer. However it should be sufficient for many cases.