SimpleTokenizer
Related Doc:
package compacom
A list of start-end pairs.
A list of start-end pairs. Anything between the start string and the end string will be ignored. Comments are not recursive. Comments are automatically closed at the end of file.
As comments are checked before anything else, the starting string of a comment must not be the prefix of any other valid token (or else that token will never be matched).
Example:
val comments = List("//" -> "\n", "/*" -> "*/")
A list of tokens that fit the rules for being identifiers, but should be tokenized as - * something else instead
A list of tokens that aren't integers, strings, comments, spaces or identifiers and don't overlap with either.
Return true if the given character can occur in the middle or end of an identifier.
Return true if the given character can occur in the middle or end of an identifier.
By default this returns true for Unicode letters, Unicode digits and the ASCII underscore. Override this if your language has different rules for what an identifier may contain.
Returns true if the given character can occur at the beginning of an identifier.
Returns true if the given character can occur at the beginning of an identifier.
By default this returns true for Unicode letters and the ASCII underscore. Override this if your language has different rules for how an identifier can begin.
Implement a tokenizer by simply providing a list of keywords, symbols and comment delimiters.
Anything that matches an item in one of the lists is tokenized accordingly. Any sequence of alphanumeric characters starting with a letter or underscore is tokenized as an identifier if it is not a keyword. A sequence of digits is tokenized as an integer. A sequence of characters starting with a quote and ending with a quote is tokenized as a string. Anything else is an invalid token.
This is the easiest, but least flexible way to define a tokenizer. However it should be sufficient for many cases.