de.uni_luebeck.isp.compacom.examples
BasicExpTokenizer
Related Doc:
package examples
A list of start-end pairs.
A list of start-end pairs. Anything between the start string and the end string will be ignored. Comments are not recursive. Comments are automatically closed at the end of file.
As comments are checked before anything else, the starting string of a comment must not be the prefix of any other valid token (or else that token will never be matched).
Example:
val comments = List("//" -> "\n", "/*" -> "*/")
Return true if the given character can occur in the middle or end of an identifier.
Return true if the given character can occur in the middle or end of an identifier.
By default this returns true for Unicode letters, Unicode digits and the ASCII underscore. Override this if your language has different rules for what an identifier may contain.
Returns true if the given character can occur at the beginning of an identifier.
Returns true if the given character can occur at the beginning of an identifier.
By default this returns true for Unicode letters and the ASCII underscore. Override this if your language has different rules for how an identifier can begin.
A list of tokens that fit the rules for being identifiers, but should be tokenized as - * something else instead
A list of tokens that fit the rules for being identifiers, but should be tokenized as - * something else instead
A list of tokens that aren't integers, strings, comments, spaces or identifiers and don't overlap with either.
A list of tokens that aren't integers, strings, comments, spaces or identifiers and don't overlap with either.