de.uni_luebeck.isp.compacom.SimpleTokens
SimpleTokenizer
Related Doc:
package SimpleTokens
A list of start-end pairs.
A list of start-end pairs. Anything between the start string and the end string will be ignored. Comments are not recursive. Comments are automatically closed at the end of file.
As comments are checked before anything else, the starting string of a comment must not be the prefix of any other valid token (or else that token will never be matched).
Example:
val comments = List("//" -> "\n", "/*" -> "*/")
A list of tokens that fit the rules for being identifiers, but should be tokenized as - * something else instead
A list of tokens that aren't integers, strings, comments, spaces or identifiers and don't overlap with either.
If the next characters in the input stream are equal to the given string, consume them and return true.
If the next characters in the input stream are equal to the given string, consume them and return true. Otherwise return false.
Same as de.uni_luebeck.isp.compacom.Tokens.AbstractTokenizer.peekWhile, but consume the characters.
Same as de.uni_luebeck.isp.compacom.Tokens.AbstractTokenizer.peekWhile, but consume the characters.
The location of the current token
The location of the current token
The location and value of the current token
The location and value of the current token
The index of the current token (a.k.a.
The index of the current token (a.k.a. the number of tokens previously read). This is used to determine whether a parser rule consumed more than one token, which is necessary when determining whether to ask the second rule for completions when evaluating alternatives (see definition of Parsers.|).
Return a Some containing the first of the given token that matches the current input or
None if none do.
Return a Some containing the first of the given token that matches the current input or
None if none do.
If a match is found, the tokenizer is advanced to after the match. Otherwise the tokenizer does not move.
This method can be used to easily match all symbolic tokens in one swoop. It can not be used to match alphanumeric keywords as you wouldn't want to match "def" when the input is "definitely".
A stream containing the characters returned by peek(0) onwards.
A stream containing the characters returned by peek(0) onwards.
Return a string containing the next length characters without consuming any.
Return a string containing the next length characters without consuming any.
Return a string containing all the characters in the input until the first that does not meet the predicate without consuming any.
Return a string containing all the characters in the input until the first that does not meet the predicate without consuming any.
The end position of the previous token, that is the position that the input stream was at before the current token was matched.
The end position of the previous token, that is the position that the input stream was at before the current token was matched. The area from this position to the starting position of the current location should contain exactly the characters that were discarded between the previous token and the current one.
Implement a tokenizer by simply providing a list of keywords, symbols and comment delimiters.
Anything that matches an item in one of the lists is tokenized accordingly. Any sequence of alphanumeric characters starting with a letter or underscore is tokenized as an identifier if it is not a keyword. A sequence of digits is tokenized as an integer. A sequence of characters starting with a quote and ending with a quote is tokenized as a string. Anything else is an invalid token.
This is the easiest, but least flexible way to define a tokenizer. However it should be sufficient for many cases.