SimpleTokens

Type Members

1. abstract class AbstractTokenizer extends Tokenizer

   

   Definition Classes

   Tokens

2. case class COMMENT(open: String, text: String, close: String) extends Token with Product with Serializable

   

3. case class ID(name: String) extends Token with Product with Serializable

   

4. case class INT(value: Int) extends Token with Product with Serializable

   

5. case class Invalid(string: String) extends Token with Product with Serializable

   

   Definition Classes

   Tokens

6. case class SPACE(spaces: String) extends Token with Product with Serializable

   

7. case class STRING(value: String) extends Token with Product with Serializable

   

8. abstract class SimpleTokenizer extends AbstractTokenizer

   

   Implement a tokenizer by simply providing a list of keywords, symbols and comment delimiters.

   Implement a tokenizer by simply providing a list of keywords, symbols and comment delimiters.

   Anything that matches an item in one of the lists is tokenized accordingly. Any sequence of alphanumeric characters starting with a letter or underscore is tokenized as an identifier if it is not a keyword. A sequence of digits is tokenized as an integer. A sequence of characters starting with a quote and ending with a quote is tokenized as a string. Anything else is an invalid token.

   This is the easiest, but least flexible way to define a tokenizer. However it should be sufficient for many cases.

9. abstract class Token extends AnyRef

   
   Definition Classes

   Tokens

10. trait Tokenizer extends AnyRef

    

    Definition Classes

    Tokens