t

de.uni_luebeck.isp.compacom

SimpleTokenizer

trait SimpleTokenizer extends Tokenizer with GeneratesSimpleTokens

Implement a tokenizer by simply providing a list of keywords, symbols and comment delimiters.

Anything that matches an item in one of the lists is tokenized accordingly. Any sequence of alphanumeric characters starting with a letter or underscore is tokenized as an identifier if it is not a keyword. A sequence of digits is tokenized as an integer. A sequence of characters starting with a quote and ending with a quote is tokenized as a string. Anything else is an invalid token.

This is the easiest, but least flexible way to define a tokenizer. However it should be sufficient for many cases.

Linear Supertypes
GeneratesSimpleTokens, Tokenizer, AnyRef, Any
Known Subclasses
BasicExpTokenizer
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SimpleTokenizer
  2. GeneratesSimpleTokens
  3. Tokenizer
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. sealed abstract class State extends AnyRef
  2. class TokenIterator extends Iterator[Token]
    Definition Classes
    Tokenizer

Abstract Value Members

  1. abstract val comments: List[(String, String)]

    A list of start-end pairs.

    A list of start-end pairs. Anything between the start string and the end string will be ignored. Comments are not recursive. Comments are automatically closed at the end of file.

    As comments are checked before anything else, the starting string of a comment must not be the prefix of any other valid token (or else that token will never be matched).

    Example:

    val comments = List("//" -> "\n", "/*" -> "*/")
  2. abstract val keywords: List[SimpleTokens.Token]

    A list of tokens that fit the rules for being identifiers, but should be tokenized as - * something else instead

  3. abstract val symbols: List[SimpleTokens.Token]

    A list of tokens that aren't integers, strings, comments, spaces or identifiers and don't overlap with either.

  4. abstract val tokens: SimpleTokens
    Definition Classes
    GeneratesSimpleTokens → Tokenizer

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
  11. def isIdentifierCont(c: Char): Boolean

    Return true if the given character can occur in the middle or end of an identifier.

    Return true if the given character can occur in the middle or end of an identifier.

    By default this returns true for Unicode letters, Unicode digits and the ASCII underscore. Override this if your language has different rules for what an identifier may contain.

    Attributes
    protected
  12. def isIdentifierStart(c: Char): Boolean

    Returns true if the given character can occur at the beginning of an identifier.

    Returns true if the given character can occur at the beginning of an identifier.

    By default this returns true for Unicode letters and the ASCII underscore. Override this if your language has different rules for how an identifier can begin.

    Attributes
    protected
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. def nextToken(source: LookaheadSource, state: State): (SimpleTokens.Token, State)
    Definition Classes
    SimpleTokenizer → Tokenizer
  16. final def notify(): Unit
    Definition Classes
    AnyRef
  17. final def notifyAll(): Unit
    Definition Classes
    AnyRef
  18. val startState: State
    Definition Classes
    SimpleTokenizer → Tokenizer
  19. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  20. def toString(): String
    Definition Classes
    AnyRef → Any
  21. def tokenize(s: Source): TokenIterator
    Definition Classes
    Tokenizer
  22. def tokenize(s: LookaheadSource): TokenIterator
    Definition Classes
    Tokenizer
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. object Util
    Definition Classes
    Tokenizer

Inherited from GeneratesSimpleTokens

Inherited from Tokenizer

Inherited from AnyRef

Inherited from Any

Ungrouped