Implement a tokenizer by simply providing a list of keywords, symbols and comment delimiters.
Implement a tokenizer by simply providing a list of keywords, symbols and comment delimiters.
Anything that matches an item in one of the lists is tokenized accordingly. Any sequence of alphanumeric characters starting with a letter or underscore is tokenized as an identifier if it is not a keyword. A sequence of digits is tokenized as an integer. A sequence of characters starting with a quote and ending with a quote is tokenized as a string. Anything else is an invalid token.
This is the easiest, but least flexible way to define a tokenizer. However it should be sufficient for many cases.
A set of default tokens emitted by SimpleTokenizer
Reimplements some Character and String methods as the originals are not supported (or, in case of toInt, don't support Unicode) in Scala.JS.