Lexer (com.gengoai:mango 1.1-SNAPSHOT API)

All Superinterfaces:

Serializable
```
public interface Lexer
extends Serializable
```
A Lexer tokenizes a string or resource into tokens.

Author:

David B. Bracewell

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Default Methods
Modifier and Type	Method	Description
`static Lexer`	`create(TokenDef... tokens)`	Creates a regular expression based lexer over the given token definitions.
`default TokenStream`	`lex(Resource resource)`	Reads from the given resource and tokenizes it into tokens
`TokenStream`	`lex(String input)`	Tokenizes the input string into tokens

- Method Detail
  - lex
```
TokenStream lex(String input)
```
    Tokenizes the input string into tokens
    
    Parameters:
    
    input - the input to tokenize
    
    Returns:
    
    A token stream wrapping the tokenization results
  - lex
```
default TokenStream lex(Resource resource)
                 throws IOException
```
    Reads from the given resource and tokenizes it into tokens
    
    Parameters:
    
    resource - the resource to read and tokenize
    
    Returns:
    
    A token stream wrapping the tokenization results
    
    Throws:
    
    IOException - Something went wrong reading from the input resource
  - create
```
static Lexer create(TokenDef... tokens)
```
    Creates a regular expression based lexer over the given token definitions.
    
    Parameters:
    
    tokens - the token definitions that include the token type (tag) and the regular expression pattern for matching.
    
    Returns:
    
    the constructed lexer