TokenStream (com.gengoai:mango 1.1-SNAPSHOT API)

All Superinterfaces:

Serializable

All Known Implementing Classes:

AbstractTokenStream, Parser
```
public interface TokenStream
extends Serializable
```
A stream of tokens extracted via a Lexer allowing for single token look-ahead. TokenStream implementations should signal the end of stream using the EOF_TOKEN which has a special EOF tag.

Author:

David B. Bracewell

Field Summary

Fields
Modifier and Type Field Description

static Tag EOF
Special tag signaling the End-of-File (i.e.

static ParserToken EOF_TOKEN
Special token signaling the End-of-File (i.e.

Method Summary

All Methods Instance Methods Abstract Methods Default Methods
Modifier and Type	Method	Description
`ParserToken`	`consume()`	Consumes a token from the stream.
`default ParserToken`	`consume(Tag target)`	Consume a token from the stream expecting the tag to be an instance of the given target type.
`default boolean`	`hasNext()`	Checks if there are more non-EOF tokens on the stream
`ParserToken`	`peek()`	Peeks at the next token on the stream.
`ParserToken`	`token()`	Returns the last token extracted via the call to consume.

- Field Detail
  - EOF
```
static final Tag EOF
```
    Special tag signaling the End-of-File (i.e. stream)
  - EOF_TOKEN
```
static final ParserToken EOF_TOKEN
```
    Special token signaling the End-of-File (i.e. stream)
- Method Detail
  - consume
```
ParserToken consume()
```
    Consumes a token from the stream.
    
    Returns:
    
    the consumed token.
  - consume
```
default ParserToken consume(Tag target)
```
    Consume a token from the stream expecting the tag to be an instance of the given target type.
    
    Parameters:
    
    target - the target tag that the consumed should be an instance of
    
    Returns:
    
    the consumed token
  - hasNext
```
default boolean hasNext()
```
    Checks if there are more non-EOF tokens on the stream
    
    Returns:
    
    True if the next consumable token exists and is not EOF, False otherwise
  - peek
```
ParserToken peek()
```
    Peeks at the next token on the stream.
    
    Returns:
    
    the next token on the stream (special EOF token if no more tokens exists)
  - token
```
ParserToken token()
```
    Returns the last token extracted via the call to consume.
    
    Returns:
    
    the last extracted token via consume