---

TokenBuffer Class Reference

#include <TokenBuffer.hpp>

List of all members.

Public Methods
	TokenBuffer (TokenStream &input_)
virtual	~TokenBuffer ()
void	reset (void)
	Reset the input buffer to empty state.
int	LA (unsigned int i)
RefToken	LT (unsigned int i)
unsigned int	mark ()
void	rewind (unsigned int mark)
void	consume ()
virtual unsigned int	entries () const
	Return the number of entries in the TokenBuffer.
Protected Attributes
TokenStream &	input
	Token source.
unsigned int	nMarkers
	Number of active markers.
unsigned int	markerOffset
	Additional offset used when markers are active.
unsigned int	numToConsume
	Number of calls to consume() since last LA() or LT() call.
CircularQueue< RefToken >	queue
	Circular queue with Tokens.

---

Detailed Description

A Stream of Token objects fed to the parser from a TokenStream that can be rewound via mark()/rewind() methods.

A dynamic array is used to buffer up all the input tokens. Normally, "k" tokens are stored in the buffer. More tokens may be stored during guess mode (testing syntactic predicate), or when LT(i>k) is referenced. Consumption of tokens is deferred. In other words, reading the next token is not done by conume(), but deferred until needed by LA or LT.

Todo:

: see if we can integrate this one with InputBuffer into one template or so.

See also:

antlr.Token , antlr.TokenStream , antlr.TokenQueue

---

Constructor & Destructor Documentation

TokenBuffer::TokenBuffer (  TokenStream &    input_ )

Create a token buffer

TokenBuffer::~TokenBuffer (    )  [virtual]

---

Member Function Documentation

void TokenBuffer::consume (    )  [inline]

Mark another token for deferred consumption

unsigned int TokenBuffer::entries (    )  [virtual]

Return the number of entries in the TokenBuffer.

int TokenBuffer::LA (  unsigned int    i )

Get a lookahead token value

RefToken TokenBuffer::LT (  unsigned int    i )

Get a lookahead token

unsigned int TokenBuffer::mark (    )

Return an integer marker that can be used to rewind the buffer to its current state.

void TokenBuffer::reset (  void    )  [inline]

Reset the input buffer to empty state.

void TokenBuffer::rewind (  unsigned int    mark )

Rewind the token buffer to a marker. Parameters: mark  Marker returned previously from mark()

---

Member Data Documentation

TokenStream& TokenBuffer::input [protected]

Token source.

unsigned int TokenBuffer::markerOffset [protected]

Additional offset used when markers are active.

unsigned int TokenBuffer::nMarkers [protected]

Number of active markers.

unsigned int TokenBuffer::numToConsume [protected]

Number of calls to consume() since last LA() or LT() call.

CircularQueue<RefToken> TokenBuffer::queue [protected]

Circular queue with Tokens.

---

The documentation for this class was generated from the following files:

• TokenBuffer.hpp
• TokenBuffer.cpp

---