#include <wx/string.h>
#include <configmanager.h>
#include <filemanager.h>
#include "token.h"
#include <stack>
#include <list>

Include dependency graph for tokenizer.h:

This graph shows which files directly or indirectly include this file:

Classes
struct	TokenizerOptions
	Whether we need to handle C-preprocessor directives. More...

class	Tokenizer
	This is just a simple lexer class. More...

struct	Tokenizer::ExpandedMacro
	replaced buffer information Here is an example of how macro are expanded More...

Enumerations
enum	TokenizerState { tsNormal = 0x0000, tsRawExpression = 0x0001 }
	Enum defines the skip state of the Tokenizer. More...

enum	PreprocessorType { ptIf = 0x0001, ptIfdef = 0x0002, ptIfndef = 0x0003, ptElif = 0x0004, ptElifdef = 0x0005, ptElifndef = 0x0006, ptElse = 0x0007, ptEndif = 0x0008, ptDefine = 0x0009, ptUndef = 0x000A, ptOthers = 0x000B }
	Enum categorizing C-preprocessor directives. More...

Class Documentation

◆ TokenizerOptions

struct TokenizerOptions

Whether we need to handle C-preprocessor directives.

Definition at line 44 of file tokenizer.h.

Collaboration diagram for TokenizerOptions:

Class Members
bool	storeDocumentation	do we store the doxygen like document
bool	wantPreprocessor	do we expand the macros in #if like conditional preprocessor directives

Enum categorizing C-preprocessor directives.

Enumerator
ptIf	#if
ptIfdef	#ifdef
ptIfndef	#ifndef
ptElif	#elif
ptElifdef	#elifdef
ptElifndef	#elifndef
ptElse	#else
ptEndif	#endif
ptDefine	#define
ptUndef	#undef
ptOthers	#include #warning and other #xxx

Definition at line 28 of file tokenizer.h.

Enum defines the skip state of the Tokenizer.

Enumerator

tsNormal

read parentheses as a single token

tsRawExpression

read parentheses as token lists, so it return several tokens like '(' ...

')'

Definition at line 19 of file tokenizer.h.