Code::Blocks  SVN r11506
Public Types | Public Member Functions | Public Attributes | Protected Attributes | Friends | List of all members
Token Struct Reference

a symbol found in the parsed files, it can be many kinds, such as a variable, a class and so on. More...

#include <token.h>

Collaboration diagram for Token:

Public Types

enum  Type {
  Undefined, OpenBrace, CloseBrace, Equal,
  String, Comma
}
 

Public Member Functions

 Token (const wxString &name, unsigned int file, unsigned int line, size_t ticket)
 constructor More...
 
 ~Token ()
 destructor More...
 
bool AddChild (int childIdx)
 add a child token More...
 
bool DeleteAllChildren ()
 delete all the child tokens of the current token, not only remove the relation, but also delete the Token instance. More...
 
bool HasChildren () const
 check if the token has any child tokens. More...
 
wxString GetNamespace () const
 get a literal string presentation of the namespace. More...
 
bool InheritsFrom (int idx) const
 check to see the current token is inherited from a specified token More...
 
wxString DisplayName () const
 a short simple string to show the token information, this usually generate for show the tip message when the user hover a mouse over the text in C::B editor. More...
 
wxString GetTokenKindString () const
 the token kind string, e.g. More...
 
wxString GetTokenScopeString () const
 the access kind string, e.g. More...
 
wxString GetFilename () const
 get a full path of the file which contains the current Token More...
 
wxString GetImplFilename () const
 get a full path of the file which contains the function implementation. More...
 
wxString GetFormattedArgs () const
 remove all '
' in the original function argument string More...
 
wxString GetStrippedArgs () const
 remove all default value of the function argument string, e.g. More...
 
size_t GetTicket () const
 get the ticket value of the current token More...
 
bool MatchesFiles (const TokenFileSet &files)
 see whether the current token belong to any files in the file set, both m_FileIdx and m_ImplFileIdx is considered More...
 
TokenTreeGetTree () const
 get the TokenTree associated with the current Token More...
 
bool IsValidAncestor (const wxString &ancestor)
 build in types are not valid ancestors for a type define token More...
 
 Token ()
 
 Token (int start_, int end_, Type type_)
 
bool operator== (Token const &t) const
 
wxString ExtractString (wxString const &s) const
 
void Trim (wxString const &s)
 

Public Attributes

wxString m_FullType
 this is the full return value (if any): e.g. More...
 
wxString m_BaseType
 this is what the parser believes is the actual return value: e.g. More...
 
wxString m_Name
 Token's name, it can be searched in the TokenTree. More...
 
wxString m_Args
 If it is a function Token, then this value is function arguments, e.g. More...
 
wxString m_BaseArgs
 stripped arguments e.g. More...
 
wxString m_AncestorsString
 all ancestors comma-separated list, e.g. More...
 
unsigned int m_FileIdx
 File index in TokenTree. More...
 
unsigned int m_Line
 Line index where the token was met, which is 1 based. More...
 
wxString m_Doc
 doxygen style comments More...
 
unsigned int m_ImplFileIdx
 function implementation file index More...
 
unsigned int m_ImplLine
 function implementation line index More...
 
unsigned int m_ImplLineStart
 if token is impl, opening brace line More...
 
unsigned int m_ImplLineEnd
 if token is impl, closing brace line More...
 
wxString m_ImplDoc
 doxygen style comments in the Impl file More...
 
TokenScope m_Scope
 public? private? protected? More...
 
TokenKind m_TokenKind
 See TokenKind class. More...
 
bool m_IsOperator
 is operator overload function? More...
 
bool m_IsLocal
 if true, means the token belong to a C::B project, it exists in the project's source/header files, not from the system's headers or other include header files More...
 
bool m_IsTemp
 local (automatic) variable More...
 
bool m_IsConst
 the member method is const (yes/no) More...
 
bool m_IsNoExcept
 the member method is noexcept (yes/no) More...
 
bool m_IsAnonymous
 Is anonymous token? (e.g. More...
 
int m_Index
 current Token index in the tree, it is index of the std::vector<Token*>, so use the index, we can first get a address of the Token, and after a dereference, we can get the Token instance. More...
 
int m_ParentIndex
 Parent Token index. More...
 
TokenIdxSet m_Children
 if it is a class kind token, then it contains all the member tokens More...
 
TokenIdxSet m_Ancestors
 all the ancestors in the inheritance hierarchy More...
 
TokenIdxSet m_DirectAncestors
 the nearest ancestors More...
 
TokenIdxSet m_Descendants
 all the descendants in the inheritance hierarchy More...
 
wxArrayString m_Aliases
 used for namespace aliases More...
 
wxString m_TemplateArgument
 template argument list, comma separated list string More...
 
wxArrayString m_TemplateType
 for a class template, this is the formal template argument list, but for a variable Token, this is the actual template arguments. More...
 
std::map< wxString, wxStringm_TemplateMap
 a string to string map from formal template argument to actual template argument More...
 
wxString m_TemplateAlias
 alias for templates, e.g. More...
 
void * m_UserData
 custom user-data (the classbrowser expects it to be a pointer to a cbProject), this field is used when user can only show the tokens belong to the current C::B project in the browser tree. More...
 
int start
 
int end
 
Type type
 
bool hasRepeatedChar
 

Protected Attributes

TokenTreem_TokenTree
 a pointer to TokenTree More...
 
size_t m_Ticket
 This is used in class browser to avoid duplication nodes in the class browser tree. More...
 

Friends

class TokenTree
 

Detailed Description

a symbol found in the parsed files, it can be many kinds, such as a variable, a class and so on.

Once constructed, the Token should be always put in the TokenTree, and we can access the Token by TokenIndex.

Definition at line 82 of file token.h.

Member Enumeration Documentation

◆ Type

Enumerator
Undefined 
OpenBrace 
CloseBrace 
Equal 
String 
Comma 

Definition at line 20 of file parsewatchvalue.cpp.

Constructor & Destructor Documentation

◆ Token() [1/3]

Token::Token ( const wxString name,
unsigned int  file,
unsigned int  line,
size_t  ticket 
)

constructor

Parameters
nametoken's name, this can be a search key in the tokentree
filethe index of the source file where the token locates
ticketan integer number, once a new Token is allocated, this value will increase by one, it is mainly used for duplicated Token checking when class browser tree get refreshed.

Definition at line 48 of file token.cpp.

◆ ~Token()

Token::~Token ( )

destructor

Definition at line 73 of file token.cpp.

References m_TemplateMap, and m_TemplateType.

◆ Token() [2/3]

Token::Token ( )
inline

Definition at line 30 of file parsewatchvalue.cpp.

Referenced by GetNextToken().

◆ Token() [3/3]

Token::Token ( int  start_,
int  end_,
Type  type_ 
)
inline

Definition at line 37 of file parsewatchvalue.cpp.

Member Function Documentation

◆ AddChild()

bool Token::AddChild ( int  childIdx)

add a child token

Parameters
childIdxint
Returns
return true if success, in the case the childIdx < 0, this function will return false

Definition at line 267 of file token.cpp.

References m_Children.

Referenced by ParserThread::DoAddToken(), ParserThread::FindTokenFromQueue(), ParseCDBWatchValue(), and ParseGDBWatchValue().

◆ DeleteAllChildren()

bool Token::DeleteAllChildren ( )

delete all the child tokens of the current token, not only remove the relation, but also delete the Token instance.

Returns
false if the TokenTree is invalid

Definition at line 275 of file token.cpp.

References TokenTree::erase(), m_Children, and m_TokenTree.

Referenced by NativeParserBase::RemoveLastFunctionChildren().

◆ DisplayName()

wxString Token::DisplayName ( ) const

◆ ExtractString()

wxString Token::ExtractString ( wxString const &  s) const
inline

Definition at line 49 of file parsewatchvalue.cpp.

References end, wxString::length(), start, and wxString::substr().

Referenced by AddChild(), and ParseGDBWatchValue().

◆ GetFilename()

wxString Token::GetFilename ( ) const

◆ GetFormattedArgs()

wxString Token::GetFormattedArgs ( ) const

◆ GetImplFilename()

wxString Token::GetImplFilename ( ) const

get a full path of the file which contains the function implementation.

Note usually only function like token has this feature.

Definition at line 192 of file token.cpp.

References _T, TokenTree::GetFilename(), m_ImplFileIdx, and m_TokenTree.

Referenced by HasChildren(), DocumentationHelper::OnDocumentationLink(), CodeCompletion::OnGotoDeclaration(), ClassBrowser::OnJumpTo(), ClassBrowser::OnTreeItemDoubleClick(), and ClassBrowser::ShowMenu().

◆ GetNamespace()

wxString Token::GetNamespace ( ) const

get a literal string presentation of the namespace.

Returns
wxString if the token has the parent token, like a member variable class AAA {int m_BBB;}, then the string is "AAA::"

Definition at line 253 of file token.cpp.

References _T, TokenTree::at(), ParserConsts::dcolon, m_Name, m_ParentIndex, m_TokenTree, and wxString::Prepend().

Referenced by DisplayName(), NativeParser::FindCurrentFunctionStart(), DocumentationHelper::GenerateHTML(), HasChildren(), CodeCompletion::OnGotoFunction(), ClassBrowser::OnSearch(), NativeParser::ParseBufferForUsingNamespace(), and CodeCompletion::ParseFunctionsAndFillToolbar().

◆ GetStrippedArgs()

wxString Token::GetStrippedArgs ( ) const

remove all default value of the function argument string, e.g.

if we have such argument "(int a = 10, float b = 3.14)" then we get "(int a, float b)"

Definition at line 206 of file token.cpp.

References _T, wxString::Alloc(), wxString::IsEmpty(), wxString::Last(), wxString::Len(), m_Args, wxString::Trim(), and wxEmptyString.

Referenced by DisplayName(), CodeCompletion::DoAllMethodsImpl(), DocumentationHelper::GenerateHTML(), and HasChildren().

◆ GetTicket()

size_t Token::GetTicket ( ) const
inline

get the ticket value of the current token

Definition at line 161 of file token.h.

References m_Ticket, and MatchesFiles().

Referenced by ClassBrowserBuilderThread::AddNodes(), and ClassBrowserBuilderThread::RemoveInvalidNodes().

◆ GetTokenKindString()

wxString Token::GetTokenKindString ( ) const

◆ GetTokenScopeString()

wxString Token::GetTokenScopeString ( ) const

the access kind string, e.g.

for a tsPrivate type Token, we get a "private" string

Definition at line 333 of file token.cpp.

References _T, m_Scope, tsPrivate, tsProtected, tsPublic, tsUndefined, and wxEmptyString.

Referenced by DocumentationHelper::GenerateHTML(), and HasChildren().

◆ GetTree()

TokenTree* Token::GetTree ( ) const
inline

get the TokenTree associated with the current Token

Returns
TokenTree pointer

Definition at line 173 of file token.h.

References IsValidAncestor(), and m_TokenTree.

Referenced by InsertClassMethodDlgHelper::DoFillMethodsFor(), and ClassBrowserBuilderThread::TokenContainsChildrenOfKind().

◆ HasChildren()

bool Token::HasChildren ( ) const
inline

check if the token has any child tokens.

Returns
true if it does have some child.

Definition at line 113 of file token.h.

References DisplayName(), GetFilename(), GetFormattedArgs(), GetImplFilename(), GetNamespace(), GetStrippedArgs(), GetTokenKindString(), GetTokenScopeString(), InheritsFrom(), and m_Children.

Referenced by NativeParserBase::FindCurrentFunctionScope().

◆ InheritsFrom()

bool Token::InheritsFrom ( int  idx) const

check to see the current token is inherited from a specified token

Parameters
idxthe specified token
Returns
true if success loop the m_DirectAncestors to see it did contains the specified token, also we may check the specified token is an ancestor of the current ancestor, so recursive call is involved

Definition at line 289 of file token.cpp.

References TokenTree::at(), InheritsFrom(), m_DirectAncestors, and m_TokenTree.

Referenced by HasChildren(), and InheritsFrom().

◆ IsValidAncestor()

bool Token::IsValidAncestor ( const wxString ancestor)

build in types are not valid ancestors for a type define token

Parameters
ancestortesting type string
Returns
false if the testing type string is a build in type

Definition at line 133 of file token.cpp.

References _T, wxString::Len(), and wxString::StartsWith().

Referenced by DocumentationHelper::ConvertTypeToAnchor(), ParserThread::DoParse(), and GetTree().

◆ MatchesFiles()

bool Token::MatchesFiles ( const TokenFileSet files)

see whether the current token belong to any files in the file set, both m_FileIdx and m_ImplFileIdx is considered

Parameters
filesa file(index) set
Returns
true if success

Definition at line 239 of file token.cpp.

References m_FileIdx, and m_ImplFileIdx.

Referenced by GetTicket().

◆ operator==()

bool Token::operator== ( Token const &  t) const
inline

Definition at line 45 of file parsewatchvalue.cpp.

References end, start, and type.

◆ Trim()

void Token::Trim ( wxString const &  s)
inline

Definition at line 55 of file parsewatchvalue.cpp.

References end, wxString::length(), start, and wxT.

Referenced by ParseGDBWatchValue().

Friends And Related Function Documentation

◆ TokenTree

friend class TokenTree
friend

Definition at line 84 of file token.h.

Member Data Documentation

◆ end

int Token::end

◆ hasRepeatedChar

bool Token::hasRepeatedChar

Definition at line 67 of file parsewatchvalue.cpp.

Referenced by GetNextToken(), and ParseGDBWatchValue().

◆ m_Aliases

wxArrayString Token::m_Aliases

used for namespace aliases

Definition at line 280 of file token.h.

Referenced by CodeCompletion::DoCodeComplete(), NativeParserBase::GenerateResultSet(), and ParserThread::HandleNamespace().

◆ m_Ancestors

TokenIdxSet Token::m_Ancestors

◆ m_AncestorsString

wxString Token::m_AncestorsString

all ancestors comma-separated list, e.g.

if a class declaration like below class AAA :public BBB, public CCC then the m_AncestorsString is "BBB,CCC", note that once m_Ancestors is constructed, this variable will be cleared.

Definition at line 204 of file token.h.

Referenced by TokenTree::AddTokenToList(), ParserThread::DoParse(), ParserThread::HandleClass(), ParserThread::ReadClsNames(), and TokenTree::RecalcInheritanceChain().

◆ m_Args

wxString Token::m_Args

◆ m_BaseArgs

wxString Token::m_BaseArgs

stripped arguments e.g.

(int arg1, float arg2)

Definition at line 197 of file token.h.

Referenced by TokenTree::AddTokenToList(), DisplayName(), ParserThread::DoAddToken(), and TokenTree::TokenExists().

◆ m_BaseType

wxString Token::m_BaseType

◆ m_Children

TokenIdxSet Token::m_Children

◆ m_Descendants

TokenIdxSet Token::m_Descendants

all the descendants in the inheritance hierarchy

Definition at line 277 of file token.h.

Referenced by ClassBrowserBuilderThread::AddDescendantsOf(), ClassBrowserBuilderThread::ExpandItem(), TokenTree::RecalcInheritanceChain(), and TokenTree::RemoveToken().

◆ m_DirectAncestors

TokenIdxSet Token::m_DirectAncestors

◆ m_Doc

wxString Token::m_Doc

doxygen style comments

Definition at line 213 of file token.h.

Referenced by TokenTree::AppendDocumentation(), TokenTree::GetDocumentation(), and TokenTree::RemoveFile().

◆ m_FileIdx

unsigned int Token::m_FileIdx

◆ m_FullType

wxString Token::m_FullType

◆ m_ImplDoc

wxString Token::m_ImplDoc

doxygen style comments in the Impl file

Definition at line 228 of file token.h.

Referenced by TokenTree::AppendDocumentation(), TokenTree::GetDocumentation(), and TokenTree::RemoveFile().

◆ m_ImplFileIdx

unsigned int Token::m_ImplFileIdx

◆ m_ImplLine

unsigned int Token::m_ImplLine

◆ m_ImplLineEnd

unsigned int Token::m_ImplLineEnd

◆ m_ImplLineStart

unsigned int Token::m_ImplLineStart

◆ m_Index

int Token::m_Index

◆ m_IsAnonymous

bool Token::m_IsAnonymous

◆ m_IsConst

bool Token::m_IsConst

◆ m_IsLocal

bool Token::m_IsLocal

if true, means the token belong to a C::B project, it exists in the project's source/header files, not from the system's headers or other include header files

Definition at line 242 of file token.h.

Referenced by ClassBrowserBuilderThread::AddNodes(), ClassBrowserBuilderThread::CreateSpecialFolders(), ParserThread::DoAddToken(), ParserThread::FindTokenFromQueue(), TokenTree::MarkFileTokensAsLocal(), and ClassBrowserBuilderThread::TokenMatchesFilter().

◆ m_IsNoExcept

bool Token::m_IsNoExcept

◆ m_IsOperator

bool Token::m_IsOperator

◆ m_IsTemp

bool Token::m_IsTemp

◆ m_Line

unsigned int Token::m_Line

◆ m_Name

wxString Token::m_Name

Token's name, it can be searched in the TokenTree.

Definition at line 188 of file token.h.

Referenced by NativeParserBase::AddConstructors(), ClassBrowserBuilderThread::AddNodes(), TokenTree::AddToken(), TokenTree::AddTokenToList(), CCTreeCtrl::CBAlphabetCompare(), NativeParserBase::ComputeCallTip(), DisplayName(), CCDebugInfo::DisplayTokenInfo(), ParserThread::DoAddToken(), CodeCompletion::DoAllMethodsImpl(), CodeCompletion::DoCodeComplete(), InsertClassMethodDlgHelper::DoFillMethodsFor(), ParserThread::DoParse(), ClassBrowserBuilderThread::ExpandNamespaces(), CCDebugInfo::FillAncestors(), CCDebugInfo::FillChildren(), InsertClassMethodDlg::FillClasses(), CCDebugInfo::FillDescendants(), InsertClassMethodDlg::FillMethods(), NativeParserBase::FindAIMatches(), NativeParserBase::FindCurrentFunctionScope(), NativeParser::FindCurrentFunctionStart(), ParserThread::FindTokenFromQueue(), ClassBrowser::FoundMatch(), DocumentationHelper::GenerateHTML(), NativeParserBase::GenerateResultSet(), Tokenizer::GetMacroExpandedText(), GetNamespace(), ParserThread::HandleClass(), ParserThread::HandleFunction(), NativeParserBase::IsAllocator(), DocumentationHelper::OnDocumentationLink(), CCDebugInfo::OnFindClick(), CodeCompletion::OnGotoFunction(), ClassBrowser::OnJumpTo(), CCDebugInfo::OnSave(), ClassBrowser::OnSearch(), ClassBrowser::OnTreeItemDoubleClick(), NativeParser::ParseBufferForUsingNamespace(), NativeParser::ParseFunctionArguments(), CodeCompletion::ParseFunctionsAndFillToolbar(), NativeParser::ParseLocalBlock(), NativeParserBase::PrettyPrintToken(), ParserThread::ReadClsNames(), ParserThread::ReadVarNames(), TokenTree::RecalcFullInheritance(), TokenTree::RecalcInheritanceChain(), TokenTree::RemoveToken(), TokenTree::RenameToken(), NativeParserBase::ResolveExpression(), and CodeCompletion::UpdateEditorSyntax().

◆ m_ParentIndex

int Token::m_ParentIndex

◆ m_Scope

TokenScope Token::m_Scope

◆ m_TemplateAlias

wxString Token::m_TemplateAlias

alias for templates, e.g.

template T1 T2;

Definition at line 294 of file token.h.

Referenced by NativeParserBase::AddTemplateAlias().

◆ m_TemplateArgument

wxString Token::m_TemplateArgument

◆ m_TemplateMap

std::map<wxString, wxString> Token::m_TemplateMap

a string to string map from formal template argument to actual template argument

Definition at line 291 of file token.h.

Referenced by ParserThread::HandleFunction(), NativeParserBase::ResolveExpression(), ParserThread::ResolveTemplateArgs(), and ~Token().

◆ m_TemplateType

wxArrayString Token::m_TemplateType

for a class template, this is the formal template argument list, but for a variable Token, this is the actual template arguments.

Definition at line 288 of file token.h.

Referenced by ParserThread::HandleClass(), ParserThread::HandleTypedef(), ParserThread::ResolveTemplateArgs(), ParserThread::ResolveTemplateMap(), and ~Token().

◆ m_Ticket

size_t Token::m_Ticket
protected

This is used in class browser to avoid duplication nodes in the class browser tree.

Once a Token is allocated from the heap, a new ticket number is assigned to the Token, so a new node is only added to the class browser tree if a Token with new ticket is detected.

Definition at line 311 of file token.h.

Referenced by GetTicket().

◆ m_TokenKind

TokenKind Token::m_TokenKind

See TokenKind class.

Definition at line 234 of file token.h.

Referenced by NativeParserBase::AddChildrenOfEnum(), NativeParserBase::AddChildrenOfUnnamed(), NativeParserBase::AddConstructors(), Tokenizer::AddMacroDefinition(), ClassBrowserBuilderThread::AddMembersOf(), ClassBrowserBuilderThread::AddNodes(), NativeParserBase::AddTemplateAlias(), TokenTree::AddToken(), NativeParserBase::CleanupSearchScope(), NativeParserBase::ComputeCallTip(), ClassBrowserBuilderThread::CreateSpecialFolders(), DisplayName(), ParserThread::DoAddToken(), CodeCompletion::DoAllMethodsImpl(), CodeCompletion::DoCodeComplete(), InsertClassMethodDlgHelper::DoFillMethodsFor(), ParserThread::DoParse(), ClassBrowserBuilderThread::ExpandItem(), ClassBrowserBuilderThread::ExpandNamespaces(), InsertClassMethodDlg::FillClasses(), NativeParserBase::FindAIMatches(), NativeParserBase::FindCurrentFunctionScope(), NativeParser::FindCurrentFunctionStart(), TokenTree::FindMatches(), ParserThread::FindTokenFromQueue(), TokenTree::FindTokensInFile(), DocumentationHelper::GenerateHTML(), NativeParserBase::GenerateResultSet(), NativeParserBase::GetTokenFromCurrentLine(), NativeParser::GetTokenKindImage(), GetTokenKindString(), ParserThread::HandleClass(), ParserThread::HandleTypedef(), NativeParserBase::IsChildOfUnnamedOrEnum(), CodeCompletion::OnGotoDeclaration(), CodeCompletion::OnGotoFunction(), CCDebugInfo::OnSave(), ClassBrowser::OnSearch(), ClassBrowser::OnTreeItemDoubleClick(), CodeRefactoring::Parse(), CodeCompletion::ParseFunctionsAndFillToolbar(), NativeParser::ParseLocalBlock(), NativeParserBase::PrettyPrintToken(), TokenTree::RecalcFullInheritance(), TokenTree::RecalcInheritanceChain(), NativeParserBase::RemoveLastFunctionChildren(), NativeParserBase::ResolveExpression(), NativeParserBase::ResolveOperator(), ParserThread::ResolveTemplateMap(), ClassBrowser::ShowMenu(), ClassBrowserBuilderThread::TokenContainsChildrenOfKind(), TokenTree::TokenExists(), and CodeCompletion::UpdateEditorSyntax().

◆ m_TokenTree

TokenTree* Token::m_TokenTree
protected

◆ m_UserData

void* Token::m_UserData

custom user-data (the classbrowser expects it to be a pointer to a cbProject), this field is used when user can only show the tokens belong to the current C::B project in the browser tree.

m_UserData is updated in the worker threaded task: MarkFileAsLocalThreadedTask

Definition at line 300 of file token.h.

Referenced by TokenTree::MarkFileTokensAsLocal(), and ClassBrowserBuilderThread::TokenMatchesFilter().

◆ start

int Token::start

◆ type

Type Token::type

Definition at line 66 of file parsewatchvalue.cpp.

Referenced by GetNextToken(), operator==(), and ParseGDBWatchValue().


The documentation for this struct was generated from the following files: