StrTok Class Reference

#include <evo/strtok.h>

Inheritance diagram for StrTok:

Detailed Description

String forward tokenizer.

• Variants: StrTokWord
• This skips whitespace (spaces, tabs) between delimiters so tokens will not start or end with any whitespace
• This references target string data – results are undefined if target string is modified while referenced
• For reverse tokenizing see: StrTokR
• For "strict" tokenizing (without skipping whitespace) see: StrTokS, StrTokRS
• Note: All tokenizers (including forward and reverse) implement the same basic interface (excluding next*() variants)

Example:

#include <evo/strtok.h>
#include <evo/io.h>
using namespace evo;

int main() {
    Console& c = con();
    SubString str = "one, two, three";

    // Tokens:
    //  one
    //  two
    //  three
    StrTok tok(str);
    while (tok.next(','))
        c.out << tok.value();

    return 0;
}

Public Types
typedef StrTokBase	BaseType
	Base type. More...
typedef SubString::Size	Size
	String size type More...
typedef StrTok	ThisType
	This type. More...

Public Member Functions
	StrTok ()
	Default constructor creates empty tokenizer. More...
	StrTok (const ThisType &src)
	Copy constructor. More...
	StrTok (const BaseType &src)
	Copy constructor. More...
	StrTok (const StringBase &str)
	Constructor to start tokenizing given string. More...
Char	delim () const
	Get current delimiter before next token. More...
Size	index () const
	Get current index before next token. More...
bool	next (char delim)
	Find next token using delimiter. More...
bool	nextany (const StringBase &delims)
	Find next token using any of given delimiters. More...
bool	nextanyq (const StringBase &delims, char ws_delim)
	Find next token using any of given delimiters with quoting support. More...
bool	nextanyq (const StringBase &delims)
	Find next token using any of given delimiters with quoting support. More...
bool	nextq (char delim)
	Find next token using delimiter with quoting support. More...
bool	nextw (char delim)
	Find next token using word delimiter. More...
ThisType &	operator= (const ThisType &src)
	Assignment/Copy operator. More...
ThisType &	operator= (const BaseType &src)
	Assignment/Copy operator. More...
ThisType &	operator= (const StringBase &str)
	Assignment operator to start tokenizing given string from beginning. More...
ThisType &	reset ()
	Reset to tokenize from beginning of string. More...
Size	skipws ()
	Advance current position for next token by skipping whitespace. More...
const SubString &	value () const
	Get current token value from last call to next(). More...

Static Public Member Functions
template<class C , class T >
static C::Size	split (C &items, const T &str, char delim=',')
	Split delimited string into item list using next(). More...
template<class T >
static SubString	splitat (const T &str, Size index, char delim=',')
	Split delimited string to extract token at index. More...

Protected Member Functions
void	copy (const BaseType &src)
	Copy data. More...

Protected Attributes
Char	delim_
	Current delimiter, null when none or at end. More...
Size	index_
	Current index, END when at end. More...
SubString	string_
	String being tokenized, NULL for none More...
SubString	value_
	Current value. More...

Member Typedef Documentation

BaseType

typedef StrTokBase BaseType

Base type.

Size

typedef SubString::Size Size

String size type

ThisType

typedef StrTok ThisType

This type.

Constructor & Destructor Documentation

StrTok() [1/4]

StrTok ( )

inline

Default constructor creates empty tokenizer.

StrTok() [2/4]

StrTok ( const ThisType &  src )

inline

Copy constructor.

Parameters

src	Data to copy

StrTok() [3/4]

StrTok ( const BaseType &  src )

inline

Copy constructor.

Parameters

src	Data to copy

StrTok() [4/4]

StrTok ( const StringBase &  str )

inline

Constructor to start tokenizing given string.

• Call next() or nextw() for each token

Parameters

str	String to tokenize

Member Function Documentation

copy()

void copy ( const BaseType &  src )

inlineprotectedinherited

Copy data.

delim()

Char delim ( ) const

inlineinherited

Get current delimiter before next token.

Returns

Current delimiter, null if none or at end

index()

Size index ( ) const

inlineinherited

Get current index before next token.

Returns

Current index, END if at end

next()

bool next ( char  delim )

inline

Find next token using delimiter.

Call value() to get token value.

• This will skip leading whitespace (spaces, tabs) before and after next token

Parameters

delim

Delimiter to use

Returns

Whether next token was found, false if no more

nextany()

bool nextany ( const StringBase &  delims )

inline

Find next token using any of given delimiters.

Call value() to get token value.

• This will skip leading whitespace (spaces, tabs) before and after next token

Parameters

delims

Delimiters to use

Returns

Whether next token was found, false if no more

nextanyq() [1/2]

bool nextanyq ( const StringBase &  delims, char  ws_delim  )

inline

Find next token using any of given delimiters with quoting support.

Call value() to get token value.

• This will skip leading whitespace (spaces, tabs) before and after next token
• Token may be single-quoted ( ' ), double-quoted ( " ), backtick-quoted ( ` ), or triple-quoted ( ''' or """ or ``` )
  • This also supports backtick-DEL quoting – backtick followed by the DEL char (7F) – used when no other quoting is possible
• Token is only considered quoted if it begins and ends with given quotes, after excluding whitespace – so an unquoted token can contain quote chars
• For best performance set ws_delim to 0 or the whitespace delimiter to skip whitespace delimiter detection

Parameters

delims	Delimiters to use – must not have more than 1 whitespace character (space, tab, newline)
ws_delim	Use to specify a whitespace char in delims so it's handled correctly, 1 to auto-detect, 0 if no whitespace delim

Returns

Whether next token was found, false if no more

nextanyq() [2/2]

bool nextanyq ( const StringBase &  delims )

inline

Find next token using any of given delimiters with quoting support.

Call value() to get token value.

• This will skip leading whitespace (spaces, tabs) before and after next token
• Token may be single-quoted ( ' ), double-quoted ( " ), backtick-quoted ( ` ), or triple-quoted ( ''' or """ or ``` )
  • This also supports backtick-DEL quoting – backtick followed by the DEL char (7F) – used when no other quoting is possible
• Token is only considered quoted if it begins and ends with given quotes, after excluding whitespace – so an unquoted token can contain quote chars
• For best performance use nextanyq(const StringBase&,char) and set ws_delim to 0 or the whitespace delimiter to skip whitespace delimiter detection

Parameters

delims

Delimiters to use – must not have more than 1 whitespace character (space, tab, newline)

Returns

Whether next token was found, false if no more

nextq()

bool nextq ( char  delim )

inline

Find next token using delimiter with quoting support.

Call value() to get token value.

• This will skip leading whitespace (spaces, tabs) before and after next token
• Token may be single-quoted ( ' ), double-quoted ( " ), backtick-quoted ( ` ), or triple-quoted ( ''' or """ or ``` )
  • This also supports backtick-DEL quoting – backtick followed by the DEL char (7F) – used when no other quoting is possible
• Token is only considered quoted if it begins and ends with given quotes, after excluding whitespace – so an unquoted token can contain quote chars
• See Smart Quoting

Parameters

delim

Delimiter to use

Returns

Whether next token was found, false if no more

nextw()

bool nextw ( char  delim )

inline

Find next token using word delimiter.

Call value() to get token value.

• Same as next(char) except this will also skip any leading extra delimiters before next token

Parameters

delim

Delimiter to use

Returns

Whether next token was found, false if no more

operator=() [1/3]

ThisType& operator= ( const ThisType &  src )

inline

Assignment/Copy operator.

Parameters

src	Data to copy

Returns

This

operator=() [2/3]

ThisType& operator= ( const BaseType &  src )

inline

Assignment/Copy operator.

Parameters

src	Data to copy

Returns

This

operator=() [3/3]

ThisType& operator= ( const StringBase &  str )

inline

Assignment operator to start tokenizing given string from beginning.

Call next() or nextw() for each token.

Parameters

str	String to tokenize

Returns

This

reset()

ThisType& reset ( )

inline

Reset to tokenize from beginning of string.

Returns

This

skipws()

Size skipws ( )

inline

Advance current position for next token by skipping whitespace.

• This gives the index where the next token starts
• This is useful for advanced parsing that needs to look at the context of next token (indent amount, characters before token, etc)

Returns

Current index for next token

split()

static C::Size split ( C &  items, const T &  str, char  delim = ','  )

inlinestatic

Split delimited string into item list using next().

• This tokenizes and adds each item to list, using convert() for conversion to list item type
• String must be convertible to list item type via convert()
• See String::join() to join list back into string

Template Parameters

C	List container for items – inferred from items parameter
T	String type to split – inferred from str parameter

Parameters

items	List to add items to [in/out]
str	String to tokenize
delim	Delimiter to use

splitat()

static SubString splitat ( const T &  str, Size  index, char  delim = ','  )

inlinestatic

Split delimited string to extract token at index.

• This will tokenize until token at index is found

Template Parameters

T	String type to tokenize – inferred from str parameter

Parameters

str	String to tokenize
index	Token index to extract
delim	Delimiter to use

Returns

Result token, set to null if not found

value()

const SubString& value ( ) const

inlineinherited

Get current token value from last call to next().

Returns

Current token value

Member Data Documentation

delim_

Char delim_

protectedinherited

Current delimiter, null when none or at end.

index_

Size index_

protectedinherited

Current index, END when at end.

string_

SubString string_

protectedinherited

String being tokenized, NULL for none

value_

SubString value_

protectedinherited

Current value.

---

The documentation for this class was generated from the following file:

• evo/strtok.h