tokens
Dict of ``{'state': [(regex, tokentype, new_state), ...], ...}``
The initial state is 'root'. ``new_state`` can be omitted to signify
no state transition. If it is a string, the state is pushed on the stack
and changed. If it is a tuple of strings, all states are pushed on the
stack and the current state will be the topmost. It can also be
``combined('state1', 'state2', ...)`` to signify a new, anonymous state
combined from the rules of two or more existing ones. Furthermore, it can
be '#pop' to signify going back one step in the state stack, or '#push'
to push the current state on the stack again.
The tuple can also be replaced with ``include('state')``, in which
case the rules from the state named by the string are included in the
current one.
- Value:
{ ' comments ' : [ ( ' (\\s*#.*) ' , Token.Comment) ] ,
' filterExp ' : [ 'variable',
'object',
( ' \\s*[+*/<>=~!%&|-]+\\s* ' , Token.Operator) ,
( ' \\s*\\) ' , Token.Text, ' #pop ' ) ] ,
' graph ' : [ ( ' \\s*(<[^>]*\\>) ' ,
Token.Name.Class,
( ' triple ' , ' predObj ' ) ) ,
...
|
|