![]() |
![]() |
Cheshire3 Objects: Tokenizer |
A Tokenizer takes a string of language and processes it to produce an ordered list of tokens.
The following implementations are pre-configured and ready to use.
They may be used out of the box in configurations for Indexes, Workflows etc.
Module: cheshire3.tokenizer
Classes:
Methods:
Function | Parameters | Returns | Description |
---|---|---|---|
__init__ | config, parent | ||
process_string | session, data | list | Process a raw string to produce an ordered list of tokens |
process_hash | session, data | list | Process the text of each item in the hash to produce an ordered list of tokens for that hash entry. |