Splits a given text into smaller units called tokens. More...
Inheritance diagram for Tokeniser:
Public Member Functions | |
| tokenise (&$text) | |
| Tokenises the given text, i.e., splits the text into smaller units called tokens. | |
Detailed Description
Splits a given text into smaller units called tokens.
Definition at line 43 of file Tokeniser.php.
Member Function Documentation
| Tokeniser::tokenise | ( | &$ | text | ) |
Tokenises the given text, i.e., splits the text into smaller units called tokens.
- Parameters:
-
text The given text.
- Returns:
- Array of text tokens.
- Precondition:
- The input text should be a valid string.
- Postcondition:
- The given text is tokenised.
Implemented in WhitespaceTok.
The documentation for this interface was generated from the following file:
