Type alias Tokenizer

Tokenizer: ((text: string) => string[])

Type declaration

- (text: string): string[]
- The tokenizer is a function that splits a text into its tokens.
  Parameters
  - text: string
  Returns string[]