Function DEFAULT_TOKENIZER

  • The tokenizer is a function that splits a text into its tokens.

    Parameters

    • text: string

    Returns string[]