Tokenization
Also known as · tokenize
Splitting a sentence into a list of tokens, usually words, to create countable, discrete units.
When to use
The third step of the Text Preprocessing pipeline, before stopword removal (the lecture's example: "the hummus was absolutely amazing and fresh" gives 7 tokens). Simple tokenizing splits on spaces; better tokenizers also handle contractions (don't becomes do + not), hyphens and special characters. Also in PS 06.
See DS Lec 7.