Tokenization

Also known as · tokenize

Splitting a sentence into a list of tokens, usually words, to create countable, discrete units.

When to use

The third step of the Text Preprocessing pipeline, before stopword removal (the lecture's example: "the hummus was absolutely amazing and fresh" gives 7 tokens). Simple tokenizing splits on spaces; better tokenizers also handle contractions (don't becomes do + not), hyphens and special characters. Also in PS 06.

See DS Lec 7.

Appears in

No references yet.