2. Lexical analysis2. Lexical analysis¶ A Python program is read by a parser. Input to the parser is a stream of tokens, generated by the lexical analyzer (also known as the tokenizer). This chapter describes how the lexical analyzer produces these tokens. The lexical analyzer determines the program text’s encoding (UTF-8 by default), and decodes the text into source characters. If the text cannot be decoded, a Synt
B. Reserved Words
