import sre def s_ident(scanner, token): return token def s_operator(scanner, token): return "op%s" % token def s_float(scanner, token): return float(token) def s_int(scanner, token): return int(token) scanner = sre.Scanner([ (r"[a-zA-Z_]\w*", s_ident), (r"\d+\.\d*", s_float), (r"\d+", s_int), (r"=|\+|-|\*|/", s_operator), (r"\s+", None), ]) print scanner.scan("sum = 3*foo + 312.50 + bar") First yo

This note discusses how to use the re module in Python 2.0 and later for lexical analysis. A Simple RE ScannerA scanner reads an input string (e.g. program code) and groups the characters into lexical units, such as keywords and integer literals. Scanners are also known as lexical analysers, or tokenizers. For example, here’s a simple expression: b = 2 + a*10In the above expression, we can identif