We want to build a tokenizer for simple expressions such as xpr = "res = 3 + x_sum*11" : We want to build a tokenizer for simple expressions such as xpr = "res = 3 + x_sum*11". Such expressions comprise only three tokens, as follows:  (1) Integer literals: one or more digits e.g., 3 11;  (2) Identifiers: strings starting with a letter or an underscore and followed by more letters, digits, or underscores e.g., res x_sum;  (3) Operators: = + *  . Leading or trailing whitespace characters should be skipped.

(c)  To find which token each lexeme is associated with, we only need to find the first non-empty item in each tuple. Write a tokenize  generator (using re.findall and map) that returns all pairs (tuples) of lexemes and tokens. The output of list(tokenize(xpr)) should thus be:

  [('res', 'id'), ('=', 'op'), ('3', 'int'), ('+', 'op'), ('x_sum', 'id'), ('*', 'op'), ('11', 'int')]