Compiler Design Project -- 2

Consider the following extended BNF grammar for a subset of the Python programming language ([login to view URL]), called TinyPython. program ::= from typing import List { function-def } main-def main-def ::= def main ( ) : block function-def ::= def function-identifier ( [ param { , param } ] ) -> type-expression : [block] return expression param ::= variable-identifier : type-expression type-expression ::= bool | int | List [ type-expression ] block ::= statement { statement } statement ::= compound-stmt | simple-stmt simple-stmt ::= assignment | print ( expression ) | pass assignment ::= variable-identifier = array-definition | variable = int ( input ( ) ) | variable = expression array-definition ::= dimension-definition | [ array-definition for in dimension-definition ] dimension-definition ::= [ integer ] ∗ integer compound-stmt ::= if expression : nested-block ; [else : nested-block ;] | while expression : nested-block ; nested-block ::= [block] simple-stmt expression ::= conjunction { or conjunction } conjunction ::= inversion { and inversion } inversion ::= comparison | not inversion comparison ::= sum comp-operator sum comp-operator ::= < | > | == | >= | <= | ! = sum ::= term { add-operator term } add-operator ::= + | − term ::= factor { mult-operator factor } mult-operator ::= ∗ | // factor ::= [add-operator] primary primary ::= variable | True | False | integer | ( expression ) | function-identifier ( [ expression { , expression } ] ) variable ::= variable-identifier { [ sum ] } Syntactic and Semantic Conventions The keywords and the token symbols in TinyPython are in bold. Note that TinyPython has symbols [, and ] which are distinguished from grammar metasymbols [, and ], respectively, by underlining. Assume that an id can only contain letters (only alphabetic characters), digits, and underscores ( ) with the restrictions that it must begin with a letter, cannot end with an underscore and cannot have two consecutive underscores. For example, give 2 Joe, tell me and A45Asm3 are valid identifiers, but 6gh, two bad, and no end are not. integer is an unsigned integer. function-identifiers and variable-identifiers are the same lexical/syntactic items as identifiers but have the semantics given by the appropriate qualifier. Comments are preceded by # and are terminated by the end of the line. Your program is as follows: 1. Determine the set of tokens which a lexical analyzer would need to recognize. 2. Design and implement a lexical analyzer procedure to read a source program in the above language and print the next token in the input stream. If the token detected is a valueless token, such as a keyword, then it is sufficient to print only the keyword. If it has a value, then both the token type and lexeme should be printed. 3. You will be given several TinyPython programs with which to test your lexical analyzer. These will be located on the class WWW page and will be of the form [login to view URL], [login to view URL], etc. Run your lexical analyzer on each test program. Suggestion: Use the lex, flex, or JFlex tool to automatically construct a lexical analyzer for TinyPython from a set of regular expressions specifying tokens.