Code in Python,Pyspark,AND output of Both -- 2

You are constantly reading files with this format and you must persist the records in parquet format. But we don’t want to duplicate records. If one file is duplicated and both are been read, your python code should avoid the duplication. Duplicate records would be something with the same id and same ts even if data field is different. We may have a lot of records with the same id but all the ts must be unique. Please build a ETL python program that will read all the json files in a filesystem path an create parquet files that will avoid duplication.