'Pyspark to parse multiples dictionaries
Imagine I have the following list of dicts in python: list = [dict1, dict2, dict3] I want to parse these dicts and transform them to a dataframe and save it as parquet. The dicts have different patterns and I have a specific function to parse it and generate the dataframes.
How can I achieve this using pyspark? Right now I am using pure python and it is working just fine. But I need to scale it.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|