'Pyspark create schema for maptype with different value types
I need to give the correct schema to an rdd I have, but struggling with a maptype that has different valuetypes. I guess the problem is that one specific Key has a string as it's value and the others have again a dictionary as value:
{row_id -> "0", Kind -> {right -> "32", left -> "8", top -> "62", bottom -> "61"}, Tab -> {right -> "32", left -> "8", top -> "62", bottom -> "61"},...},{row_id -> "1", Kind -> {right ->...}...},..
When trying to define the schema like MapType(StringType(),MapType(StringType(),StringType()))
I get a TypeError. Propably because value of row_id does not hold a dictionary.
Any ideas how to correctly define the schema?
Best!
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|