'How can i find which column is bad when a spark cast fails?
I have a spark job that is failing, saying one column has the wrong type. I'm working with tables that have thousands of columns, and the error message doesn't tell me which column is the issue. The stack trace is unhelpful as well, as the only frames in the stack trace contain generated code, all the way down to something called a 'ShuffleWriteProcessor'.
So i can't figure out how to, say, catch the exception in a debugger, and the log message doesn't give me enough details to proceed. Any ideas on how to debug this situation?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|