'SPARK SQL create table does not show / read all columns as expected
I am trying to create table in spark sql by providing the schema and giving the location.
However when i run select on the table, i see only half the columns. ( all alternate ones )
Approach 1 :
CREATE TABLE IF NOT EXISTS db1.tbl1
(
col1 STRING,
col2 STRING,
col3 STRING,
col4 STRING,
col5 STRING,
col6 STRING,
)
STORED AS PARQUET
LOCATION 's3://dir1/dir2/dir3/'
df_1 = spark.sql("select * from db1.tbl1")
df_1.printSchema()
Output -
root
|-- col1: string (nullable = true)
|-- col3: string (nullable = true)
|-- col5: string (nullable = true)
Approach 2 : Now, IF i use
df2 = spark.read.parquet("s3://dir1/dir2/dir3/")
df2.printSchema()
Output -
root
|-- col1: string (nullable = true)
|-- col2: string (nullable = true)
|-- col3: string (nullable = true)
|-- col4: string (nullable = true)
|-- col5: string (nullable = true)
|-- col6: string (nullable = true)
Request :
We use approach 1.
how do I fix the issue so that correct schema is reflected using approach 1
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|