'SPARK SQL create table does not show / read all columns as expected

I am trying to create table in spark sql by providing the schema and giving the location.

However when i run select on the table, i see only half the columns. ( all alternate ones )

Approach 1 :

CREATE TABLE IF NOT EXISTS db1.tbl1
(
  col1 STRING,
  col2 STRING,
  col3 STRING,
  col4 STRING,
  col5 STRING,
  col6 STRING,
)
STORED AS PARQUET
LOCATION 's3://dir1/dir2/dir3/'

df_1 = spark.sql("select * from db1.tbl1")
df_1.printSchema()

Output -

root
 |-- col1: string (nullable = true)
 |-- col3: string (nullable = true)
 |-- col5: string (nullable = true)

Approach 2 : Now, IF i use

df2 = spark.read.parquet("s3://dir1/dir2/dir3/")
df2.printSchema()

Output -

root
 |-- col1: string (nullable = true)
 |-- col2: string (nullable = true)
 |-- col3: string (nullable = true)
 |-- col4: string (nullable = true)
 |-- col5: string (nullable = true)
 |-- col6: string (nullable = true)

Request :

We use approach 1.

how do I fix the issue so that correct schema is reflected using approach 1



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source