'sparkR sql() returns string

We have parquet data saved on a server and I am trying to use SparkR sql() function in the following ways

df <- sql("SELECT * FROM parquet.`<path to parquet file`")
head(df)
show(df) # returns "<SQL> SELECT * FROM parquet.`<path to parquet file`"

and

createOrReplaceTempView(df, "table")
df2 <- sql("SELECT * FROM table")
show(df2) # returns "<SQL> SELECT * FROM table"

In both cases what I get is the sql query in a string format instead of the spark dataframe. Does anyone have any idea why this happens and why I don't get the dataframe?

Thank you

r parquet sparkr

Solution 1:^[1]

Don't use the show statement use showDF()...or, View(head(df2, num=20L))

Solution 2:^[2]

This was a very silly problem. The solution is to just use the full name of the method

SparkR::sql(...)

instead of the short name. Apparently the function sql is masked.

Solution 3:^[3]

show method documentation in sparkR states that if eager evaluation is not enabled it returns the class and type information of the Spark object. so you should use showDF instead.

besides that, apparently, the sql method is masked in R and you should call it with explicit package deceleration. like this:

df <- SparkR::sql("select 1")
showDF(df)

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	nate
Solution 2	Ashkan
Solution 3	Javad Bahoosh

'sparkR sql() returns string

Solution 1:[1]

Solution 2:[2]

Solution 3:[3]

Sources

Related Questions

Solution 1:^[1]

Solution 2:^[2]

Solution 3:^[3]