Update DeltaTable properties on S3

Question

I have a DeltaTable at aws S3 location (s3://bucket/myDeltaTable) which has a default table property delta.logRetentionDuration set to 30 days.

Is there a way I can set this property to 3 days without deleting & recreating table?

I tried below command but it did not work:

spark.sql("ALTER TABLE delta.`s3://bucket/myDeltaTable` SET TBLPROPERTIES(delta.logRetentionDuration = \"2 days\")")

The error I got was:

org.apache.spark.sql.AnalysisException: Table not found: delta.s3://bucket/myDeltaTable

Does anyone have any idea on altering the metadata of existing delta table stored in s3?

Answer 1

try to add those settings to your SparkConf:

"spark.sql.extensions" = "io.delta.sql.DeltaSparkSessionExtension"

"spark.sql.catalog.spark_catalog" = "org.apache.spark.sql.delta.catalog.DeltaCatalog"