Quotation causing error in ETL with terraform

Question

I'm trying to convert my data from csv to parquet while performing cleansing. I keep getting an error when moving my data from one bucket to the next! I figured out it is to do with one of my columns containing quotes in them for example it could be

we"l12

I want to change my terraform file so it allows quotes in the strings coming from an s3 bucket through aws glue and into another s3 bucket. Is there anyway of adding a none quote symbol? or other ways around this?

Answer 1

I've figured out that in your terraform file for the glue table you can set your parameter for your quoteChar to = ^B which is rarely ever used in data, this will prevent your quotes from being read. Hope this helps someone

storage_descriptor {

parameters{
quoteChar = "\u0002"
          }
}