how to read s3 files from apache beam python?

Question

I am using Apache Beam python SDK to read s3 file data.

code I am using

    ip = (pipe
          | beam.io.ReadFromText("s3://bucket_name/file_path")
          | beam.Map(lambda x: x.split(","))
          | beam.Map(print)

          )

but it gives a error

 S3ClientError('Unable to locate credentials', None)}")}

Anybody have any idea how to give credentials to s3 client thorugh apache beam python file

Answer 1

You need to ensure the credentials are available on your worker machines. One way to do this is installing them in a custom container.