Calculate the size of all files in a bucket S3

Question

I want to calculate the size of all files that are in a S3 bucket in python and this is the code I tried so far:

import boto3

s3_client = boto3.client('s3')
bucket = 'bucket-name'
prefix = 'path/to/folder'

len=0
response = s3_client.list_objects(Bucket = bucket,Prefix = prefix)
for file in response['Contents']:
    name = file['Key'].rsplit('/', 1)
    len+=name['ContentLength']

I'm not sure how to get the size of the file : name['ContentLength'] Any ideas?

Answer 1

Use file['Size'] instead. If using list_objects method, you have to check the value of response['IsTruncated'] as the response will contain a maximum of 1000 objects. If IsTruncated is True, use response['NextMarker'] as the Prefix to list the remaining objects in the bucket.

Or, you can use the Bucket class

s3 = boto3.resource('s3')
bucket = s3.Bucket('bucket-name')
total_size = 0
for k in bucket.objects.all():
    total_size += k.size