Trying to exclude specific files from AWS S3 Lifecycle Configuration

Question

We need to implement an expiration of X days of all customer data due to contractual obligations. Not too big of a deal, that's about as easy as it gets.

But at the same time, some customers' projects have files with metadata. Perhaps dataset definitions which most definitely DO NOT need to go away. We have free reign to tag or manipulate any of the data in any way we see fit. Since we have 500+ S3 buckets, we need a somewhat global solution.

Ideally, we would simply set an expiration on the bucket and another rule for the metadata/ prefix. Except then we have a rule overlap and metadata/* files will still get the X day expiration that's been applied to the entire bucket.

We can forcefully tag all objects NOT in metadata/* with something like allow_expiration = true using Lambda. While not out of the question, I would like to implement something a little more built-in with S3.

I don't think there's a way to implement what I'm after without using some kind of tagging and external script. Thoughts?

Answer 1

If you've got a free hand on tagging the object, you could use both prefix and / or a tag filter with S3 lifecycle.

You can filter objects by key prefix, object tags, or a combination of both (in which case Amazon S3 uses a logical AND to combine the filters).

See Lifecycle Filter Rules

You could automate the creation and management of your lifecycle rules with IaC, for example, terraform.

See S3 Bucket Lifecycle Configuration with Terraform

There's a useful blog on how to manage these dynamically here.

What's more, using tags has a number of additional benefits:

• Object tags enable fine-grained access control of permissions. For example, you could grant an IAM user permissions to read-only objects with specific tags.

• Object tags enable fine-grained object lifecycle management in which you can specify a tag-based filter, in addition to a key name prefix, in a lifecycle rule.

• When using Amazon S3 analytics, you can configure filters to group objects together for analysis by object tags, by key name prefix, or by both prefix and tags.

• You can also customize Amazon CloudWatch metrics to display information by specific tag filters.

Source and more on how to set tags to multiple Amazon S3 object with a single request.