How to avoid "Objects have changed outside of Terraform"?

Question

Recently upgraded my Terraform project to AWS provider 3.74.0 and TF 1.1.4 (from much older versions).

I'm suddenly getting this autoscaling schedule reporting external changes:

resource "aws_autoscaling_schedule" "api-svc-tst-down-schedule" {
  scheduled_action_name = "api-svc-tst-down-schedule"
  min_size              = 0
  max_size              = 1
  desired_capacity      = 0

  // Minute Hour DayOfMonth Month DayOfWeek
  recurrence             = "0 13 * * *" 
  autoscaling_group_name = aws_autoscaling_group.api-svc-tst-asg.name
  lifecycle {
    ignore_changes = [start_time]
  }
}

The plan command is now reporting:

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply":

  # aws_autoscaling_schedule.api-svc-tst-down-schedule has changed
  ~ resource "aws_autoscaling_schedule" "api-svc-tst-down-schedule" {
        id                     = "api-svc-tst-down-schedule"
      ~ start_time             = "2022-01-31T13:00:00Z" -> "2022-02-01T13:00:00Z"
        # (7 unchanged attributes hidden)
    }

If I apply the plan, it doesn't appear that TF changes the ASG (I'm assuming it just updates its state file) and the notification goes away until the next day.

I note that the AWS console does show that the Scheduled action has a Start time, which seems to be being set by AWS.

I tried adding start_time to ignored_changes but it didn't seem to make a difference, still reported as externally changed.

Is this a known issue with Terraform (I'm not seeing anything via googling)?
How can I prevent TF from being marked as externally changed?

Edit: I also tried setting the start_time attribute as suggested in the comments. But the detected changes warning came back the next day.

Edit 2: I also tried deleting and re-adding the resource via Terraform, but it still gets marked as changed the next day.

Answer 1

This undesirable behavior was an intentional change introduced in Terraform version 0.15.4.

It cannot currently be avoided. The only workaround is that all team members (and tooling) must be educated to ignore "expected drift".

Note that this "expected drift" behavior is not limited to just aws_autoscaling_schedule resources, or even just the AWS provider. The issue happens on many different platforms/types for any resource where the cloud vendor updates the attribute after the resource is created.

Many resources will report drift immediately after being created - often you can get rid of the report by immediately doing an apply or refresh to update the TF state and as long as AWS doesn't make changes to those attributes, you won't see the resource reported as changed again.

Other resource attributes (like aws_autoscaling_schedule.start_time) get updated by the cloud vendor regularly. These types of resources will intermittently report "Objects have changed outside of Terraform", whenever you run plan.

There is a locked open issue to track: https://github.com/hashicorp/terraform/issues/28803.

Note that the issue is locked because Hashicorp got tired of people telling them how negatively this affects their teams.

Answer 2

In 1.2 version (not released yet) they promised that it shouldn't be that bad anymore https://github.com/hashicorp/terraform/issues/28803#issuecomment-1072740861

Starting with v1.2, the goal for the refresh report is that only external changes which may have contributed to changes in the plan will be shown. This means in most cases, unused attributes changing outside of terraform will not show up in the normal plan output. If there are no changes in the plan, no external changes will be shown in the CLI at all. All refresh information is still stored within the plan, and if a user wants to see all external changes of resources in the CLI, a refresh-only plan can be used.

P.S. Btw, this stuff could be excluded from output with sed, example (macOS tested):

terraform plan | sed -n '/Objects have changed outside of Terraform/,/?????????????????????????????????????????????????????????????????????????????/!p