newrelic_entity_tags is deleting "default" tags on every plan/apply

[trunet]

Terraform Version

Terraform v0.13.0-beta3

Affected Resource(s)

• data.newrelic_entity_tags
• resource.newrelic_entity_tags

Terraform Configuration Files

data "newrelic_entity" "ssl" {
  name   = var.name
  domain = "SYNTH"

  depends_on = [newrelic_synthetics_monitor.ssl]
}

resource "newrelic_entity_tags" "ssl" {
  guid = data.newrelic_entity.ssl.guid

  dynamic "tag" {
    for_each = merge(var.tags, local.common_tags)
    content {
      key    = tag.key
      values = list(tag.value)
    }
  }
}

Expected Behavior

After first apply, it shouldn't on subsequent runs to delete tags.

Actual Behavior

On every terraform plan/apply it asks to delete some tags. Looks like these tags are re-created automatically by newrelic.

Steps to Reproduce

1. terraform apply
2. sleep 300 # sometimes these tags takes some time to be recreated by newrelic
3. terraform apply

[trunet]

I redacted below output to show which tags are being deleted in our case.

  # module.syn-landing-ssl["landing_expired"].newrelic_entity_tags.ssl will be updated in-place
  ~ resource "newrelic_entity_tags" "ssl" {
        guid = "[GUID]"
        id   = "[GUID]"

        tag {
            key    = "account_number"
            values = [
                "[SOMENUMBERS]",
            ]
        }
        tag {
            key    = "contact"
            values = [
                "[CONTACT]",
            ]
        }
        tag {
            key    = "cost_code"
            values = [
                "[SOMENUMBERS]",
            ]
        }
        tag {
            key    = "description"
            values = [
                "[DESCRIPTION]",
            ]
        }
        tag {
            key    = "environment"
            values = [
                "[ENVIRONMENT]",
            ]
        }
        tag {
            key    = "module"
            values = [
                "[MODULE]",
            ]
        }
      - tag {
          - key    = "monitorStatus" -> null
          - values = [
              - "Muted",
            ] -> null
        }
      - tag {
          - key    = "monitorType" -> null
          - values = [
              - "Scripted API",
            ] -> null
        }
        tag {
            key    = "orchestration"
            values = [
                "[URL]",
            ]
        }
      - tag {
          - key    = "period" -> null
          - values = [
              - "1 day",
            ] -> null
        }
        tag {
            key    = "product"
            values = [
                "[STRING]",
            ]
        }
      - tag {
          - key    = "publicLocation" -> null
          - values = [
              - "Frankfurt, DE",
              - "Washington, DC, USA",
            ] -> null
        }
        tag {
            key    = "repository"
            values = [
                "[URL]",
            ]
        }
        tag {
            key    = "sub_product"
            values = [
                "[PRODUCT]",
            ]
        }
    }

[MAN98]

I have seen that as well and those are OOB tags that just gets tagged as to be destroyed but not sure if TF APPLY can actually delete those tags.

[sanderblue]

@trunet Hi there! Thanks for reporting this. Oddly enough I was only able to reproduce this the first time I tried, but I haven't been able to reproduce it again after many attempts. Is this an intermittent issue or can this be reproduced every time for you?

Here is the config I'm using:

provider "newrelic" {
  version = "2.2.1"
}

locals {
  tags = {
    tagKey = "tagvalue",
  }

  common = {
    commontag = "commontagvalue",
  }
}

data "newrelic_entity" "ssl" {
  name   = "Dummy App"
  domain = "APM"
}

resource "newrelic_entity_tags" "ssl" {
  guid = data.newrelic_entity.ssl.guid

  dynamic "tag" {
    for_each = merge(local.tags, local.common)
    content {
      key    = tag.key
      values = list(tag.value)
    }
  }
}

[trunet]

@sanderblue for me, it changes almost every time. My guess is that the tags are re-added when the actual synthetics monitoring runs. like, if you're running SSL check once a day, it will be re-added every day. My more frequent monitoring I can see it changing often.

[zlesnr]

I'm trying to find someone to tell me how we can identify user-added tags, which would allow us to constrain the view that TF has. Otherwise, another suggestion for a fix here is to ignore everything that we've not defined in manifest, which would avoid the constant attempt at removing tags that the synthetics service keeps adding. I'm trying to think of unintended consequences to the later solution, but haven't come up with any yet.

[MAN98]

@zlesnr - In the New Relic One UI's Entity Exporer, we can only remove user-added tags when the "Remove Tags" option is turned-on so I would assume there would be metadata to entity tags that separates/distinguishes the system vs user-added:

[zlesnr]

Oh neat. I have the same assumption, but I need to figure out the graphql mechanics to make it work.

I'm issuing the following query, which contains no information about which of these were added by the user.

https://api.newrelic.com/graphiql

{
  actor {
    entity(guid: "<GUID_GOES_HERE>") {
      tags {
        __typename
        key
        values
      }
    }
  }
}

[MAN98]

@zlesnr - you are correct. I ran the same query for our account against one of our synthetic and I see no information to say which ones were system vs user-added.

[zlesnr]

Ah, I've learned that part of the schema is hidden, and there are details about the tag metadata there. I'm chasing it down to see if we might move that portion of the schema into the public so that TF could learn which tags are mutable.

[zlesnr]

Discussions must take place. I should know more tomorrow. 🍿

[zlesnr]

I'm hearing soonish before the schema will be public.

[MAN98]

@zlesnr - Any updates on these "default" tags?

[zlesnr]

Looks like that work is happening now and is currently in review.

[twlandmann]

Looking forward to resolution of this issue.

[MAN98]

Using New Relic Provider v2.7.1 I have confirmed that the "default" entity tags for our Synthetics are now longer showing after we run terraform plan. Thanks for fixing this issue @zlesnr.

[trunet]

I can confirm 2.7.1 fixed this issue. thank you all. Closing!

[zlesnr]

Woo woo. I forgot to close this out, so thanks for coming back.