[Bug]: Error creating a Linode Stackscript stream error INTERNAL_ERROR

[chrisdlangton]

Terraform Version

1.0.10 on linux_amd64

Linode Provider Version

1.23.0

Effected Terraform Resources

linode_stackscript

Terraform Config Files

resource "random_string" "mysql_replica_password" {
    length  = 32
    special = false
}
resource "random_string" "linode_replica_password" {
    length  = 32
    special = true
}
data "local_file" "alpine_mysql_replica" {
    filename = "${path.root}/../bin/alpine-replica"
}
resource "linode_stackscript" "mysql_replica" {
  label = "mysql-replica"
  description = "Installs mysql read replica"
  script = data.local_file.alpine_mysql_replica.content
  images = [local.linode_default_image]
  rev_note = "initial version"
}
resource "linode_instance" "mysql_replica" {
  label             = "mysql-replica"
  group             = "SaaS"
  tags              = ["SaaS"]
  region            = local.linode_default_region
  type              = local.linode_default_type
  image             = local.linode_default_image
  authorized_keys   = length(var.public_key) == 0 ? [] : [
    var.public_key
  ]
  authorized_users  = [
    var.allowed_linode_username
  ]
  root_pass         = random_string.linode_replica_password.result
  stackscript_id    = linode_stackscript.mysql_replica.id
  stackscript_data  = {
    "HOSTNAME" = local.mysql_replica_hostname
    "MYSQL_ROOT_PASSWORD" = random_string.mysql_replica_password.result
  }
  alerts {
      cpu            = 90
      io             = 10000
      network_in     = 10
      network_out    = 10
      transfer_quota = 80
  }
}

Debug Output

random_string.linode_replica_password: Refreshing state... [id=rq(K{&CIpOa[$_ii1%yCmlt60-CK}kvk]
random_string.mysql_replica_password: Refreshing state... [id=dcBWG9UX63owtCGNJyn4D6LHBjhBkD2W]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with
the following symbols:
  + create

Terraform will perform the following actions:

  # linode_instance.mysql_replica will be created
  + resource "linode_instance" "mysql_replica" {
      + authorized_keys    = []
      + authorized_users   = [
          + "",
        ]
      + backups            = (known after apply)
      + backups_enabled    = (known after apply)
      + boot_config_label  = (known after apply)
      + group              = "SaaS"
      + id                 = (known after apply)
      + image              = "linode/alpine3.14"
      + ip_address         = (known after apply)
      + ipv4               = (known after apply)
      + ipv6               = (known after apply)
      + label              = "mysql-replica"
      + private_ip_address = (known after apply)
      + region             = "ap-southeast"
      + root_pass          = (sensitive value)
      + specs              = (known after apply)
      + stackscript_data   = (sensitive value)
      + stackscript_id     = (known after apply)
      + status             = (known after apply)
      + swap_size          = (known after apply)
      + tags               = [
          + "SaaS",
        ]
      + type               = "g6-nanode-1"
      + watchdog_enabled   = true

      + alerts {
          + cpu            = 90
          + io             = 10000
          + network_in     = 10
          + network_out    = 10
          + transfer_quota = 80
        }
    }

  # linode_stackscript.mysql_replica will be created
  + resource "linode_stackscript" "mysql_replica" {
      + created             = (known after apply)
      + deployments_active  = (known after apply)
      + deployments_total   = (known after apply)
      + description         = "Installs mysql read replica"
      + id                  = (known after apply)
      + images              = [
          + "linode/alpine3.14",
        ]
      + is_public           = false
      + label               = "mysql-replica"
      + rev_note            = "initial version"
      + script              = <<-EOT
            #!/bin/sh
            # <UDF name="HOSTNAME" label="The hostname for the new Linode." />
            # <UDF name="AWS_ACCESS_KEY_ID" Label="AWS API Key ID" />
            # <UDF name="AWS_SECRET_ACCESS_KEY" Label="AWS Secret Access Key" />
            # <UDF name="AWS_REGION" Label="AWS Region" default="ap-southeast-2" />
            # <UDF name="MYSQL_ROOT_PASSWORD" Label="MySQL root user password" />
            # <UDF name="MYSQL_PORT" Label="MySQL port" default="3306" />
            # <UDF name="TIMEZONE" Label="Your timezone" example="Australia/Sydney (see a list at https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)" default="Etc/UTC" />
            # <UDF name="ALLOWED_IP_ADDRESSES" Label="MySQL allowed_ip_addresses" />
            
            set -ex
            exec >  >(tee -a /root/stackscript.log)
            exec 2> >(tee -a /root/stackscript.log >&2)
            echo "Linode Variable Values:  LINODE_ID: $LINODE_ID,  LINODE_LISHUSERNAME: $LINODE_LISHUSERNAME,  LINODE_RAM: $LINODE_RAM,  LINODE_DATACENTERID:$LINODE_DATACENTERID"
            
            system_primary_ip=$(ip route get 1 | egrep -o 'src [[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' | awk '{print $2;exit}')
            network_interface=$(cat /etc/network/interfaces | egrep "iface .* static" | awk '{print $2;exit}')
            
            ######## truncated ########

            function main() {
                setup_os
                patch_os
                add_packages
                setup_aws
                mysql_install
                setup_firewall
                configure_ports
            
                lsof -i :${MYSQL_PORT} || (
                    rc-service mysql stop
                    rc-service mysql start
                )
                netstat -ltn
                echo $(date +'%F') > /root/.deployed
                echo "Stackscript finished"
                stackscript_cleanup
            }
            
            main
        EOT
      + updated             = (known after apply)
      + user_defined_fields = (known after apply)
      + user_gravatar_id    = (known after apply)
      + username            = (known after apply)
    }

Plan: 2 to add, 0 to change, 0 to destroy.
linode_stackscript.mysql_replica: Creating...
linode_stackscript.mysql_main: Creating...
╷
│ Error: Error creating a Linode Stackscript: [002] stream error: stream ID 5; INTERNAL_ERROR
│ 
│   with linode_stackscript.mysql_main,
│   on mysql_main.tf line 12, in resource "linode_stackscript" "mysql_main":
│   12: resource "linode_stackscript" "mysql_main" {
│ 
╵
╷
│ Error: Error creating a Linode Stackscript: [002] stream error: stream ID 3; INTERNAL_ERROR
│ 
│   with linode_stackscript.mysql_replica,
│   on mysql_replica.tf line 12, in resource "linode_stackscript" "mysql_replica":
│   12: resource "linode_stackscript" "mysql_replica" {
│ 
╵

Panic Output

No response

Expected Behavior

creates a StackScript

Actual Behavior

no resources complete

Steps to Reproduce

TF_LINODE=DEBUG LINODE_DEBUG=1 terraform apply -auto-approve -refresh=true

if the random and local are not familiar, use these for your main.tf;

provider "linode" {
    token = var.linode_token
}

terraform {
    required_version = ">= 1.0.1"

    required_providers {
        linode = {
            source = "linode/linode"
            version = ">= 1.23.0"
        }
        random = {
            source = "hashicorp/random"
            version = ">= 3.1.0"
        }
        local = {
            source = "hashicorp/local"
            version = ">= 2.1.0"
        }
    }
}

[LBGarber]

Hello, thanks for the report!

I haven't been able to reproduce this issue yet on darwin_amd64 or linux_amd64, but digging into this further makes it seem like this is some sort of intermittent HTTP issue:

• x/net/http2: seeing INTERNAL_ERROR golang/go#26338
• storage: multipart: stream error: stream ID 1; INTERNAL_ERROR googleapis/google-cloud-go#987
• stream error: stream ID 19; INTERNAL_ERROR willnorris/imageproxy#156

Has this issue been occurring for you on other Terraform configurations?

[chrisdlangton]

Other Terraform configurations locally using linode_stackscript? Yes consistently fail locally

Other Terraform configurations on gitlab runner hosted on linode using linode_stackscript? Yes consistently fail from linode network

Other Terraform configurations NOT using linode_stackscript? No, all work as expected consistently do not fail, but they are not using linode_stackscript so may not be relevant to this bug report.

Actually, if i remove linode_stackscript and do a direct reference of stackscript_id = <id:int> for the linode_instance there is no problems at all so there is definitely an issue with linode_stackscript

[LBGarber]

@chrisdlangton Thanks for the info! I'll see if I can reproduce this issue in a GitLab runner environment and get back to you.

[LBGarber]

Hmm, tested this on a Linode running Alpine 3.14 inside and outside of a container and I haven't been able to reproduce anything yet.

I was originally thinking this was some sort of strange request size restraint, but you mentioned that this is occurring in all of the configurations you tested that contain a linode_stackscript.

Would you mind providing a few more details?:

• Roughly how large are the contents the StackScript you're attempting to create?
• Which Linode type/size is your GitLab runner Linode?
• Does this issue occur for you on older provider versions?
• Are you able to manually create this StackScript through Linode's web interface? (I assume so because you are able to reference your StackScript by ID)

Thanks in advance!

[chrisdlangton]

This was the issue!
I tested source <ssinclude StackScriptID="int"> as a theory and everything worked as expected.

The original (not working) size was only 18kb , now (working) the referenced script is 7.5kb and the main script is only 4.2kb
Maybe there's a 16kb memory allocation?

[LBGarber]

Glad you were able to narrow it down!
I'll take a look and see if there's anything we can do on the provider-side to mitigate this issue. 👍

[LBGarber]

Although I still haven't been able to reproduce this on my end, I dug a bit deeper into this and found where this error seems to be thrown:

https://github.com/golang/go/blob/go1.17/src/net/http/h2_bundle.go#L8006

Since this is an upstream issue there's not too much we can do to mitigate it right now, but there does seem to be some movement around this upstream (present in go 1.17.3):

https://go-review.googlesource.com/c/go/+/359776/4/src/net/http/h2_bundle.go#b8015

I'll keep this issue open for the time being and let you know if there are any developments