aws_appflow_flow has too strict validations on task.source_fields

[ulich]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.2.0
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v4.16.0

Affected Resource(s)

• aws_appflow_flow

Terraform Configuration Files

resource "aws_appflow_flow" "salesforce_to_eventbridge" {
  name = "example"

  source_flow_config {
    connector_type = "Salesforce"
    connector_profile_name = "my-connection"
    source_connector_properties {
      salesforce {
        object = "AccountChangeEvent"
      }
    }
  }

  destination_flow_config {
    connector_type = "EventBridge"
    destination_connector_properties {
      event_bridge {
        object = "aws.partner/appflow/salesforce.com/${data.aws_caller_identity.current.account_id}/example"

        error_handling_config {
          bucket_name = module.large_events_bucket.name
        }
      }
    }
  }

  task {
    task_type         = "Map_all"
    source_fields     = []
    task_properties = {
      EXCLUDE_SOURCE_FIELDS_LIST = "[]"
    }
    connector_operator {
      salesforce = "NO_OP"
    }
  }

  trigger_config {
    trigger_type = "Event"
  }
}

locals {
  large_events_bucket_name = "asdfasdf-appflow-salesforce-large-events"
}

module "large_events_bucket" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "3.2.1"

  bucket = local.large_events_bucket_name
  acl    = "private"

  attach_policy = true
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Service = "appflow.amazonaws.com"
      }
      Action = [
        "s3:PutObject",
        "s3:GetBucketAcl",
        "s3:PutObjectAcl",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts",
        "s3:ListBucketMultipartUploads",
      ]
      Resource = [
        "arn:aws:s3:::${local.large_events_bucket_name}",
        "arn:aws:s3:::${local.large_events_bucket_name}/*",
      ]
    }]
  })
}

Expected Behavior

The flow should be created. The same configuration with CloudFormation works:

Resources:
  Flow:
    Type: 'AWS::AppFlow::Flow'
    Properties:
      FlowName: example
      SourceFlowConfig:
        ConnectorProfileName: my-connection
        ConnectorType: Salesforce
        SourceConnectorProperties:
          Salesforce:
            Object: AccountChangeEvent
      DestinationFlowConfigList:
        - ConnectorType: EventBridge
          DestinationConnectorProperties:
            EventBridge:
              Object: aws.partner/appflow/salesforce.com/${data.aws_caller_identity.current.account_id}/example
              ErrorHandlingConfig:
                BucketName: module.large_events_bucket.name
      Tasks:
        - TaskType: Map_all
          SourceFields: []
          TaskProperties: 
          - Key: EXCLUDE_SOURCE_FIELDS_LIST
            Value: '[]'
          ConnectorOperator:
            Salesforce: NO_OP
      TriggerConfig:
        TriggerType: Event

The goal is to have an appflow that passes each event field 1:1 to event bridge without having to specify each field explicitly.
https://stackoverflow.com/questions/66742683/how-to-map-fields-automatically-in-appflow

Actual Behavior

The flow is not created with the following validation error:

│ Error: creating Appflow Flow (example): InvalidParameter: 1 validation error(s) found.
│ - missing required field, CreateFlowInput.Tasks[0].SourceFields.
│ 
│ 
│   with aws_appflow_flow.salesforce_to_eventbridge,
│   on main.tf line 1, in resource "aws_appflow_flow" "salesforce_to_eventbridge":
│   1: resource "aws_appflow_flow" "salesforce_to_eventbridge" {

Steps to Reproduce

1. terraform apply

[boopathijayagopal]

I'm also facing the Same issue. is there any values we need to explicitly to Map all the fields from Salesforce to Eventbridge?

[jrmarquard]

Also facing the same issue. My work around is to use serverless and define the resource there, does anyone have a terraform workaround?

[harik8]

  task {
    source_fields   = [""]
    task_type       = "Map_all"
    task_properties = {}
    connector_operator {
      salesforce = "NO_OP"
    }
  }

Configuring source_fields as [""] works for me. Yet, I get the below error when I run the same terraform code without any changes.

invalidParameter: 2 validation error(s) found.
- missing required field, UpdateFlowInput.Tasks[0].SourceFields.
- missing required field, UpdateFlowInput.Tasks[0].TaskType.

[simondch]

I'm facing the same issue while trying to update any of my flow, the plan looks correct but apply always end up with the same error.
Running apply with debug, I found this in the object sent to the appflow API for the flow update :

Tasks: [
    {

    },
    {
      ConnectorOperator: 
      ....
    }
 ]

The first empty object appears everytime I made a change in a flow, even if I don't modify the tasks.

The only way I found to be able to continue working with my flows is to delete the flow and then run terraform apply again, this will create a new flow and on On-Demand flow you will loose incremental fetching which is bad with large amounts of data. So it's not a long term solution.

I hope this can help and I can provide more logsif needed .

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

This issue now has a funding of 50.0701 USD attached to it.

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin's Discord
• $1,003,876,521.42 more funded OSS Work available on the Gitcoin Issue Explorer

[pitt-william-t]

I am also facing this problem. This should be straightforward!

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work has been started.

These users each claimed they can complete the work by 2 weeks ago.
Please review their action plans below:

1) fredydev has been approved to start work.

please let me know if you already find someone for this task
2) sandeepseeram-techfirst has been approved to start work.

Certified in Terraform and AWS, I can look into this

Learn more on the Gitcoin Issue Details page.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work for 50.07 USD (50.00 USD @ $1.0/USD) has been submitted by:

1. @fredydev

@camro please take a look at the submitted work:

• PR by @fredydev

---

• Learn more on the Gitcoin Issue Details page
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin's Discord
• $1,003,375,911.64 more funded OSS Work available on the Gitcoin Issue Explorer

[camro]

@freddygv can you point to the PR?

[n11c]

Any progress with this?
@camro's PR seems to solve the issue.
Any chance to see it merged and released soon?

[camro]

@n11c can you put a thumbs up on the PR comment?

[kk-nutmeg]

facing a similar issue, Any plans to release this fix soon? until then any workaround to create an appflow task using Map_all

[JasonSqz]

Facing the same issue, found a workaround by deleting the specific appflow each time I reran terraform apply

[boyardean]

Same issue, offering up this workaround in case it helps anyone: make use of the lifecycle meta argument. Specifically, the ignore_changes arg

lifecycle {
  ignore_changes = [
    task
  ]
}

[Rhexis]

I've also encountered the same issue. Workaround was to create an aws_cloudformation_stack resource which created the Appflow Flow instead of using the aws_appflow_flow resource. Not a fan of this at all, having to come up with workarounds is pretty unacceptable. When can we see the PR by @camro merged?

[walloul-ipsen]

Facing the same issue on my side, can't deploy Appflow task with Map_all twice without changes. This is really annoying and should not be such a pain to make this work.

[togakangaroo]

The above-mentioned PR is merged (at least according to tags). Unfortunately, upgrading to aws provider 5.12 doesn't solve the issue. When I ignore tasks that gets me past the initial required field error at tasks[0], but gives me another one at a further down index for some reason. I suspect that is yet another bug.

However, due to this issue it seems like I wouldn't be able to actually activate the flow, making this provider of limited use anyways.

The workaround of dropping down to cloudformation seems to work on limited testing

resource "aws_cloudformation_stack" "gim_test_appflow" {
  name = "gim-test-stack-appflow"

  parameters = {
    FlowName =  "gim-test-applflow"
  }

  template_body = <<-EOT
  Parameters:
    FlowName:
      Type: String
  Resources:
    AppFlow:
      Type: 'AWS::AppFlow::Flow'
      Properties:
        FlowName:
          Ref: FlowName
        TriggerConfig:
          TriggerType: OnDemand
        SourceFlowConfig:
          ConnectorType: Salesforce
          ConnectorProfileName: SFDC-${local.tenant_name}
          SourceConnectorProperties:
            Salesforce:
              Object: LeadChangeEvent
              EnableDynamicFieldUpdate: false
              IncludeDeletedRecords: false
        DestinationFlowConfigList:
          - ConnectorType: EventBridge
            ConnectorProfileName: ${var.partner_event_source[local.tenant_name]}
            DestinationConnectorProperties:
              EventBridge:
                Object: ${var.partner_event_source[local.tenant_name]}
        Tasks:
          - TaskType: Map_all
            SourceFields: []
            TaskProperties:
              - Key: EXCLUDE_SOURCE_FIELDS_LIST
                Value: '[]'
            ConnectorOperator:
              Salesforce: NO_OP
  EOT
}

[togakangaroo]

We have now moved to using the above in production as well and I can confirm it is working well

[Samira-El]

Hey, I'm running into this issue despite using the latest version of the provider 5.23.1

It doesn't look like this PR #26614 fixed the issue.

[tusharpandit18]

I am facing this issue as well

Terraform v1.5.7
on darwin_amd64

• provider registry.terraform.io/hashicorp/aws v5.26.0
• provider registry.terraform.io/hashicorp/template v2.2.0

[github-actions]

This functionality has been released in v5.39.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.