-
Notifications
You must be signed in to change notification settings - Fork 9.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make EventSourceMapping.starting_position optional. #5024
Conversation
Running acceptance testing now. |
@bflad, i've seen you help out with a lot of PRS. Could you let me know if i've done this right, and if there's anything else i need to do? Thanks! |
Can you please add an acceptance test that shows the resource working when you skip adding this argument and the output of |
@bflad Done. results are:
Note: it was good to write these tests. They revealed the need to add retry logic here as event mapping for sqs events can actually take quite a while to transition between states. This was fine locally, but revealed problems in testing when the tests wanted to create/update/delete in rapid succession. |
Amazon recently made it possible to have SQS events trigger Lambdas. See: https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/ for more details. As part of this they specifically utilize CreateSourceEventMapping to describe the mapping between SQS and Lambda: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html *However*, as part of this, they changed the StartingPosition message property from required to optional. You can see that it was previous required here: https://web.archive.org/web/20171222204154/https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-eventsourcemapping.html ``` StartingPosition The position in the stream where Lambda starts reading. For valid values, see CreateEventSourceMapping in the AWS Lambda Developer Guide. Required: Yes Type: String Update requires: Replacement ``` But now is optional in the current set of docs: ``` StartingPosition The position in the stream where Lambda starts reading. For valid values, see CreateEventSourceMapping in the AWS Lambda Developer Guide. Required: No Type: String Update requires: Replacement ``` Unfortunately, this makes is (afaict) impossible to use terraform to configure this EventSourceMapping. If i leave out the starting_position property, terraform complains with: ``` Plan apply failed: Error creating Lambda event source mapping: ValidationException: 1 validation error detected: Value '' at 'startingPosition' failed to satisfy constraint: Member must satisfy enum value set: [LATEST, AT_TIMESTAMP, TRIM_HORIZON] ``` However, if i supply any of those, AWS itself errors out with: ``` Plan apply failed: Error creating Lambda event source mapping: InvalidParameterValueException: StartingPosition is not valid for SQS event sources. ``` This PR attempts to rectify things by making this property optional to match current AWS expectations.
Just tried this PR on a toy project, can confirm that it works fine! Thanks @CyrusNajmabadi, this will be a great help on my current project! |
hey @CyrusNajmabadi - I could be wrong here, buit I'm fairly certain the The default of 100 worked well for Kinesis and DynamoDB streams. However, the default for SQS (assigned by aws) is actually 1, while the max is 10. It's not incredibly obvious from the docs, but you can see this documented within the SDK here. This means that if someone omits the |
@ryandeivert You are correct. However, i'm not sure the right thing to do here. If we remove te batch_size default, won't that break existing customers of terraform who use this for Kinesis/Dynamo, but do not provide a size?
That's true. But then it's fixable by the user. @ryandeivert @bflad I don't know enough about terraform to know what the right thing is to do here. Can either of you shed light on the preferred course of action? Thanks! |
Great! I also have validated this myself. But i'm glad to get secondary validation to ensure that there's nothing special about my setup! Thanks much @flosell ! |
@ryandeivert Looking at the AWS docs some more, it looks like what you're saying should be safe. Their docs mention:
So it sounds like we can simply remove the default for BatchSize. Anyone who is using the default today will be fine, since it will continue to be 100 if left out from the call to the actual AWS SDK. |
It no longer applies as the sqs mapping default is 10.
I've made the change to the default, and i've rerun tests:
|
depends_on = ["aws_iam_policy_attachment.policy_attachment_for_role"] | ||
function_name = "${aws_lambda_function.lambda_function_test_create.arn}" | ||
starting_position = "TRIM_HORIZON" | ||
batch_size = 100 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given that batch_size
had its default removed, it should also be removed from the existing test configurations and separately tested in its own acceptance test and test configuration.
In the cases where you remove it from the test configurations, I have a hunch you will start seeing test failures with the resource wanting to change it since the API likely always returns a value, but its not specified in the configuration. e.g. batch_size: "100" => ""
.
If that is the case, we need to add a DiffSuppressFunc
to the batch_size
attribute.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bflad Can you expand on this a bit. I went to take a look. but i think there's a small snag. If i'm understanding right, we would want to supress things if old
was 100
, and new
was <empty>
, and this was a Kinesis or DynamoDB case.
However, the problem i'm having is figuring out how to tell if this is Kinesis or DynamoDB. event_source_arn doesn't really tell me this, as it's just an arbitrary ARN. Or is the intent that i would somehow get the true value here and try to 'sniff' it to tell what was going on?
Sorry, i've looked around a bit to see if i can figure out the right way to address this. But it was not clear to me. So any help would def be appreciated. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Luckily the ARN has the service name in it 😄 Does something like this do the trick?
DiffSuppressFunc: func(k, old, new string, d *schema.ResourceData) bool {
eventSourceARN, err := arn.Parse(d.Get("event_source_arn").(string))
if err != nil {
return false
}
switch eventSourceARN.Service {
case dynamodb.ServiceID:
if old == "100" && new == "" {
return true
}
case kinesis.ServiceID:
if old == "100" && new == "" {
return true
}
case sqs.ServiceID:
if old == "1" && new == "" {
return true
}
}
return false
},
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Slightly cleaner version:
DiffSuppressFunc: func(k, old, new string, d *schema.ResourceData) bool {
if new != "" {
return false
}
eventSourceARN, err := arn.Parse(d.Get("event_source_arn").(string))
if err != nil {
return false
}
if eventSourceARN.Service == dynamodb.ServiceID && old == "100" {
return true
}
if eventSourceARN.Service == kinesis.ServiceID && old == "100" {
return true
}
if eventSourceARN.Service == sqs.ServiceID && old == "1" {
return true
}
return false
},
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could have also done similar with multiple case statements:
DiffSuppressFunc: func(k, old, new string, d *schema.ResourceData) bool {
eventSourceARN, err := arn.Parse(d.Get("event_source_arn").(string))
if err != nil {
return false
}
switch eventSourceARN.Service {
case dynamodb.ServiceID, kinesis.ServiceID:
if old == "100" && new == "" {
return true
}
case sqs.ServiceID:
if old == "1" && new == "" {
return true
}
}
return false
},
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks guys. That looks like it will do the trick. What i was missing was things like arn.Parse and kinesis.ServiceID. Much better than me manually groveling the string!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok. Was able to make this work. Note: i needed to use .ServiceName not .ServiceID. Also, i needed to check for hte "0" value, not the "" value as it looks like that's what you get by default for integral values...
ForceNew: true, | ||
}, | ||
"batch_size": { | ||
Type: schema.TypeInt, | ||
Optional: true, | ||
Default: 100, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As mentioned, this change will likely require adding a DiffSuppressFunc
to handle which API value is being returned depending on the event source type.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we've modified the behavior of the attribute, we need to appropriately document those changes in website/docs/r/lambda_event_source_mapping.html.markdown
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense. Just to check, but you're going to make that change, right? Or would you prefer i do it?
err := resource.Retry(5*time.Minute, func() *resource.RetryError { | ||
_, err := conn.DeleteEventSourceMapping(params) | ||
if err != nil { | ||
if awserr, ok := err.(awserr.Error); ok { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor nitpick: we have a helper function available for this type of logic and also prefer to use the SDK available constants, e.g. isAWSErr(err, lambda.ErrCodeResourceInUseException, "")
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No worries about nits. I definitely want to do things "the right way". I did copy this code from elsewhere in hte project, so i thought that's what i was doing. But i'll de convert to this new form. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
@@ -196,7 +213,9 @@ func resourceAwsLambdaEventSourceMappingUpdate(d *schema.ResourceData, meta inte | |||
_, err := conn.UpdateEventSourceMapping(params) | |||
if err != nil { | |||
if awserr, ok := err.(awserr.Error); ok { | |||
if awserr.Code() == "InvalidParameterValueException" { | |||
if awserr.Code() == "InvalidParameterValueException" || |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor nitpick: we have a helper function available for this type of logic and also prefer to use the SDK available constants, e.g.
if isAWSErr(err, lambda.ErrCodeInvalidParameterValueException, "") || isAWSErr(err, lambda.ErrCodeResourceInUseException, "") {
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
@@ -234,7 +325,7 @@ EOF | |||
resource "aws_iam_policy" "policy_for_role" { | |||
name = "%s" | |||
path = "/" | |||
description = "IAM policy for for Lamda event mapping testing" | |||
description = "IAM policy for for Lambda event mapping testing" | |||
policy = <<EOF |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for for
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. Thanks!
@@ -322,7 +414,7 @@ EOF | |||
resource "aws_iam_policy" "policy_for_role" { | |||
name = "%s" | |||
path = "/" | |||
description = "IAM policy for for Lamda event mapping testing" | |||
description = "IAM policy for for Lambda event mapping testing" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for for
@bflad This is ready for review again. Note, i have added the additional test to make sure batch_size works when removed. All tests pass:
Hopefully this can go in soon :) let me know if tehre are more change you want me to make. |
@bflad Anything else you need from me at this point? Or is this ready to go in. Thanks! :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks @CyrusNajmabadi! 🚀 There's two items of feedback I'll handle on merge so we can get this released tonight/tomorrow.
6 tests passed (all tests)
=== RUN TestAccAWSLambdaEventSourceMapping_kinesis_disappears
--- PASS: TestAccAWSLambdaEventSourceMapping_kinesis_disappears (77.84s)
=== RUN TestAccAWSLambdaEventSourceMapping_kinesis_import
--- PASS: TestAccAWSLambdaEventSourceMapping_kinesis_import (78.56s)
=== RUN TestAccAWSLambdaEventSourceMapping_kinesis_removeBatchSize
--- PASS: TestAccAWSLambdaEventSourceMapping_kinesis_removeBatchSize (84.54s)
=== RUN TestAccAWSLambdaEventSourceMapping_kinesis_basic
--- PASS: TestAccAWSLambdaEventSourceMapping_kinesis_basic (85.33s)
=== RUN TestAccAWSLambdaEventSourceMapping_sqs_basic
--- PASS: TestAccAWSLambdaEventSourceMapping_sqs_basic (106.63s)
=== RUN TestAccAWSLambdaEventSourceMapping_sqsDisappears
--- PASS: TestAccAWSLambdaEventSourceMapping_sqsDisappears (118.84s)
ForceNew: true, | ||
}, | ||
"batch_size": { | ||
Type: schema.TypeInt, | ||
Optional: true, | ||
Default: 100, | ||
DiffSuppressFunc: func(k, old, new string, d *schema.ResourceData) bool { | ||
// When AWS repurposed EventSourceMapping for use with SQS they kept |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
😍
return true | ||
} | ||
default: | ||
panic(eventSourceARN.Service) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Usually panicking is incorrect within a Terraform provider as this often causes very confusing operator behavior (e.g. unexpected EOF
errors). Instead, we should just silently do nothing here if another service, whether legit or not for usage for this resource, is passed in. With panic()
, when new legitimate services are added, this would forcibly require everyone to upgrade iff the code updated. Without panic()
, the more desirable behavior of perpetually showing a plan difference (ignorable with ignore_changes
if necessary) is possible. We can use future issues to properly handle new services as required.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense!
ForceNew: true, | ||
}, | ||
"batch_size": { | ||
Type: schema.TypeInt, | ||
Optional: true, | ||
Default: 100, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we've modified the behavior of the attribute, we need to appropriately document those changes in website/docs/r/lambda_event_source_mapping.html.markdown
@bflad Thanks! Just to make sure:
So you don't want me to make any changes here right? You'll just tweak things on your end? |
huge thanks to @CyrusNajmabadi and @bflad for expediting this! |
Thanks all! |
This has been released in version 1.26.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. |
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks! |
Amazon recently made it possible to have SQS events trigger Lambdas. See: https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/ for more details.
As part of this they specifically utilize CreateSourceEventMapping to describe the mapping between SQS and Lambda: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
However, as part of this, they changed the StartingPosition message property from required to optional. You can see that it was previous required here: https://web.archive.org/web/20171222204154/https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-eventsourcemapping.html
But now is optional in the current set of docs:
Unfortunately, this makes is (afaict) impossible to use terraform to configure this EventSourceMapping. If i leave out the starting_position property, terraform complains with:
However, if i supply any of those, AWS itself errors out with:
This PR attempts to rectify things by making this property optional to match current AWS expectations.
Fixes #5025
Output from acceptance testing: