service/glue: Breaking Changes Between v1.22.1 and v1.22.2

[bflad]

Please fill out the sections below to help us address your issue.

Version of AWS SDK for Go?

github.com/aws/[email protected]

Version of Go (go version)?

go version go1.12.7 darwin/amd64

What issue did you see?

The service/glue Table type introduced breaking changes between v1.22.1 and v1.22.2 of the AWS Go SDK:

v1.21.1 service/glue/api.go:

type Table struct {
	_ struct{} `type:"structure"`

	// Time when the table definition was created in the Data Catalog.
	CreateTime *time.Time `type:"timestamp"`

	// Person or entity who created the table.
	CreatedBy *string `min:"1" type:"string"`

	// Name of the metadata database where the table metadata resides. For Hive
	// compatibility, this must be all lowercase.
	DatabaseName *string `min:"1" type:"string"`

	// Description of the table.
	Description *string `type:"string"`

	// Last time the table was accessed. This is usually taken from HDFS, and may
	// not be reliable.
	LastAccessTime *time.Time `type:"timestamp"`

	// Last time column statistics were computed for this table.
	LastAnalyzedTime *time.Time `type:"timestamp"`

	// Name of the table. For Hive compatibility, this must be entirely lowercase.
	//
	// Name is a required field
	Name *string `min:"1" type:"string" required:"true"`

	// Owner of the table.
	Owner *string `min:"1" type:"string"`

	// These key-value pairs define properties associated with the table.
	Parameters map[string]*string `type:"map"`

	// A list of columns by which the table is partitioned. Only primitive types
	// are supported as partition keys.
	//
	// When creating a table used by Athena, and you do not specify any partitionKeys,
	// you must at least set the value of partitionKeys to an empty list. For example:
	//
	// "PartitionKeys": []
	PartitionKeys []*Column `type:"list"`

	// Retention time for this table.
	Retention *int64 `type:"integer"`

	// A storage descriptor containing information about the physical storage of
	// this table.
	StorageDescriptor *StorageDescriptor `type:"structure"`

	// The type of this table (EXTERNAL_TABLE, VIRTUAL_VIEW, etc.).
	TableType *string `type:"string"`

	// Last time the table was updated.
	UpdateTime *time.Time `type:"timestamp"`

	// If the table is a view, the expanded text of the view; otherwise null.
	ViewExpandedText *string `type:"string"`

	// If the table is a view, the original text of the view; otherwise null.
	ViewOriginalText *string `type:"string"`
}

v1.21.2 service/glue/api.go:

type Table struct {
	_ struct{} `type:"structure"`

	// A unique identifier for the AWS Glue Data Catalog.
	CatalogId *string `min:"1" type:"string"`

	// The name of the connection to the AWS Glue Data Catalog.
	ConnectionName *string `min:"1" type:"string"`

	// A database name in the AWS Glue Data Catalog.
	//
	// DatabaseName is a required field
	DatabaseName *string `min:"1" type:"string" required:"true"`

	// A table name in the AWS Glue Data Catalog.
	//
	// TableName is a required field
	TableName *string `min:"1" type:"string" required:"true"`
}

Steps to reproduce

Using https://github.com/terraform-providers/terraform-provider-aws/ as the test project:

$ make test # no errors on master
$ go get github.com/aws/[email protected]
$ go mod tidy
$ go mod vendor
$ go test ./aws -timeout=30s -parallel=4
# github.com/terraform-providers/terraform-provider-aws/aws
aws/resource_aws_glue_catalog_table.go:274:25: out.Table.Name undefined (type *glue.Table has no field or method Name)
aws/resource_aws_glue_catalog_table.go:277:32: out.Table.Description undefined (type *glue.Table has no field or method Description)
aws/resource_aws_glue_catalog_table.go:278:26: out.Table.Owner undefined (type *glue.Table has no field or method Owner)
aws/resource_aws_glue_catalog_table.go:279:30: out.Table.Retention undefined (type *glue.Table has no field or method Retention)
aws/resource_aws_glue_catalog_table.go:281:78: out.Table.StorageDescriptor undefined (type *glue.Table has no field or method StorageDescriptor)
aws/resource_aws_glue_catalog_table.go:285:64: out.Table.PartitionKeys undefined (type *glue.Table has no field or method PartitionKeys)
aws/resource_aws_glue_catalog_table.go:289:39: out.Table.ViewOriginalText undefined (type *glue.Table has no field or method ViewOriginalText)
aws/resource_aws_glue_catalog_table.go:290:39: out.Table.ViewExpandedText undefined (type *glue.Table has no field or method ViewExpandedText)
aws/resource_aws_glue_catalog_table.go:291:31: out.Table.TableType undefined (type *glue.Table has no field or method TableType)
aws/resource_aws_glue_catalog_table.go:293:60: out.Table.Parameters undefined (type *glue.Table has no field or method Parameters)
aws/resource_aws_glue_catalog_table.go:293:60: too many errors
# github.com/terraform-providers/terraform-provider-aws/aws [github.com/terraform-providers/terraform-provider-aws/aws.test]
aws/resource_aws_glue_catalog_table.go:274:25: out.Table.Name undefined (type *glue.Table has no field or method Name)
aws/resource_aws_glue_catalog_table.go:277:32: out.Table.Description undefined (type *glue.Table has no field or method Description)
aws/resource_aws_glue_catalog_table.go:278:26: out.Table.Owner undefined (type *glue.Table has no field or method Owner)
aws/resource_aws_glue_catalog_table.go:279:30: out.Table.Retention undefined (type *glue.Table has no field or method Retention)
aws/resource_aws_glue_catalog_table.go:281:78: out.Table.StorageDescriptor undefined (type *glue.Table has no field or method StorageDescriptor)
aws/resource_aws_glue_catalog_table.go:285:64: out.Table.PartitionKeys undefined (type *glue.Table has no field or method PartitionKeys)
aws/resource_aws_glue_catalog_table.go:289:39: out.Table.ViewOriginalText undefined (type *glue.Table has no field or method ViewOriginalText)
aws/resource_aws_glue_catalog_table.go:290:39: out.Table.ViewExpandedText undefined (type *glue.Table has no field or method ViewExpandedText)
aws/resource_aws_glue_catalog_table.go:291:31: out.Table.TableType undefined (type *glue.Table has no field or method TableType)
aws/resource_aws_glue_catalog_table.go:293:60: out.Table.Parameters undefined (type *glue.Table has no field or method Parameters)
aws/resource_aws_glue_catalog_table.go:293:60: too many errors
FAIL	github.com/terraform-providers/terraform-provider-aws/aws [build failed]
make: *** [test] Error 2

[bflad]

Strangely enough, the glue model JSON for the Table shape looks okay:

aws-sdk-go/models/apis/glue/2017-03-31/api-2.json

Lines 5532 to 5554 in 62608c7

	"Table":{
	"type":"structure",
	"required":["Name"],
	"members":{
	"Name":{"shape":"NameString"},
	"DatabaseName":{"shape":"NameString"},
	"Description":{"shape":"DescriptionString"},
	"Owner":{"shape":"NameString"},
	"CreateTime":{"shape":"Timestamp"},
	"UpdateTime":{"shape":"Timestamp"},
	"LastAccessTime":{"shape":"Timestamp"},
	"LastAnalyzedTime":{"shape":"Timestamp"},
	"Retention":{"shape":"NonNegativeInteger"},
	"StorageDescriptor":{"shape":"StorageDescriptor"},
	"PartitionKeys":{"shape":"ColumnList"},
	"ViewOriginalText":{"shape":"ViewTextString"},
	"ViewExpandedText":{"shape":"ViewTextString"},
	"TableType":{"shape":"TableTypeString"},
	"Parameters":{"shape":"ParametersMap"},
	"CreatedBy":{"shape":"NameString"},
	"IsRegisteredWithLakeFormation":{"shape":"Boolean"}
	}
	},

And locally regenerating results in no changes:

$ SERVICES=glue make gen-services
Generating SDK clients
go generate ./service
Generating glue (2017-03-31)...
$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

Diving deeper, the generated type seems to be matching GlueTable from the model:

aws-sdk-go/models/apis/glue/2017-03-31/api-2.json

Lines 4280 to 4292 in 62608c7

	"GlueTable":{
	"type":"structure",
	"required":[
	"DatabaseName",
	"TableName"
	],
	"members":{
	"DatabaseName":{"shape":"NameString"},
	"TableName":{"shape":"NameString"},
	"CatalogId":{"shape":"NameString"},
	"ConnectionName":{"shape":"NameString"}
	}
	},

Which was newly introduced to the model in v1.22.2: https://github.com/aws/aws-sdk-go/pull/2739/files#diff-90a2330ee3e649e9ac6ecb993849bb0fR4280

The service generation seems to be collapsing the GlueTable model on top of the Table model.

[bflad]

Potential fix submitted: #2742

[jasdel]

Thanks for reporting this bug @bflad, and PR. While investigating this solution, I needed to create PR #2747 to prevent this issue from happening in the future by disabling the renaming for all new API updates. The PR adds the previously squashed types, and updates the members that should of referred to those squashed types, (e.g. "ActionData" and TableData)

All existing types, that previously worked, will continue to work with the same name they had before, (e,g IotEventsAction is still Action, and GlueTable is still Table).

[jasdel]

Thanks for creating this issue @bflad We've merged in the fix for this bug correcting the SDK's behavior. This will be included in the SDK's next release.