Add Alibaba Cloud backend OSS with lock #16927

xiaozhu36 · 2017-12-15T10:52:35Z

The PR provides an new cloud backend oss that terraform can store state remotely in OSS and lock that state with OSS.

The result of running test case as follows:

TF_ACC=1 go test -v ./backend/remote-state/oss/ -run=TestBackend -timeout 120m
=== RUN TestBackend_impl
--- PASS: TestBackend_impl (0.00s)
=== RUN TestBackendConfig
2017/12/15 18:09:59 [DEBUG] Instantiate OSS client using endpoint: "oss-cn-beijing.aliyuncs.com"
--- PASS: TestBackendConfig (0.26s)
=== RUN TestBackendConfig_invalidKey
--- PASS: TestBackendConfig_invalidKey (0.00s)
=== RUN TestBackend
2017/12/15 18:09:59 [DEBUG] Instantiate OSS client using endpoint: "oss-cn-beijing.aliyuncs.com"

(some logs)

2017/12/15 18:10:03 Delete Object workspaces/foo/testTFState successfully.
--- PASS: TestBackend (4.35s)
backend_test.go:253: creating OSS bucket terraform-remote-oss-test-5a339f77 in http://oss-cn-beijing.aliyuncs.com
=== RUN TestBackendLocked
2017/12/15 18:10:03 [DEBUG] Instantiate OSS client using endpoint: "oss-cn-beijing.aliyuncs.com"

(some logs)

2017/12/15 18:10:08 Delete Object tfTest/mystate.tflock successfully.
--- PASS: TestBackendLocked (5.40s)
backend_test.go:253: creating OSS bucket terraform-remote-oss-test-5a339f7b in http://oss-cn-beijing.aliyuncs.com
backend_test.go:122: TestBackend: testing state locking for *oss.Backend
=== RUN TestBackendExtraPaths
2017/12/15 18:10:09 [DEBUG] Instantiate OSS client using endpoint: "oss-cn-beijing.aliyuncs.com"

(some logs)

2017/12/15 18:10:10 [TRACE] Preserving existing state lineage "ff0bbc96-824c-418b-9f88-866d32f6b7ea"
--- PASS: TestBackendExtraPaths (2.07s)
backend_test.go:253: creating OSS bucket terraform-remote-oss-test-5a339f81 in http://oss-cn-beijing.aliyuncs.com
PASS
ok github.com/hashicorp/terraform/backend/remote-state/oss 12.120s

TF_ACC=1 go test -v ./backend/remote-state/oss/ -run=TestRemoteClient -timeout 120m
=== RUN TestRemoteClient_impl
--- PASS: TestRemoteClient_impl (0.00s)
=== RUN TestRemoteClient
2017/12/15 18:04:32 [DEBUG] Instantiate OSS client using endpoint: "oss-cn-beijing.aliyuncs.com"

(some logs)

2017/12/15 18:04:33 State testState has no data.
--- PASS: TestRemoteClient (1.44s)
backend_test.go:253: creating OSS bucket terraform-remote-oss-test-5a339e30 in http://oss-cn-beijing.aliyuncs.com
=== RUN TestRemoteClientLocks
2017/12/15 18:04:33 [DEBUG] Instantiate OSS client using endpoint: "oss-cn-beijing.aliyuncs.com"

(some logs)

2017/12/15 18:04:36 Delete Object testState.tflock successfully.
--- PASS: TestRemoteClientLocks (3.09s)
backend_test.go:253: creating OSS bucket terraform-remote-oss-test-5a339e31 in http://oss-cn-beijing.aliyuncs.com
PASS
ok github.com/hashicorp/terraform/backend/remote-state/oss 4.566s

jbardin

Hi, thanks for putting the work in to creating the OSS remote state. There's a few issues that need to be addressed before we can evaluate this further.

Since this is an entirely new backend, basing it on s3 isn't entirely appropriate. You may want to look at some newer backends for inspiration too, like manta, or etcdv3. Specifically the handling of the default workspace doesn't need to be a special case, since there are no existing oss remote states to maintain compatibility with. Removing that special case will simplify a lot of this code.

Regarding locks, if the storage layer is fully consistent, we still need a way to conditionally write the lock. Checking for a locks existence then writing the value doesn't provide any safety guarantees on its own.

jbardin · 2018-02-21T18:29:29Z

backend/remote-state/oss/backend.go

+				Default:     "",
+			},
+
+			"workspace_key_prefix": &schema.Schema{


It looks like you used the S3 backend as a template for this. That implementation contains a lot of backwards compatibility shims like workspace_key_prefix, and probably shouldn't be used as a reference. Since this is an entirely new implementation, it's easier to just treat the default workspace the same as every other rather than special-case it.

jbardin · 2018-02-21T18:32:08Z

backend/remote-state/oss/backend.go

+				Description: "The name of the OSS bucket",
+			},
+
+			"key": &schema.Schema{


You most likely want the configuration to be a path or a prefix, not a specific key, as that can't really work for named states (workspaces).

jbardin · 2018-02-21T18:41:30Z

backend/remote-state/oss/backend_state.go

+
+// extract the object name from the OSS key
+func (b *Backend) keyEnv(key string) string {
+	// we have 3 parts, the workspace key prefix, the workspace name, and the state key name


This was copied from an older version of the s3 backend that still had prefix related bugs.
Removing the special casing of the default workspace and having a single path/prefix removes the need for this.

jbardin · 2018-02-21T18:44:38Z

backend/remote-state/oss/client.go

+		return "", fmt.Errorf("Error getting bucket: %#v", err)
+	}
+
+	log.Printf("Lock info:%#v", info)


This needs a log level. We haven't transitioned to fully structured logging yet, so you can continue to use the text prefixes, like [DEBUG]

backend/remote-state/oss/client.go

backend/remote-state/oss/backend.go

jbardin · 2018-02-21T19:17:48Z

backend/remote-state/oss/client.go

+	}
+
+	var hashChannel = make(chan []byte, 1)
+	sum := md5.Sum(buf.Bytes())


There's no need for a channel here

jbardin · 2018-02-21T19:18:56Z

backend/remote-state/oss/client.go

+}
+
+func (c *RemoteClient) Unlock(id string) error {
+	log.Printf("UnLock info %s", id)


needs log level

backend/remote-state/oss/client.go

georgedriver · 2018-03-19T12:16:52Z

@xiaozhu36 This feature is absolutely what I want integrated with Alicloud. Just wondering when can we have this pull request merged into the master. 👍

georgedriver · 2018-04-17T01:27:25Z

Any update on this? Really want to this feature support. 👍

xiaozhu36 · 2018-04-17T01:31:58Z

HI @georgedriver I have updated it but the ci test happened a problem. Please be patient. Thanks a lot.

heatmiser · 2018-05-19T08:19:55Z

Update on when this might be ready for inclusion in Terraform? Thanks in advance.

xiaozhu36 · 2018-05-31T01:15:06Z

HI @heatmiser Thanks for your attention. I am trying my best for it. Please be patient.

HI @jbardin Is there any progress for the PR? Looking forward to your new review. Thanks a lot.

jbardin · 2018-05-31T01:27:40Z

Hi @xiaozhu36,

Sorry about the delay here. The main branch of terraform is frozen at the moment, while we work on some in-depth refactoring for the 0.12 release.

Once the 0.12 development reaches a more stable state I can take a look at this PR again ASAP.

rouralberto · 2018-07-05T02:45:54Z

Any update on this? This backend is getting increasingly necessary here, @jbardin

JennyLi90 · 2018-08-09T02:43:39Z

Any update for this? is there any plan for merge?

jbardin · 2018-08-16T18:22:19Z

Hi Everyone,

Sorry about the delay here.

Unfortunately the updated PR missed the freeze we had in core in preparing for the major work being done for 0.12. This PR will be reviewed again in due time, but since there are no further planned 0.11 releases in which to include this, we are waiting until we can provide a stable branch to merge into.

Thank you for your patience while we work towards 0.12!

toamitkumar · 2019-03-04T07:30:47Z

@xiaozhu36 @jbardin should I wait or look for workaround, please advise. Thanks

xiaozhu36 · 2019-03-06T14:42:10Z

HI @toamitkumar I think this pr is ok, and if @jbardin still has other suggestion, I can fix it in the next version. Thanks a lot.

jbardin · 2019-03-06T17:35:36Z

backend/remote-state/oss/client.go

+
+	info.Path = c.lockFile
+	if exist, err := bucket.IsObjectExist(info.Path); err != nil {
+		return "", fmt.Errorf("Estimating object %s is exist got an error: %#v", info.Path, err)


This error text needs to be corrected.

jbardin · 2019-03-06T17:39:02Z

backend/remote-state/oss/client.go

+	if exist, err := bucket.IsObjectExist(info.Path); err != nil {
+		return "", fmt.Errorf("Estimating object %s is exist got an error: %#v", info.Path, err)
+	} else if !exist {
+		if err := c.putObj(info.Path, infoJson); err != nil {


Checking the existence then putting the lock object creates a race condition.
Is there some way to write the object and have it rejected the request if it exists? (this is one of the reasons the s3 backend has to delegate locking to DynamoDB). Basically, will putObj fail if the object already exists?

I am sorry for that now there is no better way, like s3 DynamoDB, to reject the request . At present, the method putObj will overwrite the existing object, see details. I think this question is not blocking issue for this PR, and this PR can be merged firstly. In the next, I will continue to find the better way like s3 DynamoDB to avoid some request.

@jbardin Do you have other ideas?

I don't think we want to advertise the storage as safe for concurrent use if the locking mechanism isn't actually safe. For now it's probably better to remove the locking altogether so there is no confusion about what this backend can provide.

HI @jbardin Thanks for your suggestion. I want to make the locking mechanism safe by using alibaba cloud tablestore. But, when I tried to make a test using a custom terraform package, I happened a issue: Error: Failed to instantiate provider "alicloud" to obtain schema: Incompatible API version with plugin. Plugin version: 4, Client versions: [5]. Can you give me some help? Thanks in advance.

@xiaozhu36, the provider needs to be updated to work with terraform 0.12 (the master branch).

HI @jbardin Thanks for your feedback. I have updated but the error still exist.

If the error says Plugin version: 4, that means the plugin you are executing was compiled with an old version of the terraform packages. The only way to fix that is to compile and execute the provider plugin using the latest version of the core package (which for now is master), and we suggest using go modules to do this.

HI @jbardin I have add tablestore config to store the state lock. Please review it.

jbardin · 2019-03-06T17:39:25Z

backend/remote-state/oss/client.go

+		return nil, fmt.Errorf("Error getting bucket: %#v", err)
+	}
+
+	if exist, err := bucket.IsObjectExist(key); err != nil {


Do we really need to check existence first?
This of course introduces the possibility of the object being deleted before the GetObject call.

Removed. You are right. Even if the object is not exist, DeleteObject will not return a error.

xiaozhu36 · 2019-04-08T03:18:43Z

HI @jbardin Is there any new comments for this PR?

jbardin · 2019-04-10T20:58:33Z

Thanks for the updates @xiaozhu36.

Before we merge I would ask that you make a separate commit for the module and vendor updates, and remove the stringer changes that you have in here as well. If the ACC tests look OK with the new locking enabled (please add the results here too), we can move on with this PR 👍

xiaozhu36 · 2019-04-10T23:19:42Z

HI @jbardin Thanks for your review. The following is the PR ACC tests results and please check it:

TF_ACC=1 go test -v ./backend/remote-state/oss/ -run=TestRemote -timeout 120m
=== RUN   TestRemoteClient_impl
--- PASS: TestRemoteClient_impl (0.00s)
=== RUN   TestRemoteClient
2019/04/11 07:01:19 [DEBUG] Instantiate OSS client using endpoint: "https://oss-cn-beijing.aliyuncs.com"
2019/04/11 07:01:20 [DEBUG] Current workspace name: default. All workspaces:[]string{"default"}
2019/04/11 07:01:20 [DEBUG] Deleting remote state from OSS: "testState/terraform.tfstate"
--- PASS: TestRemoteClient (2.96s)
    client_test.go:30: TestBackendConfig on *oss.Backend with configs.synthBody{Filename:"<TestWrapConfig>", Values:map[string]cty.Value{"prefix":cty.StringVal("testState"), "bucket":cty.StringVal("tf-remote-oss-test-5cae75bd"), "encrypt":cty.True}}
=== RUN   TestRemoteClientLocks
2019/04/11 07:01:20 [DEBUG] Instantiate OSS client using endpoint: "https://oss-cn-beijing.aliyuncs.com"
2019/04/11 07:01:21 [DEBUG] Instantiate OSS client using endpoint: "https://oss-cn-beijing.aliyuncs.com"
2019/04/11 07:01:23 [DEBUG] Current workspace name: default. All workspaces:[]string{"default"}
2019/04/11 07:01:23 [DEBUG] Current workspace name: default. All workspaces:[]string{"default"}
2019/04/11 07:01:23 [DEBUG] Recoring state lock in tablestore: &tablestore.PutRowChange{TableName:"tfRemoteTestForce5cae75c0", PrimaryKey:(*tablestore.PrimaryKey)(0xc000438d20), Columns:[]tablestore.AttributeColumn{tablestore.AttributeColumn{ColumnName:"LockID", Value:"testState/terraform.tfstate.tflock", Timestamp:0}, tablestore.AttributeColumn{ColumnName:"Info", Value:"{\"ID\":\"acc7cb35-7af4-5522-48cb-c19d4e330fdd\",\"Operation\":\"test\",\"Info\":\"\",\"Who\":\"clientA\",\"Version\":\"0.12.0\",\"Created\":\"2019-04-10T23:01:23.458233Z\",\"Path\":\"\"}", Timestamp:0}}, Condition:(*tablestore.RowCondition)(0xc000438d40), ReturnType:0, TransactionId:(*string)(nil)}
2019/04/11 07:01:23 [DEBUG] Recoring state lock in tablestore: &tablestore.PutRowChange{TableName:"tfRemoteTestForce5cae75c0", PrimaryKey:(*tablestore.PrimaryKey)(0xc0003ca540), Columns:[]tablestore.AttributeColumn{tablestore.AttributeColumn{ColumnName:"LockID", Value:"testState/terraform.tfstate.tflock", Timestamp:0}, tablestore.AttributeColumn{ColumnName:"Info", Value:"{\"ID\":\"434e7d8b-b0c2-7e35-2d11-bc76c6998db7\",\"Operation\":\"test\",\"Info\":\"\",\"Who\":\"clientB\",\"Version\":\"0.12.0\",\"Created\":\"2019-04-10T23:01:23.458248Z\",\"Path\":\"\"}", Timestamp:0}}, Condition:(*tablestore.RowCondition)(0xc0003ca580), ReturnType:0, TransactionId:(*string)(nil)}
2019/04/11 07:01:23 [WARN] Error storing state lock in tablestore: &tablestore.OtsError{Code:"OTSConditionCheckFail", Message:"Condition check failed.", RequestId:"00058635-0c7f-fa91-cdde-c00b478d6f75"}
2019/04/11 07:01:23 [DEBUG] Retrieving state lock info from tablestore: &tablestore.SingleRowQueryCriteria{ColumnsToGet:[]string{"LockID", "Info"}, TableName:"tfRemoteTestForce5cae75c0", PrimaryKey:(*tablestore.PrimaryKey)(0xc000439440), MaxVersion:1, TimeRange:(*tablestore.TimeRange)(nil), Filter:tablestore.ColumnFilter(nil), StartColumn:(*string)(nil), EndColumn:(*string)(nil), TransactionId:(*string)(nil)}
2019/04/11 07:01:23 [WARN] state lock error: &statemgr.LockError{Info:(*statemgr.LockInfo)(0xc00022ea80), Err:(*tablestore.OtsError)(0xc0004fb170)}
2019/04/11 07:01:23 [DEBUG] Retrieving state lock info from tablestore: &tablestore.SingleRowQueryCriteria{ColumnsToGet:[]string{"LockID", "Info"}, TableName:"tfRemoteTestForce5cae75c0", PrimaryKey:(*tablestore.PrimaryKey)(0xc0004399e0), MaxVersion:1, TimeRange:(*tablestore.TimeRange)(nil), Filter:tablestore.ColumnFilter(nil), StartColumn:(*string)(nil), EndColumn:(*string)(nil), TransactionId:(*string)(nil)}
2019/04/11 07:01:23 [DEBUG] Deleting state lock from tablestore: &tablestore.DeleteRowRequest{DeleteRowChange:(*tablestore.DeleteRowChange)(0xc0004fbef0)}
2019/04/11 07:01:23 [DEBUG] Recoring state lock in tablestore: &tablestore.PutRowChange{TableName:"tfRemoteTestForce5cae75c0", PrimaryKey:(*tablestore.PrimaryKey)(0xc000423f40), Columns:[]tablestore.AttributeColumn{tablestore.AttributeColumn{ColumnName:"LockID", Value:"testState/terraform.tfstate.tflock", Timestamp:0}, tablestore.AttributeColumn{ColumnName:"Info", Value:"{\"ID\":\"434e7d8b-b0c2-7e35-2d11-bc76c6998db7\",\"Operation\":\"test\",\"Info\":\"\",\"Who\":\"clientB\",\"Version\":\"0.12.0\",\"Created\":\"2019-04-10T23:01:23.458248Z\",\"Path\":\"\"}", Timestamp:0}}, Condition:(*tablestore.RowCondition)(0xc000423f80), ReturnType:0, TransactionId:(*string)(nil)}
2019/04/11 07:01:23 [DEBUG] Retrieving state lock info from tablestore: &tablestore.SingleRowQueryCriteria{ColumnsToGet:[]string{"LockID", "Info"}, TableName:"tfRemoteTestForce5cae75c0", PrimaryKey:(*tablestore.PrimaryKey)(0xc000412240), MaxVersion:1, TimeRange:(*tablestore.TimeRange)(nil), Filter:tablestore.ColumnFilter(nil), StartColumn:(*string)(nil), EndColumn:(*string)(nil), TransactionId:(*string)(nil)}
2019/04/11 07:01:23 [DEBUG] Deleting state lock from tablestore: &tablestore.DeleteRowRequest{DeleteRowChange:(*tablestore.DeleteRowChange)(0xc0003e43c0)}
--- PASS: TestRemoteClientLocks (3.54s)
    client_test.go:53: TestBackendConfig on *oss.Backend with configs.synthBody{Filename:"<TestWrapConfig>", Values:map[string]cty.Value{"bucket":cty.StringVal("tf-remote-oss-test-5cae75c0"), "encrypt":cty.True, "prefix":cty.StringVal("testState"), "tablestore_endpoint":cty.StringVal("https://tf-oss-remote.cn-hangzhou.ots.aliyuncs.com"), "tablestore_table":cty.StringVal("tfRemoteTestForce5cae75c0")}}
    client_test.go:61: TestBackendConfig on *oss.Backend with configs.synthBody{Filename:"<TestWrapConfig>", Values:map[string]cty.Value{"bucket":cty.StringVal("tf-remote-oss-test-5cae75c0"), "encrypt":cty.True, "prefix":cty.StringVal("testState"), "tablestore_endpoint":cty.StringVal("https://tf-oss-remote.cn-hangzhou.ots.aliyuncs.com"), "tablestore_table":cty.StringVal("tfRemoteTestForce5cae75c0")}}
=== RUN   TestRemoteForceUnlock
2019/04/11 07:01:24 [DEBUG] Instantiate OSS client using endpoint: "https://oss-cn-beijing.aliyuncs.com"
2019/04/11 07:01:24 [DEBUG] Instantiate OSS client using endpoint: "https://oss-cn-beijing.aliyuncs.com"
2019/04/11 07:01:25 [DEBUG] Current workspace name: default. All workspaces:[]string{"default"}
2019/04/11 07:01:25 [DEBUG] Recoring state lock in tablestore: &tablestore.PutRowChange{TableName:"tfRemoteTestForce5cae75c4", PrimaryKey:(*tablestore.PrimaryKey)(0xc000407620), Columns:[]tablestore.AttributeColumn{tablestore.AttributeColumn{ColumnName:"LockID", Value:"testState/terraform.tfstate.tflock", Timestamp:0}, tablestore.AttributeColumn{ColumnName:"Info", Value:"{\"ID\":\"2db5d704-c4b3-6514-4b78-65993401716f\",\"Operation\":\"test\",\"Info\":\"\",\"Who\":\"clientA\",\"Version\":\"0.12.0\",\"Created\":\"2019-04-10T23:01:25.654963Z\",\"Path\":\"\"}", Timestamp:0}}, Condition:(*tablestore.RowCondition)(0xc000407660), ReturnType:0, TransactionId:(*string)(nil)}
2019/04/11 07:01:26 [DEBUG] Current workspace name: default. All workspaces:[]string{"default"}
2019/04/11 07:01:26 [DEBUG] Retrieving state lock info from tablestore: &tablestore.SingleRowQueryCriteria{ColumnsToGet:[]string{"LockID", "Info"}, TableName:"tfRemoteTestForce5cae75c4", PrimaryKey:(*tablestore.PrimaryKey)(0xc0003d4d80), MaxVersion:1, TimeRange:(*tablestore.TimeRange)(nil), Filter:tablestore.ColumnFilter(nil), StartColumn:(*string)(nil), EndColumn:(*string)(nil), TransactionId:(*string)(nil)}
2019/04/11 07:01:26 [DEBUG] Deleting state lock from tablestore: &tablestore.DeleteRowRequest{DeleteRowChange:(*tablestore.DeleteRowChange)(0xc00056afc0)}
2019/04/11 07:01:26 [DEBUG] Current workspace name: test. All workspaces:[]string{"default"}
2019/04/11 07:01:26 [DEBUG] Recoring state lock in tablestore: &tablestore.PutRowChange{TableName:"tfRemoteTestForce5cae75c4", PrimaryKey:(*tablestore.PrimaryKey)(0xc0004064e0), Columns:[]tablestore.AttributeColumn{tablestore.AttributeColumn{ColumnName:"LockID", Value:"testState/test/terraform.tfstate.tflock", Timestamp:0}, tablestore.AttributeColumn{ColumnName:"Info", Value:"{\"ID\":\"ff7505e7-2a77-a2a5-c61a-7d772e781533\",\"Operation\":\"init\",\"Info\":\"\",\"Who\":\"[email protected]\",\"Version\":\"0.12.0\",\"Created\":\"2019-04-10T23:01:26.549274Z\",\"Path\":\"\"}", Timestamp:0}}, Condition:(*tablestore.RowCondition)(0xc000406500), ReturnType:0, TransactionId:(*string)(nil)}
2019/04/11 07:01:26 [DEBUG] Retrieving state serial in tablestore: &tablestore.SingleRowQueryCriteria{ColumnsToGet:[]string{"LockID", "Digest"}, TableName:"tfRemoteTestForce5cae75c4", PrimaryKey:(*tablestore.PrimaryKey)(0xc000406680), MaxVersion:1, TimeRange:(*tablestore.TimeRange)(nil), Filter:tablestore.ColumnFilter(nil), StartColumn:(*string)(nil), EndColumn:(*string)(nil), TransactionId:(*string)(nil)}
2019/04/11 07:01:26 [WARN] failed to fetch state md5: invalid md5
2019/04/11 07:01:26 [DEBUG] Retrieving state serial in tablestore: &tablestore.SingleRowQueryCriteria{ColumnsToGet:[]string{"LockID", "Digest"}, TableName:"tfRemoteTestForce5cae75c4", PrimaryKey:(*tablestore.PrimaryKey)(0xc000470580), MaxVersion:1, TimeRange:(*tablestore.TimeRange)(nil), Filter:tablestore.ColumnFilter(nil), StartColumn:(*string)(nil), EndColumn:(*string)(nil), TransactionId:(*string)(nil)}
2019/04/11 07:01:26 [WARN] failed to fetch state md5: invalid md5
2019/04/11 07:01:26 [DEBUG] Recoring state serial in tablestore: &tablestore.PutRowChange{TableName:"tfRemoteTestForce5cae75c4", PrimaryKey:(*tablestore.PrimaryKey)(0xc0003d5700), Columns:[]tablestore.AttributeColumn{tablestore.AttributeColumn{ColumnName:"LockID", Value:"tf-remote-oss-test-force-5cae75c4/testState/test/terraform.tfstate-md5", Timestamp:0}, tablestore.AttributeColumn{ColumnName:"Digest", Value:"7c031768fb90eed7a118fbba4cbded9c", Timestamp:0}}, Condition:(*tablestore.RowCondition)(0xc0003d5720), ReturnType:0, TransactionId:(*string)(nil)}
2019/04/11 07:01:26 [WARN] failed to record state serial in tablestore: OTSConditionCheckFail Condition check failed. 00058635-0cad-3430-5cde-c00b3e54d520
2019/04/11 07:01:26 [DEBUG] Retrieving state lock info from tablestore: &tablestore.SingleRowQueryCriteria{ColumnsToGet:[]string{"LockID", "Info"}, TableName:"tfRemoteTestForce5cae75c4", PrimaryKey:(*tablestore.PrimaryKey)(0xc000470820), MaxVersion:1, TimeRange:(*tablestore.TimeRange)(nil), Filter:tablestore.ColumnFilter(nil), StartColumn:(*string)(nil), EndColumn:(*string)(nil), TransactionId:(*string)(nil)}
2019/04/11 07:01:26 [DEBUG] Deleting state lock from tablestore: &tablestore.DeleteRowRequest{DeleteRowChange:(*tablestore.DeleteRowChange)(0xc0002cbb90)}
2019/04/11 07:01:26 [DEBUG] Recoring state lock in tablestore: &tablestore.PutRowChange{TableName:"tfRemoteTestForce5cae75c4", PrimaryKey:(*tablestore.PrimaryKey)(0xc000471de0), Columns:[]tablestore.AttributeColumn{tablestore.AttributeColumn{ColumnName:"LockID", Value:"testState/test/terraform.tfstate.tflock", Timestamp:0}, tablestore.AttributeColumn{ColumnName:"Info", Value:"{\"ID\":\"6101dfd6-79bc-4eb9-d131-497e1c707cac\",\"Operation\":\"test\",\"Info\":\"\",\"Who\":\"clientA\",\"Version\":\"0.12.0\",\"Created\":\"2019-04-10T23:01:26.730552Z\",\"Path\":\"\"}", Timestamp:0}}, Condition:(*tablestore.RowCondition)(0xc000471e00), ReturnType:0, TransactionId:(*string)(nil)}
2019/04/11 07:01:26 [DEBUG] Current workspace name: test. All workspaces:[]string{"default", "test"}
2019/04/11 07:01:26 [DEBUG] Retrieving state lock info from tablestore: &tablestore.SingleRowQueryCriteria{ColumnsToGet:[]string{"LockID", "Info"}, TableName:"tfRemoteTestForce5cae75c4", PrimaryKey:(*tablestore.PrimaryKey)(0xc0004131a0), MaxVersion:1, TimeRange:(*tablestore.TimeRange)(nil), Filter:tablestore.ColumnFilter(nil), StartColumn:(*string)(nil), EndColumn:(*string)(nil), TransactionId:(*string)(nil)}
2019/04/11 07:01:26 [DEBUG] Deleting state lock from tablestore: &tablestore.DeleteRowRequest{DeleteRowChange:(*tablestore.DeleteRowChange)(0xc0002d7740)}
--- PASS: TestRemoteForceUnlock (2.97s)
    client_test.go:94: TestBackendConfig on *oss.Backend with configs.synthBody{Filename:"<TestWrapConfig>", Values:map[string]cty.Value{"bucket":cty.StringVal("tf-remote-oss-test-force-5cae75c4"), "encrypt":cty.True, "prefix":cty.StringVal("testState"), "tablestore_endpoint":cty.StringVal("https://tf-oss-remote.cn-hangzhou.ots.aliyuncs.com"), "tablestore_table":cty.StringVal("tfRemoteTestForce5cae75c4")}}
    client_test.go:102: TestBackendConfig on *oss.Backend with configs.synthBody{Filename:"<TestWrapConfig>", Values:map[string]cty.Value{"bucket":cty.StringVal("tf-remote-oss-test-force-5cae75c4"), "encrypt":cty.True, "prefix":cty.StringVal("testState"), "tablestore_endpoint":cty.StringVal("https://tf-oss-remote.cn-hangzhou.ots.aliyuncs.com"), "tablestore_table":cty.StringVal("tfRemoteTestForce5cae75c4")}}
=== RUN   TestRemoteClient_clientMD5
2019/04/11 07:01:27 [DEBUG] Instantiate OSS client using endpoint: "https://oss-cn-beijing.aliyuncs.com"
2019/04/11 07:01:28 [DEBUG] Current workspace name: default. All workspaces:[]string{"default"}
2019/04/11 07:01:28 [DEBUG] Recoring state serial in tablestore: &tablestore.PutRowChange{TableName:"tfRemoteTestForce5cae75c7", PrimaryKey:(*tablestore.PrimaryKey)(0xc00042bb80), Columns:[]tablestore.AttributeColumn{tablestore.AttributeColumn{ColumnName:"LockID", Value:"tf-remote-oss-test-5cae75c7/testState/terraform.tfstate-md5", Timestamp:0}, tablestore.AttributeColumn{ColumnName:"Digest", Value:"098f6bcd4621d373cade4e832627b4f6", Timestamp:0}}, Condition:(*tablestore.RowCondition)(0xc00042bba0), ReturnType:0, TransactionId:(*string)(nil)}
2019/04/11 07:01:29 [DEBUG] Retrieving state serial in tablestore: &tablestore.SingleRowQueryCriteria{ColumnsToGet:[]string{"LockID", "Digest"}, TableName:"tfRemoteTestForce5cae75c7", PrimaryKey:(*tablestore.PrimaryKey)(0xc00042bfa0), MaxVersion:1, TimeRange:(*tablestore.TimeRange)(nil), Filter:tablestore.ColumnFilter(nil), StartColumn:(*string)(nil), EndColumn:(*string)(nil), TransactionId:(*string)(nil)}
2019/04/11 07:01:29 [DEBUG] Deleting state serial in tablestore: &tablestore.DeleteRowRequest{DeleteRowChange:(*tablestore.DeleteRowChange)(0xc0003669c0)}
2019/04/11 07:01:29 [DEBUG] Retrieving state serial in tablestore: &tablestore.SingleRowQueryCriteria{ColumnsToGet:[]string{"LockID", "Digest"}, TableName:"tfRemoteTestForce5cae75c7", PrimaryKey:(*tablestore.PrimaryKey)(0xc000407da0), MaxVersion:1, TimeRange:(*tablestore.TimeRange)(nil), Filter:tablestore.ColumnFilter(nil), StartColumn:(*string)(nil), EndColumn:(*string)(nil), TransactionId:(*string)(nil)}
--- PASS: TestRemoteClient_clientMD5 (2.26s)
    client_test.go:174: TestBackendConfig on *oss.Backend with configs.synthBody{Filename:"<TestWrapConfig>", Values:map[string]cty.Value{"tablestore_endpoint":cty.StringVal("https://tf-oss-remote.cn-hangzhou.ots.aliyuncs.com"), "tablestore_table":cty.StringVal("tfRemoteTestForce5cae75c7"), "bucket":cty.StringVal("tf-remote-oss-test-5cae75c7"), "prefix":cty.StringVal("testState")}}
=== RUN   TestRemoteClient_stateChecksum
2019/04/11 07:01:29 [DEBUG] Instantiate OSS client using endpoint: "https://oss-cn-beijing.aliyuncs.com"
2019/04/11 07:01:30 [DEBUG] Current workspace name: default. All workspaces:[]string{"default"}
2019/04/11 07:01:30 [DEBUG] Instantiate OSS client using endpoint: "https://oss-cn-beijing.aliyuncs.com"
2019/04/11 07:01:30 [DEBUG] Current workspace name: default. All workspaces:[]string{"default"}
2019/04/11 07:01:30 [DEBUG] Retrieving state serial in tablestore: &tablestore.SingleRowQueryCriteria{ColumnsToGet:[]string{"LockID", "Digest"}, TableName:"tfRemoteTestForce5cae75c9", PrimaryKey:(*tablestore.PrimaryKey)(0xc00045b1e0), MaxVersion:1, TimeRange:(*tablestore.TimeRange)(nil), Filter:tablestore.ColumnFilter(nil), StartColumn:(*string)(nil), EndColumn:(*string)(nil), TransactionId:(*string)(nil)}
2019/04/11 07:01:31 [WARN] failed to fetch state md5: invalid md5
2019/04/11 07:01:31 [DEBUG] Recoring state serial in tablestore: &tablestore.PutRowChange{TableName:"tfRemoteTestForce5cae75c9", PrimaryKey:(*tablestore.PrimaryKey)(0xc00045b460), Columns:[]tablestore.AttributeColumn{tablestore.AttributeColumn{ColumnName:"LockID", Value:"tf-remote-oss-test-5cae75c9/testState/terraform.tfstate-md5", Timestamp:0}, tablestore.AttributeColumn{ColumnName:"Digest", Value:"6c34aec414b0c5a5358b5ef35a047d13", Timestamp:0}}, Condition:(*tablestore.RowCondition)(0xc00045b480), ReturnType:0, TransactionId:(*string)(nil)}
2019/04/11 07:01:31 [DEBUG] Retrieving state serial in tablestore: &tablestore.SingleRowQueryCriteria{ColumnsToGet:[]string{"LockID", "Digest"}, TableName:"tfRemoteTestForce5cae75c9", PrimaryKey:(*tablestore.PrimaryKey)(0xc000447920), MaxVersion:1, TimeRange:(*tablestore.TimeRange)(nil), Filter:tablestore.ColumnFilter(nil), StartColumn:(*string)(nil), EndColumn:(*string)(nil), TransactionId:(*string)(nil)}
2019/04/11 07:01:31 [WARN] state md5 mismatch: expected '6c34aec414b0c5a5358b5ef35a047d13', got ''
2019/04/11 07:01:31 [DEBUG] Retrieving state serial in tablestore: &tablestore.SingleRowQueryCriteria{ColumnsToGet:[]string{"LockID", "Digest"}, TableName:"tfRemoteTestForce5cae75c9", PrimaryKey:(*tablestore.PrimaryKey)(0xc00047d280), MaxVersion:1, TimeRange:(*tablestore.TimeRange)(nil), Filter:tablestore.ColumnFilter(nil), StartColumn:(*string)(nil), EndColumn:(*string)(nil), TransactionId:(*string)(nil)}
2019/04/11 07:01:31 [WARN] state md5 mismatch: expected '6c34aec414b0c5a5358b5ef35a047d13', got '5c62ebda9125c2352f91d501e61cf932'
2019/04/11 07:01:31 [DEBUG] Retrieving state serial in tablestore: &tablestore.SingleRowQueryCriteria{ColumnsToGet:[]string{"LockID", "Digest"}, TableName:"tfRemoteTestForce5cae75c9", PrimaryKey:(*tablestore.PrimaryKey)(0xc000447e00), MaxVersion:1, TimeRange:(*tablestore.TimeRange)(nil), Filter:tablestore.ColumnFilter(nil), StartColumn:(*string)(nil), EndColumn:(*string)(nil), TransactionId:(*string)(nil)}
2019/04/11 07:01:31 [WARN] state md5 mismatch: expected '6c34aec414b0c5a5358b5ef35a047d13', got '5c62ebda9125c2352f91d501e61cf932'
2019/04/11 07:01:31 [INFO] retrying OSS RemoteClient.Get...
2019/04/11 07:01:31 [DEBUG] Retrieving state serial in tablestore: &tablestore.SingleRowQueryCriteria{ColumnsToGet:[]string{"LockID", "Digest"}, TableName:"tfRemoteTestForce5cae75c9", PrimaryKey:(*tablestore.PrimaryKey)(0xc000438220), MaxVersion:1, TimeRange:(*tablestore.TimeRange)(nil), Filter:tablestore.ColumnFilter(nil), StartColumn:(*string)(nil), EndColumn:(*string)(nil), TransactionId:(*string)(nil)}
--- PASS: TestRemoteClient_stateChecksum (2.31s)
    client_test.go:224: TestBackendConfig on *oss.Backend with configs.synthBody{Filename:"<TestWrapConfig>", Values:map[string]cty.Value{"bucket":cty.StringVal("tf-remote-oss-test-5cae75c9"), "prefix":cty.StringVal("testState"), "tablestore_endpoint":cty.StringVal("https://tf-oss-remote.cn-hangzhou.ots.aliyuncs.com"), "tablestore_table":cty.StringVal("tfRemoteTestForce5cae75c9")}}
    client_test.go:257: TestBackendConfig on *oss.Backend with configs.synthBody{Filename:"<TestWrapConfig>", Values:map[string]cty.Value{"bucket":cty.StringVal("tf-remote-oss-test-5cae75c9"), "prefix":cty.StringVal("testState")}}
PASS
ok      github.com/hashicorp/terraform/backend/remote-state/oss 14.064s

TF_ACC=1 go test -v ./backend/remote-state/oss/ -run=TestBackend -timeout 120m
=== RUN   TestBackend_impl
--- PASS: TestBackend_impl (0.00s)
=== RUN   TestBackendConfig
2019/04/11 07:04:24 [DEBUG] Instantiate OSS client using endpoint: "https://oss-cn-beijing.aliyuncs.com"
--- PASS: TestBackendConfig (1.41s)
    backend_test.go:43: TestBackendConfig on *oss.Backend with configs.synthBody{Filename:"<TestWrapConfig>", Values:map[string]cty.Value{"region":cty.StringVal("cn-beijing"), "tablestore_endpoint":cty.StringVal("https://terraformstate.cn-beijing.ots.aliyuncs.com"), "tablestore_table":cty.StringVal("TableStore"), "bucket":cty.StringVal("terraform-backend-oss-test"), "key":cty.StringVal("first.tfstate"), "prefix":cty.StringVal("mystate")}}
=== RUN   TestBackendConfig_invalidKey
--- PASS: TestBackendConfig_invalidKey (0.00s)
=== RUN   TestBackend
2019/04/11 07:04:24 [DEBUG] Instantiate OSS client using endpoint: "https://oss-cn-beijing.aliyuncs.com"
2019/04/11 07:04:24 [DEBUG] Instantiate OSS client using endpoint: "https://oss-cn-beijing.aliyuncs.com"
2019/04/11 07:04:25 [DEBUG] Current workspace name: default. All workspaces:[]string{"default"}
2019/04/11 07:04:25 [DEBUG] Current workspace name: foo. All workspaces:[]string{"default"}
2019/04/11 07:04:25 [DEBUG] Current workspace name: bar. All workspaces:[]string{"default", "foo"}
2019/04/11 07:04:25 [DEBUG] Current workspace name: foo. All workspaces:[]string{"default", "bar", "foo"}
2019/04/11 07:04:25 [DEBUG] Current workspace name: bar. All workspaces:[]string{"default", "bar", "foo"}
2019/04/11 07:04:25 [DEBUG] Deleting remote state from OSS: "multi/level/path/foo/terraform.tfstate"
2019/04/11 07:04:25 [DEBUG] Current workspace name: foo. All workspaces:[]string{"default", "bar"}
2019/04/11 07:04:25 [DEBUG] Deleting remote state from OSS: "multi/level/path/foo/terraform.tfstate"
2019/04/11 07:04:25 [DEBUG] Current workspace name: default. All workspaces:[]string{"default", "bar"}
2019/04/11 07:04:25 [DEBUG] Current workspace name: default. All workspaces:[]string{"default", "bar"}
2019/04/11 07:04:25 [DEBUG] Current workspace name: default. All workspaces:[]string{"default", "bar"}
2019/04/11 07:04:25 [DEBUG] Current workspace name: default. All workspaces:[]string{"default", "bar"}
2019/04/11 07:04:25 [DEBUG] Current workspace name: default. All workspaces:[]string{"default", "bar"}
2019/04/11 07:04:25 [DEBUG] Current workspace name: default. All workspaces:[]string{"default", "bar"}
--- PASS: TestBackend (1.93s)
    backend_test.go:89: TestBackendConfig on *oss.Backend with configs.synthBody{Filename:"<TestWrapConfig>", Values:map[string]cty.Value{"prefix":cty.StringVal("multi/level/path/"), "bucket":cty.StringVal("terraform-remote-oss-test-5cae7678")}}
    backend_test.go:94: TestBackendConfig on *oss.Backend with configs.synthBody{Filename:"<TestWrapConfig>", Values:map[string]cty.Value{"bucket":cty.StringVal("terraform-remote-oss-test-5cae7678"), "prefix":cty.StringVal("multi/level/path/")}}
    backend_test.go:103: TestBackend: testing state locking for *oss.Backend
    backend_test.go:103: TestBackend: *oss.Backend: empty string returned for lock, assuming disabled
    backend_test.go:104: TestBackend: testing state locking for *oss.Backend
    backend_test.go:104: TestBackend: *oss.Backend: empty string returned for lock, assuming disabled
PASS
ok      github.com/hashicorp/terraform/backend/remote-state/oss 3.351s

xiaozhu36 · 2019-04-12T12:31:05Z

HI @jbardin This is just a reminding that if there is no any questions, please merge it. Thanks a lot.

apparentlymart · 2019-04-12T22:48:46Z

Hi @xiaozhu36,

@jbardin is currently on vacation, and will review this again when he returns.

Because the release process for Terraform v0.12.0 is already in progress, this cannot be merged until after v0.12.0 final is released and any urgent bugs have been addressed in minor releases, but once @jbardin has approved it we'll merge it for a v0.12.x minor release. The "v0.12.1" milestone is a marker for us to revisit this once v0.12.0 is stable; it may actually be a later minor release if we need to make urgent minor releases to address bugs in v0.12.0, but either way we'll use that milestone to mark this for inclusion once the major release is stable.

xiaozhu36 · 2019-04-15T10:11:29Z

HI @apparentlymart Thanks for you feedback. There are many customers are waiting for this PR. I hope this PR can be merged and published asap. If there is a minor release can support it. I think it also can be accepted.

paolomainardi · 2019-05-16T20:13:10Z

Are there any news about merging this MR ? cc @jbardin

bittopaz · 2019-06-14T04:02:07Z

This is included in 0.12.2, but the documentation for OSS backend seems not live on https://www.terraform.io/docs/backends/types/index.html

toamitkumar · 2019-06-15T11:12:41Z

this is great, thanks for the push guys :)

ghost · 2019-07-25T02:03:45Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

xiaozhu36 force-pushed the master branch 4 times, most recently from 75d5061 to 4a12685 Compare December 15, 2017 13:28

jbardin self-assigned this Dec 15, 2017

apparentlymart added new-backend enhancement labels Dec 18, 2017

xiaozhu36 force-pushed the master branch from 4a12685 to 0f65885 Compare December 20, 2017 06:11

jbardin requested changes Feb 21, 2018

View reviewed changes

xiaozhu36 force-pushed the master branch 2 times, most recently from fa1096c to 7cf5584 Compare April 16, 2018 05:25

xiaozhu36 force-pushed the master branch from 7cf5584 to 7fb4ba3 Compare May 16, 2018 02:29

xiaozhu36 force-pushed the master branch from 7fb4ba3 to f7990b1 Compare May 29, 2018 11:43

xiaozhu36 force-pushed the master branch 2 times, most recently from 0826485 to 8bb6ebc Compare August 9, 2018 12:35

xiaozhu36 force-pushed the master branch from 8bb6ebc to e292e98 Compare August 31, 2018 12:32

xiaozhu36 force-pushed the master branch 5 times, most recently from 66c7c72 to 7e6bb98 Compare October 27, 2018 09:08

xiaozhu36 force-pushed the master branch from 9338629 to 6ca906c Compare March 1, 2019 04:12

jbardin reviewed Mar 6, 2019

View reviewed changes

xiaozhu36 force-pushed the master branch from 6ca906c to 3584b13 Compare March 7, 2019 10:32

xiaozhu36 force-pushed the master branch 2 times, most recently from 002eb2d to 7fcbfb1 Compare April 3, 2019 13:41

xiaozhu36 mentioned this pull request Apr 3, 2019

Support using OSS as a backend aliyun/terraform-provider-alicloud#944

Closed

jbardin approved these changes Apr 10, 2019

View reviewed changes

xiaozhu36 added 2 commits April 11, 2019 07:06

Add Alibaba Cloud backend OSS with lock

b887d44

Add tablestore config to store state lock

3f44dd9

xiaozhu36 force-pushed the master branch from 7fcbfb1 to 3f44dd9 Compare April 10, 2019 23:11

apparentlymart added this to the v0.12.1 milestone Apr 11, 2019

pselle modified the milestones: v0.12.1, TBD Jun 4, 2019

Merge branch 'master' into master

0c91d22

jbardin merged commit ef42be7 into hashicorp:master Jun 10, 2019

ghost locked and limited conversation to collaborators Jul 25, 2019

Add Alibaba Cloud backend OSS with lock #16927

Add Alibaba Cloud backend OSS with lock #16927

Conversation

xiaozhu36 commented Dec 15, 2017

jbardin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

georgedriver commented Mar 19, 2018

georgedriver commented Apr 17, 2018

xiaozhu36 commented Apr 17, 2018

heatmiser commented May 19, 2018

xiaozhu36 commented May 31, 2018

jbardin commented May 31, 2018

rouralberto commented Jul 5, 2018

JennyLi90 commented Aug 9, 2018

jbardin commented Aug 16, 2018

toamitkumar commented Mar 4, 2019

xiaozhu36 commented Mar 6, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiaozhu36 commented Apr 8, 2019

jbardin commented Apr 10, 2019

xiaozhu36 commented Apr 10, 2019

xiaozhu36 commented Apr 12, 2019

apparentlymart commented Apr 12, 2019

xiaozhu36 commented Apr 15, 2019 • edited Loading

paolomainardi commented May 16, 2019

bittopaz commented Jun 14, 2019

toamitkumar commented Jun 15, 2019

ghost commented Jul 25, 2019

xiaozhu36 commented Apr 15, 2019 •

edited

Loading