Remote state management using git provider

[anujg]

We need a feature to store the remote state of terraform using git as a backend store , apart from s3 buckets. This would be really helpful as it gives us an extra and probably more secure option than s3.

[apparentlymart]

Hi @anujg! Thanks for this suggestion.

I want to make sure I understand properly your suggestion:

I think what you would like is for Terraform to make a commit and push to a remote git repository each time it needs to save a new version of the state. This would mean that each state update would have its own git commit.

I assume you'd want to keep many states in the same repository, rather than having a separate repository per state, and so that would imply that Terraform must preserve all existing files in the tree when it writes a new state, and that it must be defensive against conflicts where two different concurrent Terraform runs try to commit and push at the same time.

This would be a pretty different sort of backend than any of the others, due to git's decentralized model requiring a lot more state to be maintained client-side, so I don't expect we'd be able to work on such a thing for the foreseeable future, but it is an interesting idea so I'll leave this open here to remind us about it in the future.

[omeid]

I assume you'd want to keep many states in the same repository.

For me, just being able to use a single repository per state would be great.

Git being decentralized by design means that you get locking for free.

[dev-dull]

First, apologies if this is spammy.

The company I work for had a similar need and I created an HTTP backend that, among other things, can keep your state file in a Git repo. There's still a lot I want to add to it, but we've been using it for managing our prod and non-prod environments for the last 4 or 5 months and it's been pretty stable for us.

project
docs

[tdmalone]

Git being decentralized by design means that you get locking for free.

I don't think this is quite true... to stop someone else from pulling and working on state, wouldn't you have to make a commit to 'lock' the state, and then another one to 'unlock' it once you're done? You'd also need to pull before you lock and then have the lock fail if you couldn't push it back. If you run terraform plan regularly without any changes, you'll also end up with a lot of junk commits.

[omeid]

I haven't thought too much into it, but wouldn't one be able to pull changes on plan, and lock on apply?

[megakoresh]

@apparentlymart

I think what you would like is for Terraform to make a commit and push to a remote git repository each time it needs to save a new version of the state. This would mean that each state update would have its own git commit.

I assume you'd want to keep many states in the same repository, rather than having a separate repository per state, and so that would imply that Terraform must preserve all existing files in the tree when it writes a new state, and that it must be defensive against conflicts where two different concurrent Terraform runs try to commit and push at the same time.

This would be a pretty different sort of backend than any of the others, due to git's decentralized model requiring a lot more state to be maintained client-side, so I don't expect we'd be able to work on such a thing for the foreseeable future, but it is an interesting idea so I'll leave this open here to remind us about it in the future.

We could use this too. And what you describe seems too complicated. I would say for most people it would be enough to be able to specify a git url (http or ssh) to use for terraform_remote_state data source and thats it.

For example we already have all our terraform configurations in git. Whenever we update the state that automatically means a new commit.

Making sure there are no conflicts (i.e. someone else doesn't try to use your state before you push your changes, if you are in the process of making them) goes under standard working with VCS routines and shouldn't be in Terraform's scope to handle.

[apparentlymart]

Thanks for sharing the use-case @megakoresh.

It sounds like you're using Terraform in a non-standard way (state in version control alongside the configuration), but then finding that this means you lose the ability to retrieve another state using terraform_remote_state. Saving the remote state to somewhere other than version control -- and automatically acquiring locks to detect concurrent runs -- is considered part of Terraform's scope by design, but I understand that there are unusual situations where it's convenient to use Terraform in other ways. Some of Terraform's "normal" features may not be available to you as a result, though.

At present it's not possible to have a backend that only works with terraform_remote_state and cannot be used to also write that state. We may think about such a thing in the future, but I'd also note that in general we consider terraform_remote_state to be a convenience for users that are already using a remote state backend to exploit it for an additional purpose, but it's always possible to write data from Terraform into other places explicitly using pairs of resource types and data sources, like consul_key_prefix+consul_keys, or dns_cname_record_set+dns_cname_record. In many cases this is preferable because it reduces the coupling between the two configurations and potentially makes that data available to other non-Terraform consumers that are also able to read from it.

[AvnerCohen]

I wonder if a possible way to get some of the git/SCM features could be a totally separate non blocking cron that would check S3 backend (for example) for delta changes, and on changes, will pick them up and commit to git.
This means there is always one committer (this cron) and changes are easily visible and browsable in the git repo.
I wonder if this would satisfy majority of the need, without losing any terraform features or added complexity.

[dee-kryvenko]

You might be interested in this: https://github.com/plumber-cd/terraform-backend-git

[danieldreier]

I really appreciate this proposal, and I think there's been healthy discussion. I'm going to close this issue for 3 reasons:

1. @apparentlymart has explained why the ideas proposed aren't immediately feasible
2. A community-contributed version of this that uses the HTTP backend as the interface has been linked (above)
3. We've decided to put a pause on community-contributed state backends in the Terraform codebase until we are able to build a plugin system, because maintaining community-contributed state backends has been unsatisfactory for our users, open source contributors, and our engineers. I've written more about this in recently-closed PRs for cmd, TiKV, and MySQL state backends. @pkolyvas and I have also both spoken to the git-specific use case in Terraform backend: Git repository #24603.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.