parse json string?

[ebarault]

Hi,

I'm using jsonencode() in combination with external data source to pass a json object as as string to an external program.

This external program also produces a json object which i'm able to pass back to terraform as a string.

Now I can't find a way to cast this stringified json object to a map. Somehow the jsondecode() seems to be missing.

Is there a way to do it?

[nbering]

The External data provider should parse your program's STDOUT as JSON. So... if I understand it correctly, your data resource should have an attribute result which contains a map parsed from the JSON data your program returned.

[ebarault]

hi @nbering,

I wish to pass back to terraform a json object as follows

[
  {
    foo: "bar",
    tar: ["baz", "fuz"]
  }
]

But, as per the doc here, result is expected to be "A map of string values returned from the external program".

And i can confirm that providing a more complex map raises the following type error :

data.external.flatten_users_groups: command "node" produced invalid JSON: json: cannot unmarshal array into Go value of type string.

This is why a provide back to Terraform a json stringified version of my object. But i can't find a way to parse it back to a Go object (a list of maps in this case)

[nbering]

Yes, that is a bit of a limitation. I can't say for certain why, but Terraform has a bit of a Schema layer built on top of the Go-lang type system. This does introduce some limitations in things you can do inside configuration. I'm not a Go-lang expert by any means, but my observation is that there is some limitation to Terraform's ability to evaluate certain types of nesting in arbitrary structures which is likely because they need to avoid accidental evaluation of incorrect types. I would guess that's why there is no JSON parsing function.

Plugins

My thought is this: configuration as code is not intended to be a general-purpose programming language. When you get to a certain level of complexity in your configuration, it may be advisable to encapsulate that complexity as a provider plugin. Doing so allows you to define a schema for the result, which makes it easier for Terraform core to understand what your values are.

In support of my opinion - from the External Provider documentation:

This provider is intended to be used for simple situations where you wish to integrate Terraform with a system for which a first-class provider doesn't exist. It is not as powerful as a first-class Terraform provider, so users of this interface should carefully consider the implications described on each of the child documentation pages (available from the navigation bar) for each type of object this provider supports.

And from the data resource documentation page:

Warning This mechanism is provided as an "escape hatch" for exceptional situations where a first-class Terraform provider is not more appropriate. Its capabilities are limited in comparison to a true data source, and implementing a data source via an external program is likely to hurt the portability of your Terraform configuration by creating dependencies on external programs and libraries that may not be available (or may need to be used differently) on different operating systems.

Work with the Limitations

The other (relatively) simple option if you really can't move away from the external data provider would be to convert your structure into flattened map with predictable key names using the program that returns the JSON data. For example, your above object would become:

{
  "root_0_foo": "bar",
  "root_0_tar_0_baz": "foo",
  "root_0_tar_0_bar": "baz"
}

This is clearly not very pretty - especially since your root element is an array - but it is one way of encoding complex structures into a simple map.

[ebarault]

thanks @nbering for taking the time on this !

first, i fixed my example which was not proper json.

plugins would be my last option since I don't know go much.

regarding the your other option, then how would convert it back to a valid terraform object so I can use built-in function on it?

i have also considered passing arrays as strings so I can split() them back in terraform. Consider this form:

[
  {
    foo: "bar",
    tar: "baz,fuz"
  }
]

A least i can use something like
tar = "${split(",", lookup(var.root[count.index], "tar"))}"
to build an array from the tar string,

But I'm not figuring out yet how I can iterate both on each items of the inner array, for all objects of the outer array.

[nbering]

No problem.

Generally speaking, if you're doing a lot of data manipulation in your configuration, you're falling into the anti-pattern I mentioned where you're trying to use a configuration file like a general-purpose language. You probably don't want to have a lot of variance in the structure of the object you're passing to Terraform. Having consistency in the object structure will allow you to sort of "statically" assign the values from the external data to the places they are needed and build back up the structure of your data there.

But I have to ask, "why do you need such a complex structure?" Are you trying to do this because it's the pattern you'd use in a programming language? There are different constraints here and you may need to look at adopting or developing new patterns to work in those constraints.

If you really need to work from arrays... one way to do that would be to have multiple external data resources that return a map, and then transpose the values to an array (I believe there's an interpolation function for that).

[ebarault]

Yes, you're right and I'm definitely aware of this anti-pattern paradigm, but I'm building reusable modules for several projects and you always face situations where you want the inputs you ask users to fill in to be as human-readable and DRY as possible, hence the json format.

That being said, i just figured a way to pass my object back to terraform in an iterable form, this would work in my situation only, but here it is:

Consider my initial object :

[
  {
    foo: "bar",
    tar: ["baz", "fuz"]
  }
]

...this can also be expressed as:

[
  {
    foo: "bar",
    tar: "baz"
  },
 {
    foo: "bar",
    tar: "fuz"
  }
]

With some imagination you can flatten this as:

"foo,bar,tar,baz,foo,bar,tar,fuz"

which you can read this in terraform as an array of arrays with

"${chunklist(split(",", lookup(data.external.my_script, "data")),2)}"

where my_script is my external script handling the conversion of my json object to its terraform's string representation.

and which creates:

[
  [ "foo", "bar" ],
  [ "tar", "baz" ],
  [ "foo", "bar" ],
  [ "tar", "fuz" ]
]

which is fairly easy to iterate on with Terraform as the inner arrays size is fix (key / value pairs).

Not ideal, but once packaged, it's not as bad as it looks since the internal logic is not visible by the module's end-user.

[apparentlymart]

Hi @ebarault!

I'm glad you figured out a way to get what you needed here. As @nbering mentioned, there are some limitations in Terraform's type system today that mean we can't support functions or attributes whose data type varies based on input, and so the result of the external data source is currently defined as being a map from strings to strings, and it's not possible to offer a jsondecode function because there's no way to express that its return type depends on the value passed to it.

We're currently in the process of integrating a new version of the configuration language parser and expression evaluator that doesn't have these limitations, so both of these constraints should be addressed eventually but there's quite a lot of work between here and there due to how much of Terraform Core needs to be updated to support the new type system.

#10363 is already covering the request for a jsondecode function, so we'll post further updates there when we have them. There isn't yet an issue for making the external provider support dynamically-typed results because the necessary core changes for that have not yet been implemented, but we do plan to do it once we are further along with the core work.

In the mean time, marshalling things as delimited strings, as you did here, is a common workaround.

[ebarault]

Hey @apparentlymart !,

Many thanks for all the team's effort to bring more scripting capabilities to terraform! I will stay tuned on #10363, this will be a great move forward.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.