Introduce jsonset operator

[Jermolene]

This PR introduces a new jsonset operator that can set property values within JSON strings.

Properties within a JSON object are identified by a sequence of indexes. In the following example, the value at [a] is one, and the value at [d][f][0] is five.

{
    "a": "one",
    "b": "",
    "c": "three",
    "d": {
        "e": "four",
        "f": [
            "five",
            "six",
            true,
            false,
            null
        ],
        "g": {
            "x": "max",
            "y": "may",
            "z": "maize"
        }
    }
}

The following examples assume that this JSON data is contained in a variable called jsondata.

The jsonset operator uses multiple operands to specify the indexes of the property to set. When used to assign strings the final operand is interpreted as the value to assign. For example:

[<jsondata>jsonset[d],[Jaguar]] --> {"a": "one","b": "","c": "three","d": "Jaguar"}
[<jsondata>jsonset[d],[f],[Panther]] --> {"a": "one","b": "","c": "three","d": "{"e": "four","f": "Panther","g": {"x": "max","y": "may","z": "maize"}}"}

Indexes can be dynamically composed from variables and transclusions:

[<jsondata>jsonset<variable>,{!!field},[0],{CurrentResult}]

The data type of the value to be assigned to the property can be specified with an optional suffix:

• string - The string is specified as the final operand
• boolean - The boolean value is true if the final operand is the string "true" and false if the final operand is the string "false". Any other value for the final string results prevents the property from being assigned
• number - The numeric value is taken from the final operand. Invalid numbers are interpreted as zero
• json - The JSON string value is taken from the final operand. Invalid JSON prevents the property from being assigned
• object - An empty object is assigned to the property. The final operand is not used as a value
• array - An empty array is assigned to the property. The final operand is not used as a value
• null - The special value null is assigned to the property. The final operand is not used as a value

For example:

Input string:
{"a":"one","b":"","c":1.618,"d":{"e":"four","f":["five","six",true,false,null]}}

[jsonset[]] --> {"a":"one","b":"","c":1.618,"d":{"e":"four","f":["five","six",true,false,null]}}
[jsonset[],[Antelope]] --> "Antelope"
[jsonset:number[],[not a number]] --> 0
[jsonset[id],[Antelope]] --> {"a":"one","b":"","c":1.618,"d":{"e":"four","f":["five","six",true,false,null]},"id":"Antelope"}
[jsonset:notatype[id],[Antelope]] --> {"a":"one","b":"","c":1.618,"d":{"e":"four","f":["five","six",true,false,null]},"id":"Antelope"}
[jsonset:boolean[id],[false]] --> {"a":"one","b":"","c":1.618,"d":{"e":"four","f":["five","six",true,false,null]},"id":false}
[jsonset:boolean[id],[Antelope]] --> {"a":"one","b":"","c":1.618,"d":{"e":"four","f":["five","six",true,false,null]}}
[jsonset:number[id],[42]] --> {"a":"one","b":"","c":1.618,"d":{"e":"four","f":["five","six",true,false,null]},"id":42}
[jsonset:null[id]] --> {"a":"one","b":"","c":1.618,"d":{"e":"four","f":["five","six",true,false,null]},"id":null}
[jsonset:array[d],[f],[5]] --> {"a":"one","b":"","c":1.618,"d":{"e":"four","f":["five","six",true,false,null,[]]}}
[jsonset:object[d],[f],[5]] --> {"a":"one","b":"","c":1.618,"d":{"e":"four","f":["five","six",true,false,null,{}]}}
[jsonset[missing],[id],[Antelope]] --> {"a":"one","b":"","c":1.618,"d":{"e":"four","f":["five","six",true,false,null]}}

A subtlety is that the special case of a single operand sets the value of that operand as the new JSON string, entirely replacing the input object. If that operand is blank, the operation is ignored and no assignment takes place. Thus:

[<jsondata>jsonset[Panther]] --> "Panther"
[<jsondata>jsonset[]] --> {"a": "one","b": "","c": "three","d": "{"e": "four","f": ["five", "six", true, false, null],"g": {"x": "max","y": "may","z": "maize"}}"}

(This PR was cherry picked from #7406)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
tiddlywiki5	✅ Ready (Inspect)	Visit Preview	Sep 15, 2023 11:17am

[rmunn]

One operation that will be pretty common, but would be very difficult (not impossible, just convoluted) with this design, is appending one item to a JSON array, i.e. the equivalent of arr.push(newValue) in Javascript. It would require using jsonindexes to get the indexes of the array, then using the count operator to count how many there are, assigning the count to a variable idx. Then finally [jsonset:string[d],[f],<idx>,[newValue]] would append the string "newValue" to the end of the array.

An alternative approach would involve using jsonextract, stripping off the final ], adding a comma and the value desired (as a pre-formatted JSON string) and then putting the final ] back, then passing that in using jsonset:json[d],[f],<newArray>.

Either one of these is convoluted. Would it be possible to get a design for jsonset where you can write jsonset:string[d],[f],[],[newValue] in order to append `"newValue" to the array? I.e. an empty operand means "if this is an array, take the array's current length and treat that as the index to set; if this is not an array, then abort and return the original JSON value unchanged". That would not require much code changes from what you've already written AFAICT, and would allow appending values to existing arrays without too much hassle.

Alternately, a new jsonappend operator might make more sense, that appends values to an array. Though that would then immediately make me want jsoninsert and jsonremove operators as well, for inserting and removing values from the array, both of which would be a little convoluted to do with just jsonset as-is. So instead of introducing jsonappend, perhaps a simple rule that "[] means append at the end of the array" is the best approach.

[Jermolene]

Hi @rmunn I was thinking that we'd make a separate jsonpush operator which I guess would be equivalent to your jsonappend operator. I agree that we'd also need a bunch of other array operations, but I think that's OK because it's consistent with what we see in other high level languages.

[rmunn]

jsonpush sounds good, better than jsonappend actually. There's one other thing I'll probably want to ask for, but it's probably best as a separate issue because it will need to apply to jsonget and the other json* functions as well. My request is negative indexing for arrays: -1 means last item, -2 means next-to-last item, and so on. Basically, in the getDataItem function, before doing item = item[indexes[i]] you would check if indexes[i] is a negative number. If it's negative and item is an array, then the index that's fetched is item.length + indexes[i]. (No such check for objects, because someone might have an object with a -1 property).

The rationale for this is because it's quite useful in lots of situations to be able to access the last item of an array without knowing ahead of time how long the array is. (Indexes like -2 are useful a lot less often, but if you've already added the -1 feature than why not allow all negative numbers as indexes?) One place it might be useful, for example, is once a jsonremove operator is added. Someone might want to treat a JSON array as a stack, and it's most efficient to add and remove at the end. jsonpush would take care of adding at the end, but removing from the end would be best done with either jsonremove[-1] or else a jsonpop operator. If we allow jsonremove[-1] then jsonpop would not be needed.

Anyway, I'll create a new issue to track that suggestion, because it would involve changing all the json operators, not just jsonset. (Though if negative indexes are allowed, I'd like them to apply to jsonset as well, naturally.)

[Jermolene]

Hi @rmunn I like the idea of negative indices for accessing arrays from the end, and agree that it should be consistently applied to all the JSON operators.

A slight tangent, but it is interesting to consider a jsonpop operator. The difficulty is the way that we want to return both the modified array and the item that was pulled from it. I wonder if it would be confusing if the operator returned two results. One would be able to access them with nth[], perhaps.

[AnthonyMuscio]

Have come from the pre-release. It is great there is the addition of the JSONset operator.

Since 5.3.x we have had the introduction of the parameters widget, we also got the $params=vaname attribute which stores all the parameters ans a JSON array of name/value pairs.

Would it be possible to document the relationship and interoperability of the jsonset operator with the array generated by the $params attribute?

For may users the parameters widget may be the first time they encounter a JSON array and may be inclined to explore the use of json operators with the parameters array.

I ask here, rather than wait to contribute to the documentation because I need a developer to tell me if the above is interoperable (although I suspect it is)

Post Script

• I see the possibility of using parameter arrays, for some mechanisms I have long wanted to build to allow a suite of actions to be stored as parameters in a field or variable for application when needed.
• among other possibilities such as storing named filters.