[IDEA] json operators for array manipulation: insert, remove, push, pop

[rmunn]

Is your feature request related to a problem? Please describe.
The new jsonset operator allows modifying an array at a specific index, but doesn't easily allow operations that would change the length of the array: adding or removing an item, for example. In order to insert an item into a JSON array, you'd first have to turn it into a TW-style title list with jsonget, then manipulate the list, then do a rather complicated operation to turn that list back into JSON format, then finally use jsonset:json to put it back in place.

Describe the solution you'd like
It would be far nicer to have the following operators available:

• jsoninsert - takes a series of indexes (like jsonset does), which must lead to an array and a specific index i within that array. The item to be inserted is then placed before i. (If i is equal to the length of the array, then the item gets appended to the end of the array, just like what jsonpush below does). Returns the new JSON value with the item inserted.
• jsonpush - takes a series of indexes which must lead to an array. The item to be inserted is appended to the end of the array. Returns the new JSON value with the item inserted.
• jsonremove - like jsoninsert, but removes the item at index i. Returns the new JSON array with the item removed. If you need the value of the item, do jsonget or jsonextract first before doing jsonremove.
• jsonpop - like jsonpush, but removes the last item of the array. Returns the new JSON array with the item removed. If you need the value of the item, do jsonget or jsonextract first before doing jsonremove. Note that negative indexes (as per [IDEA] Allow negative indexes in jsonget/set/etc operators for array access from end #7839) should help here: [<jsonarray>jsonget[-1]] would fetch the last item.

Describe alternatives you've considered
Technically all of these can be done already using jsonget and jsonset, but the filter operations to achieve that result would be quite tedious to write, and rather inefficient to boot.

Additional context
In #7742 (comment) there was some discussion about whether jsonpop should return two results: the removed value, and the resulting list. I think it's best to have jsonpop just return a single value, which is the list minus its last item; getting the value of the last item would be done with jsonget[-1] first, and the jsonpop operation would be a separate step. That would avoid the complication of handling the two return values in a filter run. (You could get at one of them with nth, but what would you do with the other one? You'd end up having to store the result in a variable anyway).

[rmunn]

One design decision not yet considered: what to do if the index parameter(s) wind up selecting an object that isn't a JSON array? I think the correct answer is to use the same behavior that jsonset uses on invalid input (such as setting a Boolean value that isn't "true" or "false"): return the original JSON string unchanged.

[Jermolene]

Thanks @rmunn those operators would appear to be a solid set of elementary primitives. They don't address all use cases, so I wonder if it's worth trying to map out a bit more of the problem space before we commit to this design.

For example, I can imagine that it might be useful to be able to push all the input list items onto an array. Perhaps that would be a two step process: first make a JSON array from the input list, and then push that array onto another array within a JSON object.

Perhaps a good way forwards might be a PR that starts with just the documentation for the new operators. Then we can collaborate on adding further operators.

We don't need to implement all these operators immediately but I do think it would be useful to have a clearer sense of what other features we're going to need.

My reasoning here is that perhaps there are demanding requirements that can only be satisfied by inventing a new mechanism (eg a minor extension to the filter syntax). It would be useful to have sight of that possibility early, because that maximises the chances that we'll be able to reuse that extension to the filter syntax elsewhere.

[rmunn]

Sure, we can take some time to think about the design more. In that case I won't try to get this ready for 5.3.2, because rushing it could lock us into a design that wasn't fully baked. It can easily wait until 5.3.3.

Yes, I imagine that adding two arrays together is going to be a pretty common need.

One idea I just had for expanding the filter syntax is to allow some sort of filter run prefix that goes at the very beginning of the filter expression, like ":lazy" or ":nodedupe", that would affect the entire expression. I.e., the filter "1 2 3 2" uses dominant append and produces "1 3 2", but the filter ":nodedupe 1 2 3 2" produces "1 2 3 2". Such prefixes would be "option prefixes", would not have a filter run attached, and would only be allowed at the start of an expression, much like how pragmas work.

This would involve a fairly large rewrite of the filter code base, but not too large: the options parameter that's passed to every filter would be an ideal place to put things like options.dedupe = false. The tricky part would be that every filter operator would need to be rewritten to use the options parameter, because options.dedupe affects whether they should use push or pushTop.

But that's getting a bit far afield from json operators. Over the next few days, I'll see about creating a new feature request issue to track that idea about "filter option prefixes". Meanwhile, I'll hold off on working on the JSON operators until we have a bit more time to hash out the design.

[Jermolene]

Sure, we can take some time to think about the design more. In that case I won't try to get this ready for 5.3.2, because rushing it could lock us into a design that wasn't fully baked. It can easily wait until 5.3.3.

Great, thanks.

One idea I just had for expanding the filter syntax is to allow some sort of filter run prefix that goes at the very beginning of the filter expression, like ":lazy" or ":nodedupe", that would affect the entire expression.

Yes, I think that makes us an effective escape hatch, and I think your observations are correct.

[AnthonyMuscio]

I really like this idea, a comprehensive set, coming from a user perspective I wonder if we could use other names or provide a set of aliases for each operator that uses more plain language for manipulating the JSON arrays?

The $params attribute now on the parameters widget is making an array more accessible to users. For example we could use this to store a list of fieldnames and filters to apply to each fieldname and use this to drive more complex actions against tiddlers. Similar to set multiple fields widget.

• That is, as the ability to manipulate such JSON arrays improves, I think we can add additional support so they can be made use of more extensively.

• jsonpush / jsonappend

  • jsonprepend

• Perhaps to get a value and remove it jsonpull ?

• Access via name or index number.

It would also help if we could get the total items, set a position and do a jsonnext or jsonprev

Defaults to current tiddler makes sense but what if we could provide a tiddler name as well if needed?