JMESPath Community Features 1.1 Brainstorming

[springcomp]

This discussion started to list topics that could be included in a next version of JMESPath but still feels a bit abstract or not sufficiently scoped.

A new discussion has been started to define the very next short term milestone.

Here are the topics we currently have:

• Lexical Scoping
  • JEP-11 Lexical Scoping
  • JEP-17 Root Reference. Overloading the $ operator?
• Miscellaneous Topics
  • Third-Party Functions (wording in progress)
  • any(), all() functions.
  • group_by() function.
  • range() function.
  • to_set() function.
  • try() function.
  • Support some form of regular expressions.
  • Compact-style for multi-select-hash.
  • Reusable expressions
  • Dynamic slice values
  • Dynamic object subscription
• Arithmetic operators:
  • Simple numeric operations.
  • Set operations.
• Functional programming operations:
  • reduce operation.
• String manipulation functions:
  • slice() function accepting string arguments.
  • split() function (does this conflict with slice() accepting string arguments?
  • format() function.
  • trim_left() and trim_right() functions. Variations could be named trim_start() or trim_end(). Do we need a standalone trim() function?
  • pad_left() and pad_right() functions.
  • upper() and lower() functions. How to handle Unicode characters, for instance, Turkish ı->I, vs i->İ?
  • title() function to change case. Could be expanded with camel(), snake(), etc. May seem a bit specific.
• Object manipulation functions
  • JSON functions specifying parse_json() and to_json() functions.
  • spread operator.
  • items(), from_items() and zip() functions. In particular, from_items() would be the favored name for the known to_object() function.

Those topics lead to discussions that reveal issues around structural behaviour of JMESPath. Those could lead to breaking changes.

• Should we have projections not discard null values after all? Or produce error values ?
• Should we drop backtick literals?
• Should we support true, false and null keywords that have the same meaning as their JSON counterparts? This impacts regular identifiers.
• Should we deprecate * vs [*] which are identical apart from the type of the object to which they apply? Should we remove one of the two constructs altogether ?
• Should we remove some redundant functions (e.g map($expr, $array) is equivalent to $array[*].$expr)? Or just deprecate those functions?
• Should we re-order arguments to some functions (e.g map() and join()) for consistency?
• Should we strive to introduce functions or focus and improving syntax when both are possible?

Please, note that JEP numbers used here are subject to change. The anticipated workflow is to start a discussion on a topic, then proceed to assign a JEP number when this is sufficiently fleshed out.

[innovate-invent]

I think we should divide the currently proposed changes into backwards compatible and backwards incompatible.
Crank out a version that is backwards compatible and then immediately abandon it for backwards incompatible changes.

For example, try() is somewhat useless given the current behaviour of returning null for an invalid expression.
There is also this subtle behavior of projections skipping null values, resulting in smaller output arrays compared to the input. I think we should eliminate that behavior and remove the map() function.

Array wildcard expressions ([*]) should be dropped by simply defining the unbracketed version * on an array. I can't see any value for having these separate syntax's for the different input types.

We could also drop literals, keys(), values(), merge(), sum(), reverse(), min_by() and max_by().

• map(&expr, input) is effectively being replaced with input[*].expr
• merge() with the + operator
• keys() with *.$
• values() with *.@
• min_by(&expr) and max_by(&expr) can be replaced with [&].(@.expr < $.expr && @) || $

I think some function operators should be reordered to be consistent. For example, contains() and ends_with() have what is normally the static value as the second argument, but join() has it as the first.

[springcomp]

I think *.@ is currently not working on an input JSON object.
The equivalent to the values() function would be more akin to the simple * projection.

[innovate-invent]

Hm, I am surprised that @ isn't generally useable. You are right though, I was just using *.@ to be more illustrative.

[springcomp]

Literals

I think dropping literals completely would be detrimental to expression some expressions with static JSON content.

min_by and max_by

I think we touch on an issue around ease of use and readibility.
I must admin I struggle to grok the following expression at first sight: [&].(@.expr < $.expr && @) || $.

If we go back to replacing map() with a simple projection, the resulting expression is still straightforward.
I’m torn with the principle to insist on only one way of expressing something in JMESPath.

The only compelling example that I see is the sum() function which is quintessentially a behaviour that can be expressed using a reduce primitive.

Argument ordering

I have listed the signatures for all functions with more than one argument:

   boolean contains( array|string $subject       , any $search )
  boolean ends_with( string $subject             , string $prefix )
        string join( string $glue                , array[string] $elements )
     array[any] map( expression $expr            , array[any] $elements )
         any max_by( array $elements             , expression->number|expression->string $expr )
       object merge( object $argument            , object $.. )
         any min_by( array $elements             , expression->number|expression->string $expr )
       any not_null( any $argument               , any $.. )
      array sort_by( array $elements             , expression->number|expression->string $expr )
boolean starts_with( string $subject             , string $prefix )

Apart from functions with a variadic number of arguments, only two functions indeed stand out map and join.
Here is what it would look like if we re-ordered their input arguments:

   boolean contains( array|string $subject       , any $search )
  boolean ends_with( string $subject             , string $prefix )
        string join( array[string] $elements     , string $glue )
     array[any] map( array[any] $elements        , expression->any $expr)
         any max_by( array $elements             , expression->number|expression->string $expr )
       object merge( object $argument            , object $.. )
         any min_by( array $elements             , expression->number|expression->string $expr )
       any not_null( any $argument               , any $.. )
      array sort_by( array $elements             , expression->number|expression->string $expr )
boolean starts_with( string $subject             , string $prefix )

That would however be a breaking change but I think you already alluded to that.

Projections and null values

I’m torn with changing the behaviour of projections that drop null values.
Also, touching on the subject of readibility and ease of use refererred to above, I think we may use functions to explicitely apply to arrays without dropping null values.

Thus, even though values() can be expressed with projection instead, I think the distinct behaviour of both warrant keeping the function.

Considering the following input object

{
  "a": "a",
  "b": "b",
  "c": "c",
  "null": null
}

We would have:

search( values(@), { "a": "a", "b": "b", "c": "c", "null": null} ) -> ["a", "b", "c", null]
search( *,         { "a": "a", "b": "b", "c": "c", "null": null} ) -> ["a", "b", "c"]

[innovate-invent]

I’m torn with the principle to insist on only one way of expressing something in JMESPath.

I agree, this was more of an exploration of that. Perhaps we should focus on dropping functions that would create confusion with other syntax like map(), but retain functions that simplify.

I’m torn with changing the behaviour of projections that drop null values.

I am actually quite concerned about this functionality. It is very subtle and when I learned of it I had a panic moment going back through past work to make sure that it wasn't affecting anything. I generally believe that an operation should do one thing, this is doing two. I think that the filter expression should be used to filter values *[?@]. Do you know what the argument for this functionality is?

[springcomp]

I can’t find any reference to the grammar before JEPs were introduced so I think projections were part of the grammar from day one.
I don’t know why this behaviour was chosen, but it does not bother me.

I think the simplest explaination is that you would not want to have downstream expressions applied to null values after a projection.

[innovate-invent]

In an application where null is significant, this forced filtering behaviour is problematic. map() only exists as an alternative to this filtering, it feels like a backwards compatible bandaid for a bad early decision.

[innovate-invent]

I found where James discusses the addition of the map() function: jmespath/jmespath.site#15