proposal: add downcasts

[cueckoo]

Originally opened by @zombiezen in cuelang/cue#454

Is your feature request related to a problem? Please describe.
I have a configuration where I have a struct containing a mapping that is frequently edited. Different declarations across my mapping require different subsets of that mapping inside their struct, but frequently not the whole thing.

In this same configuration, I sometimes use template-like objects that have definitions and I want to convert them to a more basic type without the definitions. For example:

#Object: {
  name: string
}

#MyTemplate: {
  #foo: string
  name: "foo-\(#foo)"
}

// ERROR: does not unify
#Object & {
  #MyTemplate
  #foo: "bar"
}

In both cases, I have to write a rather verbose and hard-to-understand comprehension to get the desired effect:

#Object & {
  let #expanded = {
    #MyTemplate
    #foo: "bar"
  }
  for k,_ in #Object { "\(k)": #expanded[k] }
}

Describe the solution you'd like

I'd like a kind of "downcasting" conversion that removes any fields in a struct that aren't in another struct. I don't have an informed opinion about syntax, but borrowing from Go's type conversion syntax, something like:

#Object & #Object({
  #MyTemplate
  #foo: "bar"
})

Describe alternatives you've considered

As mentioned above, this is possible in the language already using comprehensions, but is not obvious to those who haven't already dived deep into CUE what this is doing. A lack of user-defined functions makes it difficult to abstract the operation, thus bringing me to ask for language support. I could do something like (untested):

#Downcast: {
  #value: {...}
  #type: {...}
  #output: { for k,_ in #type { "\(k)": #value[k] } }
}

#Object & #Downcast{
  #value: {
    #MyTemplate
    #foo: "bar"
  }
  #type: #Object
}.#output

But this doesn't add terribly much clarity IMO, it just adds indirection.

[cueckoo]

Original reply by @zombiezen in cuelang/cue#454 (comment)

I just realized that the downcasting should probably imply unifying with its value, so my desired solution would actually look like:

#Object({
  #MyTemplate
  #foo: "bar"
})

[cueckoo]

Original reply by @mpvl in cuelang/cue#454 (comment)

I just realized that the downcasting should probably imply unifying with its value,

Yes, absolutely, it would imply unification.

The original spec had a definition of downcast that was exactly this (syntax + semantics). But as I couldn't find a good use case, it was dropped (for now). There have been several uses for it since, though, including this one. For instance, the downcast operator also indicates the difference between a protobuf converted to an open definition (proto over the wire) or a compiled closed definition (defining a proto value in Go).

The syntax may not work, though. There is some thought of using the function syntax for macro-like substitution, a syntactic sugar that lets structs behave as functions. This does not seem to be compatible with casting.

An alternative syntax could be

#Object{
  #MyTemplate
  #foo: "bar
}

or

#Object[{
  #MyTemplate
  #foo: "bar
}]

The latter would overload the [] operator, although there is some thought of eliminating that operator altogether (allowing foo.0, foo."bar", and foo.(expr)` as syntactically more regular alternatives).

The cast and "macro" operator are related, though, with a bit of squinting. So maybe it is possible to have a single solution.

The "macro" operator would translate Foo("a", "b") to

{
  Foo
  #0: "a"
  #1: "b"
}

for instance, where Foo could be

Foo: {
  #0: string
  #1: string
  "\(#0)-\(#1)"
}

If a downcast operator that could be used to achieve similar convenience would make a macro operator redundant, though.

[cueckoo]

Original reply by @mpvl in cuelang/cue#454 (comment)

Note an objection against the Foo{} syntax would be that it is too much an easy substitute for Foo&{} disabling the "closedness" checking and thus making it easier to miss typos.

To fix this, there should be a rule that says that fields defined within the struct should either be defined within Foo or used within the struct. This would not hold for fields defined in embeddings, but at least Foo&Bar is still shorter than Foo{Bar}.

[cueckoo]

Original reply by @proppy in cuelang/cue#454 (comment)

I wonder if having a binary operators like Python has for sets:

• & for interestion
• - for difference
• ^ for symetic difference
• < subset test
• > superset test
  could be a different take to this issue?

@jlongtine pointed me to a recent discussion https://cuelang.slack.com/archives/CLT3ULF6C/p1599911085226700?thread_ts=1599910691.226200&cid=CLT3ULF6C which discuss a similar %!(MISSING) operator.

It could user to keep the same mental model they have about unification (and other binary operator):

#Object %!{(MISSING)
  #MyTemplate
  #foo: "bar"
}

While allowing easier chaining:

#Object %!{(MISSING)
  #MyTemplate
  #foo: "bar"
} %!{(MISSING)
  #MyTemplate
  #foo: "bar"
}

[cueckoo]

Original reply by @mpvl in cuelang/cue#454 (comment)

@proppy : I think having full set logic is entering scary territory. Perhaps in a struct package.

@zombiezen : a problem with the T(V) notation is that it doesn't jive well with the builtin syntax and possible functional interpretation of structs that can be a result of generalizing struct. I wasn't able to come up with something better.

One thought I had though: the plan is to expand the selection operator to allow more types to both facility the query mechanism and label mechanism:

[pattern]: T // Apply T to fields matching pattern (already exists, but the pattern can match more)
(value): T   // create field label from dynamic value.

then the idea was to allow the following corresponding RHS selectors

a.[pattern]
a.(value)

These could be used wherever comprehensions are used now (they create streams). Not that this makes the a[x]operator somewhat redundant.

It is possibly problematic, but one idea would be to overload a.(value). That is, for concrete scalars it looks up a field, but for structs and lists, a.(T) would projects the streamed values of a onto T, discarding any field or element that does not exist in T. This can arguably be sold as a natural extension of the selection mechanism.

Coincidentally, it is also very close in syntax to Go's dynamic cast operator.

Anyway, just brainstorming here. I'm not sure if this actually would make sense logically. But it seems more feasible than T(a) at least.

For completeness of the whole query extension direction: the idea of the pattern selection operator would be allowed to be of the form [fieldPattern: T], either LHS or as RHS selector, where fields are matched as usual, and values of this field are further matched by T. There is more but this is probably already enough context.

[seh]

This desire came up again in discussion in the "language" channel of the "CUE" Slack team.

[rogpeppe]

As this is probably not a hugely common operation to perform, I wonder whether a builtin function might be more appropriate than using an operator.

Given that this is essentially cutting out everything from one operand that's not in the other, I wonder if a nice name might be just cut.

For example:

obj: {
	a: 123
	b: {
		arble:    "baz"
		aardvark: "foo"
		beetle:   23
	}
	c: "ppp"
}

#T: {
	a: int
	b: [=~"^a"]: string
}

obj1: cut(#T, obj)

// the above produces the following:
obj1: #expect
#expect: {
	a: 123
	b: {
		arble:    "baz"
		aardvark: "foo"
	}
}

Note: I suspect we don't want entirely regular unification between the two operands, because we probably want to specifically allow fields that aren't mentioned in the closed struct. For example, #T above wouldn't allow the fields c or b.beetle but we'd probably want to allow the downcast in that situation.

One useful invariant to consider: cue(a, b) & a would be exactly the same as cue(a, b).

[myitcv]

One useful invariant to consider: cue(a, b) & a would be exactly the same as cue(a, b).

I assume you mean cut() here?

I'm not clear that the first proposed argument to cut() needs to be closed. I'm not sure whether it's the right word to use here, but I see this operation as being merely a projection.

So your example should (to my mind) work with #T being simply T.

[nyarly]

In part to clarify my own understanding:

It seems like downcasting would be a very special case of "Querying" in #165, right?

Like cut() could be modeled with something like

#Cut: {
  fit: {} // i.e. the struct to "fit" to
  from: {} // the struct to cut from
  fields: [for k,v in fit { k }]
  out: from.[@ in fields]  // not sure about the syntax here? 
}

so cut() would be (#Cut & {fit: #T, from: obj}).out (in prep also for function syntax...)

[rogpeppe]

@nyarly FWIW I'm seeing it as more comprehensive than that - specifically it would act recursively, which would make it different from the model you just suggested. Also, that model doesn't cater for pattern constraints, because the [for k, v in fit ...] expression can't produce all the possible fields that could be matched by the pattern constraint.

[nyarly]

@rogpeppe That's an important distinction! You can see my initial grasp of the problem in #165 - I've been laying a lot of hopes at the doorstep of Query that may not be founded.

If there's to be a cut(), then an analogous patch() would be very valuable to me.

(which leads to the very odd case, I think, of intentionally including _|_ values in a struct as a way to say "remove this field entirely")

[myitcv]

Adding notes from a recent conversation with @rogpeppe that there might well be merit in a first cut of some downcast mechanism only working on concrete values. For two reasons:

• It is much more clearly defined what to do in the case of concrete values, or put another way it is much less clear what to do when values are not concrete.
• Such a decision leaves the door open for doing more later.

In this regard I note my comment in #943 (comment), specifically that a downcast operator could be spelt as follows:

#largeDef: _ // a large definition 
d: #largeDef

#smallDef: _ // a small definition that subsumes #largeDef (ignoring closedness)

v: concrete(d, #smallDef)