Proposal: required fields and related issues

[cueckoo]

Originally opened by @mpvl in cuelang/cue#822

With extensive inputs from @myitcv

We propose a couple of small additions to the language, as well as a sharpening of the language specification, to solve a myriad of shortcomings related to specifying required and optional fields. We believe the result allows for a much more natural representation of required fields in CUE. This proposal also allows us to define tooling in a more consistent manner and lays the foundation for finalizing the query proposal.

The changes are mostly backwards compatible, but:

• may allow configurations that before would fail (where the failure was likely unintended)
• may allow configurations that worked in v0.2 and early v0.3, but fail as of v0.3.0-beta.2, and beyond.
• may require the use of the -c flag for cue eval and cue export to get the same results with the old representation. This makes this flag consistent with the behavior of vet as well.

Automated tooling can be provided to rewrite CUE files.

Background

We now cover some core CUE concepts necessary to understand the proposal introduced in this document. In some cases, it points out general issues with these topics. People familiar with these concepts can skip this section.

Closeness

CUE closedness is intended to solve two problems:

• be able to catch typos in field names
• be able to represent mutual exclusivity of fields, like in Protos

A specific non-goal of defining closed structs is to define APIs that promise to never add an additional field. CUE does not offer a structured way to specify this, as it is believed that APIs should be open by definition.

A good analogy of the role of closedness in CUE is to compare it to Protobufs. By definition, protobufs are extendible. Fields with unknown tags on incoming messages should typically be ignored. However, when compiling a proto definition to, say, Go, the fields defined on a proto
message is closed. An attempt to access an unknown field in a struct will result in a compile error. In CUE, the fact that a definition closes a struct should be interpreted in the sense of a compiled proto message and not as an indication that an API is closed for extension.

In that sense, enforcing mutual exclusivity in protos is a bit of an abuse of this mechanism. Although it has served the representation of protos quite well, it may help to consider alternatives.

Error types

To understand the below proposal, it is important to realize that CUE distinguishes between several error modes:

1. compile errors: any error that can never be resolved and can be spotted before evaluation. This may be otherwise valid CUE, such as string + “foo”
2. permanent errors: an error that occurs during evaluation and cannot be solved by adding more values to the configuration.
3. incomplete errors: an error that could be solved by adding more values, that is making a CUE value more specific.

In the end these are all errors, but CUE evaluation and manifestation will fail at different points when encountering these.

Evaluation modes

CUE distinguishes two evaluation modes:

Unification

Unification combines two CUE values, preserving the entire space of possible values that are the intersection of the two input values. Unification is commutative, idempotent, and associative.

For instance, cue def presents the result of unification without any simplifications like picking defaults or resolving references if this would change the meaning of the CUE program.

There can be multiple ways to represent the same schema. For instance bool is equivalent to true | false. The goal of unification is not to find a normalized representation of the result. Finding an optimal representation, by some definition, is an option, but not necessity.

Default selection

Default selection is used when a full specification of a value is required, but an intermediate concrete value is needed to proceed. Essentially, this happened for references of the form a.b and a["b"], where a needs to be a concrete list or struct to be able to select the appropriate value.

The steps are as follows:

1. eliminate values with incomplete errors from disjunctions
2. select default values

The differentiating property of default selection is that all optional fields, pattern constraints and closedness information needs to be preserved. This puts limits on the amount of disambiguation that can be done ahead of time. It is the user’s responsibility to disambiguate structs if applicable.

Manifestation

Manifestation projects a result from unification to a concrete value. It is used in cue export, but also, for instance, for:

• expressions: x and y in x + y
• elements projected by a query: x.* (corresponding to the Query Proposal Proposal: CUE Querying extension #165)
• arguments for builtins that must be concrete
• when outputting concrete values, like turning into JSON
  We will write manifest(x) to denote the manifested value of x. This is not a CUE builtin.

Right now this encompases:

1. default selection
2. eliminate duplicates values from disjunctions.

Note that after this elimination different structs with different optional values are retained.

The Issues

We present some issues for the various use cases of CUE that motivated this proposal.

Encourage good API design

When evolving and augmenting an API, it is generally strongly recommended to not introduce backwards incompatible changes. This means that any added fields to an API should typically be optional.

In CUE, optional fields are indicated by a ?. This means that when augmenting an API, one should always remember to add the ?. Especially for larger APIs, required fields should be relatively rare. It would be better if only the required fields had to be marked, while all other fields are optional. This would avoid inadvertently making fields required, and makes it more likely that making a field required is a deliberate choice.

Specifying minimum and maximum number of required fields

The protobuf oneOf representation

OneOf fields in protobufs are currently represented as

#Foo: {
    {} | { a?: int } | { b?: int }
}

This relies on all three disjuncts ultimately collapsing into the same value, assuming that optional fields can be ignored and the closedness to enforce mutual exclusivity.

The following could be used as an alternative.

*{} | { a: int } | { b: int }

This approach is a bit of an abuse of the closedness mechanism, whose predominant goals to catch typos. It mostly works, although it does have a few issues. For instance, embedding #Foo would allow users to redefine a oneOf field, allowing it to be used in configurations that were not allowed by #Foo. To some extent this is okay, considering what closedness is for, and the fact that embeddings disables this check. But it may be nice to address if possible.

In the end, there is currently no explicit “official” way of specifying oneOf fields in CUE that can capture the most likely intent. CUE currently deploys a hacky workaround for this, but that is not a permanent solution.

Specifying a certain number of required fields

The above mechanism does not generalize to specifying that, for instance, at least one of a set of fields should be given by a user.

Policy definitions and querying

One of the core ideas of CUE is that, with the right model, constraints can function as templates, resulting in a powerful combination. The way optional and required fields are marked now works fairly well in this context. These same mechanisms don’t work as smoothly in the context of querying and policy definitions, that is, using CUE to specify filters or additional constraints.

There are currently some gaps in the query proposal that can only be addressed with some refinement in the optionality model in CUE.

Requiring fields

Requiring the user to specify a field is currently achieved by specifying a field with a non-concrete value and then requiring non-concrete values to be concrete when exporting. Note that to CUE, non-concrete values are just values. So, in effect, this mechanism is also an abuse of the current semantics. Better would be to mark fields as required explicitly.

Requiring concrete values

A related problem is the ability to require the user to specify a concrete value. This is currently not possible. For instance

#Def: intList: [...int]

says that intList must be a list of integers. But since [...int] defaults to [], CUE will happily fill in this value if a user doesn’t specify it. Similarly, requiring that a user specify a concrete value is not possible: unifying a user-supplied value with

#X: {
    kind: "X"
    ...
}

would just happily fill in the missing kind.

Error messages

A side-effect of having a mechanism to explicitly specify a field as required is that it will also lead to better error messages. Currently, when CUE discovers a non-concrete message it just complains about that: it has no way of knowing the author’s intent.

Having an error message explicitly state that the error is a missing field would be considerably clearer.

Evaluation

Disambiguating disjuncts

CUE currently only disambiguates identical disjuncts. This means that disjuncts with different optional fields will not disambiguate. The following is an amended example from #353

e: #Example

e: a: "hello"

#Example: {
	a: string
	{
		value?: string
	} | {
		externalValue?: string
	}
}

As the two values have two different sets of optional fields, they are not considered equal.
This can result in confusing error messages to the user, like incomplete value {a: 1} | {a: 1}, for instance when running a command that does not show optional fields, like cue evalorcue export`.
CUE does not do this as disambiguating optional fields is NP-complete in the general case. It is nonetheless confusing for the user.
A similar issue occurs with regular (non-optional) fields.
This can be solved by giving users better tools to disambiguate as well as having more permissive disambiguation.

cue commands

Avoid confusion for users

CUE currently allows non-concrete values in the output. Unifying

a: int
b?: int

and

b: 2

succeeds with vet, because for CUE int is a valid value. The -c option, will verify values are concrete. For cue export, however, values must be concrete to succeed.

Overall, there are some discrepancies in how optionality and non-concreteness is handled across commands. A more consistent handling across commands would be useful.

What is cue eval for? How does it differ from cue export? More clarity for CUE commands

cue eval prints a form of CUE that is somewhat in between schema and concrete data. It may not even be correct CUE. The purpose of this was to get good feedback for debugging. But as CUE evolved this output has become increasingly confusing. A clearer model of output modes is in order.

A more consistent output with more explicit debugging options shared between commands seems less confusing.

Proposal

This section gives a brief overview of the proposal. Details are given in the next section.

Required fields

We introduce the concept of a required field, written as foo!: bar, which requires that the field be unified with a namesake regular field (not required and not optional) which has a concrete value. A field that violates this constraint results in an “incomplete” error (the failure could be solved by adding a concrete value of the field).

Consider this simple example:

#person: {
    name!: string
    age: int
}

jack: #person & {
    name: string // incomplete error; string is not concrete
    age:  int    // ok
}

Further examples can be seen below.

A required field can be referenced as if it were a regular field:

a!: string
b: a   // string

c!: "foo"
d: c   // “foo”

A required field may have a concrete value. The same limitation holds for such fields, meaning that it must be unified with a regular field with that exact same concrete value.

numexist builtin

We introduce a numexist builtin with the signature

numexist(<num>, <expr>+)

which takes a numeric constraint and a variable list of expressions and returns
_: if the number of expressions evaluating to a concrete value unifies with num.
_|_ otherwise, where the error is an incomplete error if the number could still be satisfied by having more concrete values.
When evaluating the expressions, incomplete errors are ignored and counted as 0.

For instance, a protobuf oneOf with fields named a and b could be written as:

#P1: {
    numexist(<=1, a, b)
    a: int
    b: int
}

There is nothing special about the arguments a and b here, they are just resolved as usual. In this case, numexist passes as both a and b evaluates to int, and thus the number of concrete values is 0, which matches <=1. It would fail with a fatal error if a and b were both, say, 1.

Requiring that at least one of a set of fields is non-concrete can be written as:

#P2: {
    numexist(>=1, a, b)
    a: int
    b: int
}

In this case, numexist fails with an “incomplete” error, as the condition can still be resolved by setting either a, b, or both to a concrete value.

Ignoring incomplete errors is needed to allow the same construct for fields that have struct values:

#P3: {
    numexist(<=1, a, b)
    a?: #Struct
    b?: #Struct
}
#Struct: { c: int }

without making a and b optional fields, they would both always evaluate to a concrete value. For consistency, it is probably good style to always mark fields referred to by this builtin as optional.

When optional fields are used, as in #P3 above, we could consider another builtin numexists that checks for the existence of a reference, instead of concreteness.

More specific manifestation

We propose adjusting manifestation as follows (new or changed steps emphasized):

1. default selection
2. eliminate optional fields, pattern constraints
3. [optional]: drop definitions, hidden fields and/or fields with non-concrete values.
4. eliminate duplicates values from disjunctions.

Step 3 can be optional and only affects the outcome insofar it may lead to different disambiguation for structs. Which to choose may depend on the output mode requested by the user.

Unlike with unification, there is logically only one correct representation (disregarding field order for now).

Tooling alignment

The introduction of required fields and a more specific manifestation can be used to simplify the commands of the tooling layer.

The biggest change is that we will generally not interpret non-concrete fields as errors, making them behave somewhat similar to optional fields today. The old behavior would still be available through the -c option. Some commands, like cue vet have always taken this interpretation. So this change will bring more consistency between commands, while addressing many of the above errors.

Details of the command alignment is discussed below.

Detailed design and examples

Required fields

Semantics

A required field, denoted foo!: bar, requires that the field be unified with one or more regular fields (not required and not optional) giving a concrete result. A field that violates this constraint results in an “incomplete” error.

For instance, consider this example:

x!: {
    a!: string
    b: int
}

This would require that the user specify field x which must have a field a with a concrete value. Dropping the ! from x would mean that users don’t have to specify a field x but when they do, they also need to specify a concrete value for a.

The required constraint is a property of fields. Here are some examples of how required fields unify with regular fields and other required fields:

{foo!: int} & {foo: int}   	→  {foo!: int}
{foo!: int} & {foo: <=3}   	→  {foo!: <=3}
{foo!: int} & {foo: 3}  	→  {foo: 3}

{foo!: 3} & {foo: int}   	→  {foo!: 3}
{foo!: 3} & {foo: <=3}   	→  {foo!: 3} & {foo: <=3}
{foo!: 3} & {foo: 3}  	→  {foo: 3}

A total ordering of all types of fields can be expressed as

foo?: int > foo: int > foo!: int > foo!: 1 > foo: 1

Note that {foo!: 3} & {foo: <=3}: cannot be simplified further. The definition requires that the values of the regular fields together be concrete. Logically, {foo: <=3} could unify with {foo: >=3} to become {foo: 3} (it represents the same set of numbers), so rewriting {foo!: 3} & {foo: <=3} as either {foo!: 3} would result in loss of information. Retaining this distinction is important to keep associativity and a valid lattice structure.

Limitations

The definition imposes some limitations on the use of the ! flag. For instance, if a definition were to have both a field foo!: 3 and foo: 3, the latter would remove the requirement for the user to specify a field, because foo: 3 satisfies the requirement of foo!: 3 to provide a concrete value. It doesn’t matter that this comes from the same definition. In general, when marking a field as required, it is advisable that all other constraints on the same field in a definition (or at least those with concrete values) should specify the ! as well. These cases could be caught with a cue vet rule, requiring that in a single definition all fields must be either required or not. This would also help readability.

Implementation

We now present a possible implementation. Understanding this section is not key to the proposal.

The implementation of required fields could be done with an “internal builtin” __isconcrete defined as follows:

__isconcre(x) if unified with a value y returns
a fatal error if unification fails
_ if it succeeds and y is concrete
an incomplete error otherwise

This would be implemented along the lines of current validators (like <10) and the proposed must builtin.

Example rewrites:

foo!: int   	→   foo: __isconcrete(int)
foo!: 3   	→   foo: __isconcrete(3)
foo!: {name: int} 	→   foo: __isconcrete({}) & {name: int}
foo!: [...int]	→   foo: __isconcrete([...]) & [...int]

Also __isconcrete(int) & __isconcrete(3) → __isconcrete(3).
After this conversion, unification proceeds along the usual lines.

We opted for a syntactic addition to the language and not to have the user define this builtin directly, as its usage is rather confusing. For nested structures we only care about concreteness at the top level. So using __isconcrete({name: int}) would work, but would do unnecessary work and also give the impression that field name itself should be concrete.

Another reason is that people will usually think of this functionality as specifying a required field. Technically speaking, though, as CUE treats non-concrete values as “present”, it should be called “concrete” and not “required”. The use of ! avoids this issue.

Implications for ?

Note with the addition of !, the use of ? would be eliminated in most cases. As we saw for the proposal for numconcrete, however, there are still some use cases for it. For this reason, as well as backwards compatibility, we expect that ? will stay.

Example: querying

Regardless of whether subsumption or unification is used for querying (see #165), it is currently not possible to specify a field should be concrete but of a certain set of values.

However, with the ! constraint, we can write

a.[_:{name!: =~”^[a-z]”}]

to query all values of a that have a name field starting with a lowercase letter.

Note, the ! would not be necessary when using subsumption and querying using concrete values like name: “foo”. It would also not be necessary if all values in a can be assumed to be concrete or if including non-concrete values is desirable.

Example: policy specification

It is currently awkward to write that a value should either have value of one kind or another. For instance:

{a: >10} | {b: <10}

would match both variants if a user didn’t specify any of the fields (assuming a and b are valid fields).

Using the exclamation mark ({a!: >10} | {b!: <10}) would explicitly require that one of these fields be present.

Example: require user to specify values

This proposal allows for requiring a configuration to specify concrete values:

• a!: [...]
• a!: {...}
• a!: 1

This can be useful, for instance, to require the use to explicitly specify a discriminator field:

Example: discriminator fields

The use of ! can signal the evaluator the presence of discriminator fields. This could form the basis of a substantial performance boost for many configurations. Tooling could later help to annotate configurations with such fields (detecting discriminator fields is akin to anti-unification).

Consider:

#Object: {
    kind!: string
}
#Service: #Object & {
    kind!: “Service” // Require the user to explicitly specify this.
}

In this case, the user is required to explicitly specify the discriminator field. But even if the ! is dropped from #Service.kind, as a convenience to the user, the ! from #Object would still signal that this is likely a discriminator field. The same holds if a #MyService: #Service & kind: “Service” instantiation would make the field concrete.

Example: at least one of

The required field can be used to specify that at least one of a collection of fields should be concrete:

#T: {
    {a!: _} | {b!: _}
    a: int
    b: int
}

If both were concrete, the resulting value will be identical and duplicate elimination will take care of the disambiguation.

numconcrete builtin

The numconcrete builtin allows mutual exclusivity of values where this would be hard or expensive to express using the required field construct introduced above. It has the additional benefit that it more directly conveys the intent of the user.

In many cases it is possible to express the required field annotation in terms of this builtin. In For instance, a!: int could be written as

numrequired(1, a)
a: int

However, it is not possible to express requiring the user to specify a specific concrete value, such as a!: 1. For this reason, as well as for convenience in specifying required fields in policy definitions, it makes sense to have both constructs.

Things are a bit tricker when requiring fields of structs or lists to be concrete. As these are always concrete, all values would pass.

Need for ?

In the current proposal, we rely on ? to allow specifying a non-concrete struct for a field. Alternative, this could be done by a builtin, like must:

#P: {
    numconcrete(<=1, a, b)
    a: must(a&#Struct1)
    b: must(b&#Struct2)
}

or some dedicated builtin. The must builtin (as proposed in #575) would essentially remain incomplete as long as it is not unified with a concrete value.

The use of ?, though, seems more elegant.

numvalid

Considering the need for ?, it may be useful to also have a builtin numexist which counts the number of valid references (values that are not an error). This has the advantage that it will enforce the consistent use of ? for those fields that are part of a oneOf, making them stand out a bit more. In the case of Protobuffers, this works well, and may more accurately reflect intent.

We considered using numvalid instead of numexist, but it does not cover all cases correctly. A separate builtin proposal should lay out all the builtins and their relevant consistency.

Naming

numexist seems to accurately describe what the builtin does. It may not be the most intuitive naming, though. numrequired seems on the surface to be a likely candidate, but it would be a misnomer, as this is specifically not what that means. (In fact, the term required field is also not quite correct, even though it conveys its predominant use case.)

An possible name, albeit less descriptive, could be numof. As may be confusing, however, as it alludes to the JSON schema constructs anyOf and oneOf, which are different in nature. In that sense numof(count, ...expr) would seem more to indicate a disjunction (like the or builtin), where the user can indicate the number of disjuncts that should unify.

Implementation

numconcrete is a so-called non-monotonic construct: a failing condition may be negated by making a configuration more specific. CUE handles these constructs in a separate per-node post-evaluation validation phase. It must be careful to not return a permanent error if a resolution of the constraint is still possible. In this regard, it will be similar to the implementation of struct.MinFields and struct.MaxFields.

Other than that, the implementation is straightforward: evaluate the expressions, count the number of concrete values (non-recursively), counting incomplete errors as 0. The latter is necessary to allow non-required fields of structs and lists to fill this pattern.

vet rules

It may be surprising to the user that

numexist(>=1, a, b)
a: {}
b: {}

always passes ({} is always considered concrete). A vet rule should detect use cases where an argument is always concrete, and suggest the use of ? or possibly numexists.

Example: protobuf

To annotate protobuf oneOfs, under this proposal one could write

#X: {
    *{} | {a!: int} | {b!: int}
}

Note that this doesn’t change all that much for the current model and the current model would still work. The main difference is that it enables a stronger hint for early elimination of alternatives to the CUE compiler.

This notation also doesn’t address the embedding issue: it is still possible to add a field that before was mutually exclusive, even overriding its original type. For instance:

#Y: {
    #X
    a: string

would evaluate to

#Y: *{a: string} | {a: string, b!: int}
}

Arguably, this comes with the territory of using embedding, which gives the power of extension, but disables checking: it is already the responsibility of the user to embed with care. Also, one can imagine having a vet check to guard against such likely mistaken use.

That said, the numexist builtin would allow writing the above proto definition as:

#X: {
    numexist(<=1, a, b)
    a?: int
    b?: int
}

The use of optional fields here is unnecessary, but are needed to cover the general case, for instance if a and b were fields with a struct value. One advantage is that this sets these fields visually apart from fields that are not subject to concreteness counts.

Note how this closely resembles the “structural form” as used for OpenAPI for CRDs.

The resulting type cannot be extended to redefine a. Also, the definition gets rid of disjunctions, and defines the intent of the constraint more clearly. This, in turn, can help the CUE evaluator be more performant.

Evaluation modes

We now briefly discuss the phases of CUE evaluation, to show how they will interplay with optional and non-concrete fields.

Manifestation

We proposed that in the manifestation phase we disambiguate disjuncts based on concrete values only. It will be important to not leak this final step into earlier phases of evaluation, such as default selection. Doing so may, for instance, cause a disjunct with arbitrary optional fields to be used for closedness checks.

Consider this example:

a: { b: {foo?: >=1} } | { b: { foo?: <1 } } 

Purely on the basis of concrete values, these two are identical. However, simply picking the first or second when resolving a.b would give different results for a.b & {foo: 1}.

Doing disambiguation early, however, has quite considerable performance benefits. An implementation can work around this by clearly marking a result as ambiguous. For instance, deduped elements can have counts of the number of values that were collapsed into it.

Example: Disambiguating disjuncts.

With the new semantics of manifestation, the current example from issue #353 resolves as expected: consider

e: #Example

e: a: "hello"

#Example: {
	a: string
	{ value?: string } | { externalValue?: string }
}

In this case all two disjuncts in the resulting disjunction will have the same concrete values. This will even be the case for non-optional fields if we allow non-concrete fields to be ignored.

Example: Schema simplification

A typical Kubernetes definition now looks like

#Object: {
    kind: string
    foo?: int
    bar?: int
} 

littered with question marks.

Under this proposal, the above can be written as

#Object: {
    kind!: string
    foo: int
    bar: int
} 

This eliminates the majority of uses for ? and marks required fields more explicitly.

Printing modes

• cue
• cue export
• cue def

cue printing

The majority of use cases in cue seem to be to use cue export or cue eval along similar lines.

We propose a “default command” cue <selection> which uses manifestation with the following specification:

• Non-concrete values are dropped by default. Disjunctions are disambiguated based on their value after removal.
• Lists and structs that were only defined in definitions and that have no elements or fields with concrete values, recursively, are dropped from the output.

Omitting non-concrete values from the output avoids flooding the printing from non-concrete fields merged in from definitions. For similar reasons CUE omits optional fields (those with a ?) from the output today. So the choice to omit non-concrete values is a logical consequence of allowing users to omit the use of ? in the majority of cases in schema.

Ignoring non-concrete values when exporting is a departure from how cue export works and how generally constraints are enforced in CUE. To provide backwards compatibility users could use the -c flag to

• fail when fields are not concrete for export
• print non-concrete values unconditionally in CUE mode.

Printing modes:

• -D: also print definitions. (TODO: evaluated in schema mode or manifestation mode)
• -H: show hidden fields of the current package
  • TODO also show hidden fields of imported packages? Probably not. We could do so if we supported _foo#bar-style package classification. This may be a debug option to reflect it is not valid CUE as of this moment.
• -A: show attributes
• -O: show optional fields (will be rare because we anticipate that ? might become obsolete)
• -C: print comments
• -i: ignore errors and show in context of an evaluation.
• -a: show non-concrete values (essentially showing what other values can be set).
• --debug==: useful debugging information in the form of comments below each field and value
  • original expressions/ conjuncts
  • line information
  • default values
  • dependencies
  • hidden fields from other packages
  • etc.

The default printing mode is CUE. Output formats can be chosen with the flag --out.

cue export

Much of the current functionality of cue export is reproduced in command cue. We propose to repurpose the export command adding the functionality described below.

Backwards compatibility

cue export would differ in one major way: non-concrete fields would be omitted from export; only those explicitly marked as required would result in an error when not concrete.

The old behavior can be obtained by using the -c flag, just as one would have to use it today for cue eval and cue vet, making behavior between all of these more consistent.

File output

cue export would otherwise be repurposed to be the inverse of cue import. It would be like command-less cue, but would interpret @export attributes to generate specific files.

More specifically, any struct may contain one or more export attributes that evaluates to a CUE “filetypes” specifier to direct export to write a file of a certain type.

Consider the following CUE file.

a: 2 + 3
baz: {
    @export("baz.json")
    b: a
}
bar: {
    @export("jsonschema:/foo/bar/bar.json")
    string
}

This would instruct cue export to generate two files.

By default the files will be exported in thetxtar format as follows:

// File
import baz ":/foo/bar/baz.json”
import bar "jsonschema:/foo/bar/bar.json”

a: 5
"baz": { baz, @export("baz.json") }
"bar": { bar, @export("jsonschema:bar.yaml") }
-- baz.json --
b: 5
-- bar.yaml --
type: string

Note that bar.yaml represents a JSON schema here.

The comment section of the txtar output (above all files) would describe how to reconstruct the original file from the generated files. Note that this example utilizes several currently unsupported features, like JSON imports and JSON Schema output.

Options:

• -z: write the output to a ZIP archive
• -u/--update: actually generate the files

cue def

The define command cue def will remain as the main way to simplify CUE schema without resolving them to concrete values.

It is allowed to simplify disjunctions (retaining semantics), but it may be on a best-effort basis. Default values are retained

Special features to be addressed in a different design doc could include:

• Make self-contained unification versus retaining imports. (-S)
• Do potentially expensive simplifications.

Query extension

One big motivation for this proposal is to narrow down some of the gaps needed for the query proposal (see #165). One such gap was how to define queries and whether such queries should only return concrete matches or everything.

It is not the goal of this proposal to fully define these remaining gaps, but at least to show in detail how the main proposal interacts with this proposal and solves some remaining puzzles.

Selectors

CUE is different from other query languages in that a user can query concrete and non-concrete values. At the same time, we would like to avoid confusion between these two modes of operation. In particular, we need to define what to expect for selecting values in projections.

CUE currently has one form of selection: a.foo. To understand the various modes of selection in projections, we also propose a variant of this: a.foo?. For regular selection they are defined as follows:

1. a.foo: as it is today:
2. the value for foo if it exists in a.
3. an incomplete error if it does not exists, but is allowed and could be added later
4. a fatal error if foo is never allowed in a
5. a.foo?: like a.foo, but instead of an incomplete error it would return the constraints for foo if foo were defined as _.

So a.foo? where a is a struct is equivalent to (a&{foo:_)).foo, and b.1?, where b is a list (allowing integer indexes for lists in selection) is equivalent to (b & [_, _] )[1]. The foo? variant works around a common issue reported by users.

Now pulling in the query proposal (#165), let’s consider the equivalent for projections, that is, how these selectors behave for a “stream” of values that is the result of a query. Firstly, we expect that typical query usage will either center on querying concrete data, or on API definitions that have not been manifested.

To facilitate this view and translating the semantics of selectors to that of projections, we assume that for the purpose of projections a non-concrete value (like <10) is treated as an “incomplete” error.

Given this definition, we can then define:

1. a.*.foo: value is dropped for incomplete errors and if the value is non-concrete (does not apply recursively)
2. a.*.foo?: also allow non-concrete or optional (match pattern constraints)

Forms 1 and 2 will fail if an illegal value is selected (fatal error).

The semantics of treating non-concrete values as an incomplete error when querying was partly chosen to be consistent with the proposed default mode for cue commands to silently ignore non-concrete values (unless the -c option is used), making behavior consistent and predictable across the spectrum.

Note that there is precedence within CUE to expect values to be concrete. For instance, operands most binary expressions (except & and |) will result in an incomplete error when not concrete. The proposed semantics for queries is identical.

Querying with subsumption (instance-of relation)

The ! notation solves an issue with allowing a value to be used as subsumption in query filters. Consider the following:

a: {
    foo: { name: string }
    bar: { name: “bar” }
}

Using a subsumption filter {name: string} would also match foo, as it is, strictly speaking subsumed. Using !, we can work around this:

query: a.[:{name!: string}]

will select only bar.

We could require that if a field is specified in a query we required it to be concrete. That is a bit of an odd rule, though. The ! notation seems a natural solution.

Subsumption variants

Subsumption in the general case is an expensive operation. Using the definition of the different evaluation modes, however, we can distinguish two different kind of subsumption:

1. subsumption patterns without pattern constraints (such as in the above example)
2. subsumption patterns with pattern constraints.

Note that closed structs are defined in terms of pattern constraints, so any closed struct classifies as 2.

Patterns of type 1 would be executed as actual subsumption.

For patterns of type 2, however, we would require that the queries values must have been explicitly unified with this value. For instance, the query

query: a.[:v1.#Service]

would search for any value in a that were unified with v1.#Service. So a value that has all the same fields in a as a #Service would still not match unless it was unified with this explicit definition. In effect, this introduces a notion of explicit typing, rather than just relying on isomorphic equivalence.

Such selection is easy to implement efficiently, and may be a good compromise.

Transition

Although this is a big change to the language, we foresee a relatively smooth transition. The meaning of ? would largely remain unaltered.

Phase 0

Introduce the new disambiguation semantics. This should be done before v0.3. Although somewhat different, v0.2 has similar semantics, and introducing this before a final release will allow for a smoother transition.

Phase 1

In phase one we would introduce the ! annotation and the numconcrete builtin to work as proposed.

Phase 2

Add an experiment environment variable, like CUE030EXPREQUIRED to enable the new semantics eliding non-concrete fields. In this mode, the -c flag would mimic old behavior.

A cue fix flag allows users to rewrite their CUE files to the new semantics.

If the flat does not allow for a fine-grained enough transition, we could consider defining a transitionary field attribute to define the interpretation of such field on a per-field level.

Phase 3

Decide on whether to proceed with step 4.

Phase 4

The biggest change will be moving to relaxing the rules for non-concrete fields and moving away from excessive usage of ?. This would be done as a minor pre-1.0.0 change (e.g. v0.4.0 or v0.5.0).

The biggest issue is for installations that rely on not using ? meaning required. It will be good to ask users whether the use of cue fix and/or -c is sufficient or whether an API feature is supported as well.

Adding a feature to cue fix to “promote” fields on a one-off basis would be straightforward. Generated configurations could just be regenerated.

Note that default values and other concrete values specified in definitions would still be printed. It is only the non-concrete values that are omitted.

The removal of ? may also have performance implications, as CUE processes them differently internally. The implementation can be adjusted however to overcome performance issues. Experience with v0.2, however, which processed optional fields similarly to regular fields, showed that the performance impact of this is relatively small. Structure sharing can further mitigate this issue, and we should probably ensure this is implemented before the transition.

Alternatives considered

Alternative disjunction simplifications

We also considered eliminating non-concrete values from disjunctions. For instance, at manifestation time (only!):

a: int | 1

could in such a case be simplified to a: 1. This would obviate the need for the default marker in this case.

The overall intuition here is that this would be weird, though.

By extension, we also chose not to simplify

{ a: 1, foo: int } | { a: 1, bar: int }

To achieve this effect without using defaults, users would have to write

{ a: 1, foo?: int } | { a: 1, bar?: int }

A variant that would allow such simplification is open for consideration, though, especially if it can help fully deprecating the use of ?.

Other definitions of foo!: int

We have considered the following meanings of foo!: int

foo!: int as an optional field

foo!: bar is an optional field that must unify with any concrete field to be valid.

It sort of would still work if people would diligently use ? for fields in definitions, but this would defeat one of the main purposes of introducing ! in the first place.

But logically the foo: int > foo!: int relation makes sense, as the ! constrains the regular field. This alone gave too much of a contradiction to work well.

require to be unified with a non-definition field with concrete value

The main advantage of this approach is that it would allow adding the required constraint to a schema that already defines that field as a concrete value.

The main drawback of this approach is that it is not possible to create a derivative definition of a schema that defines a required field that fills in the required field for the user.

For instance, suppose we have a definition

#Service: { kind!: string }

And we create the derivative:

#MyService: #Service & {
kind: “Service” // still not set
}

then kind would still not be set.

It seems that this should be possible, though. Specifying a concrete field in this case is akin to setting a default value. In general it is considered good CUE style to define defaults separately from an API, so this would be consistent with that definition.

Also, although the CUE evaluator can track the origin of fields quite easily, there is no representation for a “field that has already been unified with a concrete field.

require to be unified with a non-definition field and a concrete value

The distinction from the former is that the concrete value may originate from a definition. This definition however, is not associative.

Required indication on the right-hand side

We considered writing foo: int! or foo: required(int) instead of foo!: int making the requiredness a property of the value instead of the field.

This didn't sit well with the requirement of needing to unify with a concrete value from an optional field: {foo?: 1} & {foo: int!} would be hard to represent correctly in CUE. A goal is to allow representing evaluation results in CUE itself, but we did not see a way to accomplish that here.

List type

One motivation for this proposal was for the ability to define a non-concrete list. For this we considered introducing Go style

a: []T
b: [<10]T

types.
However, indicating the size can already done by syntax introduced in the query notation:

a: [<10]: int

which would just be a generalization of CUE.

Also, this would still not solve the same problem for non-concrete structs or scalar values.

So overall, this would be a heavy-weight solution for little benefit.

Querying using unification

The ! operator would also be useful for querying values using unification. Normally, a query like a.[: {name: 2}] would produce quite unexpected results: it would unify with any element that either has name set to 2 or for which name is undefined, possibly setting that value to 2.

To avoid this, users could write a.[: {name!: 2}].

We considered this to be too cumbersome and surprising to be a viable solution, though.

Alternate definitions for querying by subsumption

There are really many variants possible. We mention a few. All of these could be considered.

Definitions as types

1. If the subsumption is a definition, the subsumed instances must have unified with this value.
2. For non-definitions, we do a full subsumption, but put restrictions on what values are allowed.

This would allow queries like

a.[:{a: [string]: name!: <”Q”}]

but perhaps not more funky queries using pattern constraints.

Definitions as types, subsume concrete values only

If the subsumption is a definition, the subsumed instances must have unified with this value.
For non-definitions, we only match the manifestation (concrete values) of the value.
This would allow full pattern matching.

Always subsume manifested values only

This would allow unrestricted unification. This seems limited, though, as people may want to query APIs with certain properties.

The syntax a.*.[:{}]? could be used to query the non-manifested value. Similar restrictions may still have to be applied to subsumption in this mode though, though they would typically be irrelevant to the casual user.

Alternative semantics for projection selectors

We’ve considered a more direct correspondence between selectors for projection and regular selection. This results in the following definitions

1. a.*.foo: fatal error if not allowed, dropped for incomplete errors, otherwise value returned (including non-concrete).
2. a.*.foo?: also allow non-concrete or optional (match pattern constraints)
3. a..foo!: like a..foo, but additionally filters for concreteness..

For the common use case of requiring concrete data, this would mean that users would have to almost always use the third form. This seems undesirable and will likely result in too many gotchas. In the end, we were able to get the desired behavior for selectors in projections by only considering a non-concrete value to be an “incomplete” error. This seems to be a reasonable solution. Consider also that interpreting a non-concrete value as incomplete already happens at various points in CUE evaluation.

[cueckoo]

Original reply by @eonpatapon in cuelang/cue#822 (comment)

One motivation for this breaking change is "good API design", ok but what about configuration ?

Most of my configuration inputs are required values. Am I the only one ?

[cueckoo]

Original reply by @mpvl in cuelang/cue#822 (comment)

One motivation for this breaking change is "good API design", ok but what about configuration ?

Most of my configuration inputs are required values. Am I the only one ?

Can you give a concrete example (ideally real-life) where you would expect things to be concrete?

Also note that the plan is to not initially change it and give people plenty of opportunity to play around with it. But would be great to see good arguments in favor of the current semantics.

[cueckoo]

Original reply by @mpvl in cuelang/cue#822 (comment)

@eonpatapon: part of the reasoning: either configurations are small, in which case it won't hurt to specify a few extra !'s, or they are not, in which case the likelihood that all fields are required is tiny, while specifying ? everywhere is both tedious and error prone.

[cueckoo]

Original reply by @verdverm in cuelang/cue#822 (comment)

Anecdotal experience from my DevOps work...

When I look at Terraform, Ansible, Kubenetes, Helm, cloud provider APIs, and a host of other programs, I find very few required fields and many more optional fields that typically have defaults.

When creating configuration for tools I develop in this role, I try to follow this pattern as well.

[cueckoo]

Original reply by @niemeyer in cuelang/cue#822 (comment)

Thanks for the extensive context given in the proposal.

I'm a bit concerned with the direction of CUE after reading it, though, as much of the justification for the suggested changes seems to be either about performance or about cases in which the most obvious approach doesn't work as it implies.

I'll exemplify the concern with a few quotes:

A required field, denoted foo!: bar, requires that the field be unified with one or more regular fields (not required and not optional) giving a concrete result.

The intuitive notion we all have is that things are either required or optional, without in betweens. I realize that this is really concrete vs. non-concrete and present vs. non-present, but if we have to explain optional fields -- such a simple idea -- to the reader and point to a spec, we've lost a shot in terms of design. This is such a visible issue that even the proposal itself confuses the point by calling it required sometimes, and the chosen syntax reinforces the problem by them being opposites (! vs. ?).

Intuitively, it seems fair to say that when one expresses something as simple as this:

{ foo: string, bar?: int }

What is meant here is that the outcome should be foo being present and a string, while bar is optional but if present must be an int. If the goal is generating CUE source (via queries, or any other mechanism), then it's okay for foo to remain defined as string. If the goal is generating json, yaml, or validating one of these, then they should resolve to a concrete value because the statement wasn't declared as optional. Again, it's clearly subjective, but it seems fair to say that this is the intuition.

Now, speaking specifically about whether optional is more frequent than required or not, we could easily preserve the above semantics and have ! but no ?, but for similar reasons to the stated above, this sounds like a mistake because the inherent notion that people have about fields is that when they enter a name, that name is there and is required.

Demonstrating these points with another quote from the proposal:

It is currently awkward to write that a value should either have value of one kind or another. For instance:

{a: >10} | {b: <10}

would match both variants if a user didn’t specify any of the fields (assuming a and b are valid fields).

This is a great example because it seems hard to argue about what the expression should mean. Instead of adding syntax to enable what people want to do to be possible, it would be better to make sure that this does what people think it would do.

Another one:

Using a subsumption filter {name: string} would also match foo, as it is, strictly speaking subsumed. Using !, we can work around this:

query: a.[:{name!: string}]

Again, similar idea: there's little doubt when reading that query that the user did in fact expect name to be present in the document as a concrete value if the query was done against a body of source to extract a known value. It is only the lack of context that generates the possibility of string matching string being an acceptable answer. Having to type ! on every field every time one wants to write such a query would be unfortunate.

One more:

CUE currently allows non-concrete values in the output. Unifying

a: int
b?: int

and

b: 2

succeeds with vet, because for CUE int is a valid value.

Same idea: seeing both a: and b?: makes it look like indeed this should fail on vet. This is inherent to the way we've experienced such ideas before, so it would be better to make vet indeed match people's expectations, and address other use cases differently.

It's worth saying I'm fresh on CUE, which means I may still be misunderstanding concepts and be missing important design decisions, but it also means that for now I still speak as most people that spend less time on it would see it. Those groups are an important audience for projects I oversee more closely, at least.

As a simplistic suggestion, given the issues I've learned about in this proposal, I would look into transforming the behavior of the bare foo to be the one from the suggested foo! when the context suggests that the user really wants a concrete value and not a CUE expression out. At least for such simple use cases it seems worth making the language match people's expectations, instead of teaching them to workaround via additional knowledge.

Again, thanks for the clearly stated proposal, and I apologize for speaking without much background at this stage. Hopefully I didn't miss the key ideas too much.

[cueckoo]

Original reply by @seh in cuelang/cue#822 (comment)

Like @niemeyer, my first reading of this proposal had me feeling unworthy. So much of it seemed backwards and confusing to me, but I left it assuming that I don't understand CUE well enough yet to see why it should be this way.

[cueckoo]

Original reply by @seh in cuelang/cue#822 (comment)

Another thought, again echoing @niemeyer's statements: I read "required" and "optional" as antonyms, with no third choice. Including both a required designator (!) and an optional designator (?) for anything but a brief transition period is rather confusing, unless every field requires one or the other.

I find it most intuitive to treat an unadorned field as required, with a special adornment to indicate that a field is optional.

[cueckoo]

Original reply by @mpvl in cuelang/cue#822 (comment)

@seh and @niemeyer, thanks for the feedback. I'm not entirely happy, of course, about both flags being present, but so far I have not been able to find a satisfactory solution with only one of those. The current semantics does not function well for expressing policy as well as a slew of constraints. The introduction of ! solves this, but it then leaves a few cases where something like ? is needed (and, in fact, there are only few cases where a ! would be needed as well).

Part of the issue is exactly having data and types in a single view. It simplifies a lot, but it does sometime require some extra expressiveness in clarifying intent.

Perhaps the best way forward is to collect a few key examples that expose the entire set of problems and requirements to see if there is a solution where all cases can be satisfied by using either ! or ?.

Of course I would be very open to a simpler solution that satisfies all these cases.

[cueckoo]

Original reply by @seh in cuelang/cue#822 (comment)

All the big thoughts come to me after I write something here, so take whatever I'm writing this time with a grain of salt.

As you noted, I can see how CUE's various roles for fields—or for the values in fields—makes this more complicated than something like Protocol Buffer messages. CUE fields can include default values, which makes them "optional" in the sense that no one has to provide a value, but the field might still be required to be present in an exported view. That's different from an optional field that has no default value, but could be omitted from an exported view if it lacks a concrete value.

A few questions:

• What would it mean for an optional CUE field to have a default value?
  If no one provides a different concrete value for it, would the default value make it equivalent to a regular field?
• What would it mean to define an optional CUE field with a concrete value?
  Doesn't that just make it a regular field?

[cueckoo]

Original reply by @mpvl in cuelang/cue#822 (comment)

@seh, note that this proposal does not change the semantics of optional fields (using ?):

What would it mean for an optional CUE field to have a default value?
If no one provides a different concrete value for it, would the default value make it equivalent to a regular field?

Nope, it would remain optional. A default value of an optional field is only relevant if unified with a non-optional field with a more general value, that leaves the default and non-default value.

The same would hold for required fields with optional values.

I haven't really seen a use case for this behavior in practice. It's just a consequence of keeping these constructs orthogonal.

[cueckoo]

Original reply by @mpvl in cuelang/cue#822 (comment)

What would it mean to define an optional CUE field with a concrete value?
Doesn't that just make it a regular field?

Same applies here. For both ! and ? specifying a concrete value does not make it a regular field. So foo?: 1 means the field is optional, but if it is specified, it must be 1. There have been some use cases for this but it is rare.

Similarly, foo!: 1 means: foo must be specified and it must be 1. This is actually quite commonly needed (e.g. #Service: kind!: "Service") and it is not possible to specify this is CUE today. This is one of the cases where ! wins over ?.

[cueckoo]

Original reply by @seh in cuelang/cue#822 (comment)

Similarly, foo!: 1 means: foo must be specified and it must be 1. This is actually quite commonly needed (e.g. #Service: kind!: "Service") and it is not possible to specify this is CUE today.

When you say "must be specified," is it necessary for someone to restate that, say, the "kind" field's value is "Service," to confirm the required value, or is that kind!: "Service sufficient to declare the required value that will be exported unconditionally?

If you instead wrote kind: "Service" (without the exclamation point), doesn't that lock the value in just the same? Any other value wouldn't unify with "Service," right?

[cueckoo]

Original reply by @niemeyer in cuelang/cue#822 (comment)

Perhaps the best way forward is to collect a few key examples that expose the entire set of problems and requirements to see if there is a solution where all cases can be satisfied by using either ! or ?.

Indeed that would make it easier to explore options. It would also be useful to collect some context about the use case, as given the proposal above it seems that part of the issue is contextual and perhaps we can do without some of the syntax by taking that into consideration.

Same applies here. For both ! and ? specifying a concrete value does not make it a regular field. So foo?: 1 means the field is optional, but if it is specified, it must be 1. There have been some use cases for this but it is rare.

Similarly, foo!: 1 means: foo must be specified and it must be 1. This is actually quite commonly needed (e.g. #Service: kind!: "Service") and it is not possible to specify this is CUE today. This is one of the cases where ! wins over ?.

It's easy to see the motivation for both of these use cases. What is not clear is what else sits in between them. That is, I'd guess most people would think that specifying kind: "Service" should be enough to express the intent of requirement, and the alternative is the field being optional, and that's it.

Going back to the proposal to illustrate both this and the note above on contextual behavior, we see that example:

#person: {
    name!: string
    age: int
}

jack: #person & {
    name: string // incomplete error; string is not concrete
    age:  int    // ok
}

Isn't the following a more natural way to express the intent above:

#person: {
    name: string
    age?: int
}

jack: #person & {
    name: string // incomplete error; string is not concrete
    // age:  int // ok to be missing
}

Per earlier comment, the error is contextual since the latter expression remains valid while manipulating the state. It is the action on that value -- exporting, vetting, etc -- that turns the lack of concrete values into an error. Can't we take that context into account, instead of adding syntax to express the same thing in a slightly different way?

Depending on use cases, it could be useful to expose some of that contextual difference to the language by having a built-in that does the same operation to a specific value. Using the example above, it might be possible to offer something along the lines of complete(#person) to constrain jack into the exported shape early on, or valid(#person), to enforce completeness without modifying the lattice.

Again I need to add that disclaimer here pointing out that I don't have knowledge of the engine implementation at this time, and I'm trying to understand both the problem and the space of feasible solutions by extrapolating the proposal above.

[cueckoo]

Original reply by @niemeyer in cuelang/cue#822 (comment)

A correction to the above example:

#person: {
    name: string
    age?: int
}

jack: #person & {
    name: string // incomplete error; string is not concrete
}

The presence of age: int in the second struct would turn the optional field of #person into a requirement for jack, and would also generate an incomplete error. For it to not be an error on a completeness test, much like the name it either needs to not be present, or present as optional, or present and concrete.

[cueckoo]

Original reply by @mpvl in cuelang/cue#822 (comment)

@seh: with "must be specified," I mean indeed that it must be specified and cannot be omitted. So it will not be the same as just saying kind: "Service". This is a common case in validation (see e.g. #740, but this has come up more often), and we see this ability to be a critical feature when CUE is used as a policy or validation language in general.

[cueckoo]

Original reply by @niemeyer in cuelang/cue#822 (comment)

@mpvl We both understand that. What we're pointing out is that the intuition is a bare string being in fact required, and reading that issue (#740) I can see @narqo was saying precisely the same there:

    kafka_topic: "topic1" // field must exists and the value must be exactly "topic1"

(...)
Note, in the example above, I could set the constraint for kafka_topic field via a RegExp =~"^topic1$". The issue is that the current behaviour of cue vet feels unintuitive.

Emphasis on feels unintuitive. He's not asking for a different feature. He's saying the current language surprises him, which is what I've been exploring above.

We already have three different constructs that surround this behavior:

• Regular fields
• Optional fields
• Default values

We shouldn't need a fourth "actually required" syntax, but rather just fine tune the existing one to match intuition a bit more closely.

[cueckoo]

Original reply by @rogpeppe in cuelang/cue#822 (comment)

In case it isn't clear, the problem is that non-optional fields are often used to hold calculated values that aren't necessary to provide explicitly.

#Foo: {
    x: int
    y: x * 2
}
f: #Foo
f: x: 1

This is currently fine: #Foo.y isn't an optional field but we don't need to specify it (if we did specify it, we'd need to specify it as exactly 2).

If #Foo.y was specified as a required field, the above would be invalid AIUI.

If we removed the existing behaviour, it would be a radical change to CUE. For example it would mean that cue trim couldn't work AFAICS, because that relies on the existing semantics.

[cueckoo]

Original reply by @seh in cuelang/cue#822 (comment)

I'm still missing something about Roger's point. In the example above, #Foo.y is not an optional field. I take that to mean that it must be present in exported data, and it must therefore be concrete. In this case, left to its stated value of x * 2, that means that field x must be concrete at export time.

[cueckoo]

Original reply by @seh in cuelang/cue#822 (comment)

Also, at risk of putting words into @narqo's mouth from #740, I do now see that he intended that each #Env value had to restate the "kafka_topic" field again, due to his comment about an expected validation failure for env.dev1:

env.dev1 — missing required kafka_topic field,

My current interpretation of the meaning of his #Env definition is that the Kafka topic is nonnegotiable, established as "topic1." Given that, I wouldn't expect any #Env values to restate the field. If they did, of course it would have to match "topic1," but leaving it out should be fine.

I don't understand the motivation to require each #Env value to restate that field with the fixed value.

[cueckoo]

Original reply by @mpvl in cuelang/cue#822 (comment)

@niemeyer:

We shouldn't need a fourth "actually required" syntax, but rather just fine tune the existing one to match intuition a bit more closely.

Can you give an example of how you would express the issue in #740 with the constructs that are currently there?

What kind of changes do you see that making possible?

[cueckoo]

Original reply by @rogpeppe in cuelang/cue#822 (comment)

I'm still missing something about Roger's point. In the example above, #Foo.y is not an optional field. I take that to mean that it must be present in exported data, and it must therefore be concrete. In this case, left to its stated value of x * 2, that means that field x must be concrete at export time.

Here's a concrete example (you can run it with the testscript command):

exec cue export foo.json schema.cue
cmp stdout exported.json
-- foo.json --
{
	"f": {
		"x": 1
	}
}
-- schema.cue --
#Foo: {
    x: int
    y: x * 2
}
f: #Foo
-- exported.json --
{
    "f": {
        "x": 1,
        "y": 2
    }
}

In foo.json, there is no y field present despite it not being an optional field, and that's fine because it's implied by the schema.

In other words, it's concrete in the exported data but not required in the input data. AIUI, the ! qualifier would require it to be explicitly present in the input data too.

[cueckoo]

Original reply by @niemeyer in cuelang/cue#822 (comment)

Can you give an example of how you would express the issue in #740 with the constructs that are currently there?

@mpvl I'm still trying to understand use cases and details of CUE itself, which means my proposal will likely miss important points, but with the conversation so far and the examples from the proposal, what if:

1. Regular fields become required

In other words, the behavior for the proposed ! becomes standard for definitions, so in the case of #740, this would indeed yield a vet error:

#person: {
    name: "Jack"
    age: int
}

jack: #person & {
    // Vet error, no concrete name
    age: 42
}

So this sorts out #740, but opens up problems in current usage. So let's fix those.

2. Default values are extended

We already have the idea of default values today, which is something people understand well from past experiences. So, this already works fine today and we'd keep it working as-is:

#person: {
    name: *"Jack" | "Joe"
    age: int
}

jack: #person & {
    // No error, name is exported as "Jack"
    age: 42
}

In addition, we'd extend default values so that providing a single default is supported:

#person: {
    name: *"Jack"
    age: int
}

jack: #person & {
    // No error, name is vetted/exported as "Jack"
    age: 42
}

This matches what we have today as regular fields in definitions. The field is required per point 1, but has a default.

3. Optional fields are untouched

Doesn't look like we have any issues with those, so they'd stay as-is.

...

As I said, I'm probably missing important details. What issues would that create?

[cueckoo]

Original reply by @mpvl in cuelang/cue#822 (comment)

@seh

My current interpretation of the meaning of his #Env definition is that the Kafka topic is nonnegotiable, established as "topic1." Given that, I wouldn't expect any #Env values to restate the field. If they did, of course it would have to match "topic1," but leaving it out should be fine.

This interpretation works well for configuration generation, but it is unsatisfactory for validation or policy in general. A very simple but important use case is to be able to ask the question: "Is this yaml file valid for consumption by some service as is?". This cannot be specified currently.

What is currently possible, and what concurs with your interpretation, is to answer "is this a valid YAML file for consumption after unifying it with a CUE schema converting it to YAML again?". But this is not the use case that is being addressed here. This extra unification step if often not an option. As far as I can tell, this is unsolvable with the current CUE.

The use case in #740 is a pure validation use case. In the analysis of using CUE as a policy language as well as within the realm of querying, this inability gets more pervasive and also becomes more an intertwined problem where interpretation of meaning within the context of a certain CUE command will not help. I would say that this is the most important issue addressed by the required proposal.

It would be possible to solve this without a language change, for instance kind: required("Service"), or something. But this is not necessarily better and has its own issues, as mentioned in the proposal.

Overall, the suggested semantics of ! more tightly and clearly reflects intent by itself, without relying on context like as part of which command it is run etc. This has several implementation benefits, including better errors. I argument may be too implementation-centric, but in my experience, what makes an implementation preciser and clearer often also helps the user.

@niemeyer @seh @rogpeppe

In a nutshell, so far I can see a transition path to introduce ! and deprecate ?, and keep only one, but I don't really see a path where we can only rely on ? and not have some additional notion of "required concreteness". Which is not to say it doesn't exist, necessarily, of course, I could be wrong.

I know I should compile more examples to expose the issues. In the meantime, though, seeing how #740 could be solved as a first step without the use of ! or a RHS solution (e.g. foo: "x"! or foo: required(x)) would be useful and may provide answers in another direction.

Not saying it isn't possible. In fact, some recent thoughts on CUE patterns and discriminator fields, as well as more reliance on numconcrete (which has some quite neat properties), may negate some of the benefits of using !. But none of that solves #740 and derivatives, unfortunately.

[cueckoo]

Original reply by @seh in cuelang/cue#822 (comment)

Thank you for the "concrete" example. I see that specifying the #Foo.y field is not required here. What I—and, I think. @niemeyer—don't understand is why you want to express that need.

We are missing the ability to do something. I don't know why someone would want to do that thing, so it's hard to think through a good design for a feature that I don't think should exist. If this feature did exist, I expect that I'd find it annoying. When it's time to write file foo.json, why do I need to restate a value that's fixed by other declarations?

[cueckoo]

Original reply by @seh in cuelang/cue#822 (comment)

I wrote the comment above before reading @mpvl's cuelang/cue#822 (comment), but got interrupted before I could post it.

You are correct that I've been ignoring use of CUE to validate input data. My only use of CUE so far has been to help generate output data more concisely and correctly. That one language must carry the burden of checking input data while not augmenting it into compliance and also augmenting data for output makes for a difficult design, as you're struggling with here. Well, it's not so much that you're struggling to design it; the rest of us are struggling to figure out when each concern and feature matters.

[cueckoo]

Original reply by @niemeyer in cuelang/cue#822 (comment)

@seh To be clear, I actually see the issue, and there are apparently at least two of them:

1. No way to validate a missing field against a schema that includes a concrete value, because definitions today inject the value that was supposed to be a validator into the value being analyzed, turning into into a valid object.

2. Abstract values (string, int) are currently considered valid for vetting purposes.

Hopefully we can address both and still eat cake at the end of the day.

@mpvl

In a nutshell, so far I can see a transition path to introduce ! and deprecate ?

I'm curious about that option. If we can have only two distinct behaviors, wouldn't having "foo" and "foo!" be equivalent to having "foo?" and "foo", respectively? Wouldn't it be just syntax?

Overall, the suggested semantics of ! more tightly and clearly reflects intent by itself,

Sometimes that feels true, but in the above examples that's often not the case. For example, I hope we can type such expressions in the CLI eventually, and having to tell people to say foo!: X in a query feels like a design wart leaking out into the user. The case of {a: 1} | {b: 1} described above is also similar. It doesn't feel reasonable to have to type ! to do what seems like the obvious intention.

Or do I misunderstand the examples?

(Please note I've replied above simultaneously. Just want to make sure the reply doesn't get lost due to UI.)

[cueckoo]

Original reply by @mpvl in cuelang/cue#822 (comment)

@niemeyer

As I said, I'm probably missing important details. What issues would that create?

There are some interesting suggestions there I hadn't considered. As usual with CUE, one has to consider what the implications are for the value lattice.

But some quick observations:

• This conflates the idea of requiring concreteness and defaults. This doesn't always have to be bad, it is just an observation.
• This essentially introduces a "required concreteness" mechanism, just with different syntax. In the analysis of required fields, having a RHS mechanism to specify required concreteness (which is done here) resulted in some issues. I don't recall the details. I don't believe they were critical, though.
• I am less clear as to how a transition path would look like here, as there seems to be more syntactic overlap.
• This would break the interpretation for standard JSON. Basically, we do not want to break the semantics of taking two JSON files and unifying them. (Note that CUE is a superset of JSON).
• This could be fixed by swapping the semantics of the "naked star", so making name: *"Jack" the required field, which seems weird.

[cueckoo]

Original reply by @seh in cuelang/cue#822 (comment)

We've been writing back and forth concurrently, so I'll start by saying that I'm writing in response to cuelang/cue#822 (comment).

When validating input data that we don't intend to augment, the CUE field values express constraints the input data must meet. If the constraint is a concrete value, the input data must have that same concrete value.

When generating output data that we can augment, the CUE field values express recipes to generate output data, safely combined with any starting values.

Is it correct that we use unification during both validation and export? I take it that the problem is that with a regular field today that has a concrete value, unifying it with input data that lacks that field just causes the field to come into being, as opposed to catching that the field was absent in the input data.

[cueckoo]

Original reply by @niemeyer in cuelang/cue#822 (comment)

@mpvl

This would break the interpretation for standard JSON.

The merging of two values is a power feature which is great indeed, and not worth breaking. But definitions are already more schema-like than value-like, so flipping their regular fields to become a strict concrete requirement there unless adorned by the default marker would not break that aspect, or would it?

This could be fixed by swapping the semantics of the "naked star", so making name: *"Jack" the required field, which seems weird.

Agreed. If we do need to adorn the key or value, the original ! feels like a better option. I'm just concerned about having to use it in cases which feel like it should be the default behavior for a regular field, and to avoid the confusion of having optional, semi-required, and actually-required fields, but given how nice CUE already is I'm sure you'll figure something good out.

[steeling]

One (interesting) use case for required fields just presented itself in the govim project. As part of our tests, we assert JSON marshalled output from various Vim function calls, in order to make assertions about Vim's state. For example, we assert that the quickfix list has specific contents in order to assert that the user can see the corresponding error list in Vim. Our assertion therefore looks something like this:

[
  {
    "bufname": "main.go",
    "col": 36,
    "lnum": 6,
    "module": "",
    "nr": 0,
    "pattern": "",
    "text": "undeclared name: i",
    "type": "",
    "valid": 1,
    "vcol": 0
  },
  {
    "bufname": "other.go",
    "col": 3,
    "lnum": 66,
    "module": "",
    "nr": 0,
    "pattern": "",
    "text": "another error",
    "type": "",
    "valid": 1,
    "vcol": 0
  }
]

What's problematic though is that if Vim adds another field to the returned value (a backwards compatible change), then the JSON marshalled result is different on a byte-by-byte comparison basis.

Whilst this doesn't happen that often, I wonder whether the problem presents a nice solution pattern in the context of required fields.

Using required fields we could instead assert using a much more succinct form, only those fields we really care about. Here we assert that there is at least one error, and that in the 0th position the error has the four fields with values as follows:

[
  {
    bufname!: "main.go",
    col!: 36,
    lnum!: 6,
    text!: "undeclared name: i",
  },
  ...
]

This would be validated by unifying this assertion with the actual JSON, and confirming there are no incomplete errors (as proposed, these are the errors that would result from a required field not being satisfied).

One thing that jumps off the page in the example above, is that ! could well appear everywhere under such a usage pattern. Perhaps a builtin that recursively applies the "required" requirement could alleviate that pain? It would certainly be consistent with the usage.

One question about the example above is whether top-level lists work under all circumstances. I think the answer to that question is yes, because lists have length. Hence asserting a value of [] would only match an actual empty list, [_] would only match an actual single element list etc. ... can be used to assert "the list might be longer than this".

An alternative to this, without required fields is to use some form of an explicit translation instead of inheritance:

#Input: {
 ....
  bufname?: string
  col?: int
  lnum?: int
  text?: string
}

#ImportantFields: {
  bufname: string
  col: int
  lnum: int
  text: string
}

convert: {
   input: #Input
   output: #ImportantFields & {
     bufname: input.bufname
     col: input.col
     lnum: input.lnum
     text: input.text
   }
}   

I'd much rather see a solution here that

1. evaluates optional, non-concrete types to null, when exporting to json
2. exports optional, concrete, non-null fields
3. omits optional, null fields when exporting to json
4. Allows field access of optional types -- since they evaluate to null when exporting, it would return null (ie: x.y.foo == null).

[joulaud]

intro

I did not read everything as the proposal seems really complicated (and long) to me.
So as a lazy reader I will put here my reflexion after some days of initial exploration of CUE.

I have the sensation that we mix here two different things which are concreteness and presence/absence of field.
In my opinion there is two end-user cases regarding concreteness:

1. you want all expanded fields to be concrete (e.g. in cue export or in validation of a concrete file)
2. you just want to check the coherence of a configuration (i.e. make sure unification does not trigger _|_)

I am not sufficiently proficient in CUE to understand what is said about "intermediate concreteness" (as in Default selection is used when a full specification of a value is required, but an intermediate concrete value is needed to proceed.) so I will skip this part without any shame ;-)

Regarding presence/absence of field, what I want to be able to express in term of validation:

• a field must be present
• a field can be present
• a field must be absent

I did some contorted things to be able to express all this.

proposal

I wonder if we could add this notion of presence/absence in the lattice.

In the lattice, present would be immediately under top and above all other values (even null) except absent.
absent would be like null i.e. between top and bottom with no other value.
As these types have no meaning if they do not refer to fields, e.g. in list context, we would have at this moment absent conflated to bottom and present conflated to top.

This would allow to express things which can only be expressed using tricks and closedness.
This would also allow to express default values.

For example I could write

requiredField: present
optionalField: present | absent

which would be equivalent to current syntax

requiredField: _
optionalField?: _

forbiding field

We could have too:

forbiddenField: absent

for which the only way I found to do it was

forbiddenField?: _|_

optional Field

The real equivalent to

optionalField?: 1

would be

optionalField: *absent | 1

default presence

But we could also write

x: optionalField: *1 | absent

which at the moment necessitates something like

x: *{optionalField: 1} | {optionalField?: _|_}

What do you think ?

[joulaud]

Just thinking out loud here but could the problem with requiring concreteness while sticking with unification be solved by having a way to declare a value as incomplete and thus force unification with a concrete value when validating.
e.g.

#Def: { requiredExplicit: incomplete("X") }
GoodVal: #Def & { requiredExplicit: "X" } // incomplete("X") and "X" unify to "X"
BadVal: #Def & {} // BadVal: requiredExplicit: incomplete value incomplete("X")

[SephVelut]

The proposal is a bit long and lacking in minimal examples to get at the main point being made. From my reading I assume the need here is a way to distinguish between definitions that evaluate to concrete types and definitions that evaluate to a universe of types.

The current default behavior is to treat definitions

// LHS: RHS
field: string

as an assertion that the RHS value will be anything in the set of things of the named type. So string is itself a string. "value" is also a string.

What I think the proposal is meant to achieve is to allow definitions to assert that RHS must be in the set of things of the named type, except the named type itself. This can be achieved without changing the default behavior of the type system.

What is being desired here is a way to exclude possible types within the inhabited set of possible types. "These, but not those". This sounds like a need for a refinement assertion like that of universe polymorphism in dependent type systems. "Type 1. would have to be strictly smaller than itself".

[myitcv]

What I think the proposal is meant to achieve is to allow definitions to assert that RHS must be in the set of things of the named type, except the named type itself.

That's not the case with this proposal. Because this is also valid:

c!: "test"

See the section "Example: discriminator fields" for motivation on this case.

[mpvl]

@joulaud Yes we did think about that. See the section "Required indication on the right-hand side" in the proposal, almost at the end. We did not find a way to represent this in the value lattice. (See the respective section.) That doesn't mean it isn't possible.

[mpvl]

Working on an updated and simplified proposal for this.

[antong]

I have a quick comment on the statement that APIs should be open and most fields should be optional (re closedness being a soft constraint, and optional fields and required fields): Some applications with extensible data formats have a concept of "critical" or "non skippable" fields. This is often seen in security sensitive applications and Internet protocols. Examples are X.509 certificates and the Mobile IPv4 protocol messaages (RFC5944). These have extension mechanisms in the data formats, that are similar to an open API accepting unknown fields. However, such extensions may have critical security implications which require that if the entity processing a message or file does not understand such an extension, it must produce an error and not handle the message or file (or API call) at all. This is in contrast to "non critical" or "skippable" extensions, that can just be disregarded and the message/file/call can be processed without handling the unknown field. As a concrete example, X.509v3 certificates have a list of extensions, each with an explicit boolean "critical" field for this purpose.

[mpvl]

A replacement proposal has been published here:

https://github.com/cue-lang/proposal/blob/main/designs/1951-required-fields-v2.md

This follows a new proposal process. Discussion of this new proposal is to be held here:

#1951

[mvdan]

The proposal was discussed at #1951 and got accepted, and then implemented in the CL chain at https://review.gerrithub.io/c/cue-lang/cue/+/551206. The feature first got released in v0.6.0-alpha.1: https://github.com/cue-lang/cue/releases/tag/v0.6.0-alpha.1

I think we can close this issue now, as it has served its purpose and has been replaced by the new proposal as described above. Any further input on the proposed design can be left in the linked discussion, and any bugs or other issues should be raised in the issue tracker following the bug template. Thank you all!